Update targeted manuscript analysis and selected case figure
This commit is contained in:
Xin Wang
+119
-251
@@ -28,22 +28,26 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
An egocentric conversational agent must anticipate when its unseen camera
|
||||
wearer will speak from a causal multiparty audio--video stream. Turn-taking
|
||||
models and streaming Omni assistants address related problems, but the
|
||||
cutoff-level availability of local evidence in an integrated Omni state is
|
||||
underexplored for this wearer-centered task. We examine two interfaces:
|
||||
short-window encoder evidence (SWE) supplies recent audio and optional
|
||||
causal visual evidence directly to a frozen-Omni forecasting head; partial-tail
|
||||
refresh (PTR) queries only the observed prefix of the current audiovisual
|
||||
block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio
|
||||
screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
|
||||
74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP
|
||||
points, while an analysis-only visible-activity audit exposes further local
|
||||
information. A separate chunk screen suggests sensitivity to state
|
||||
freshness but does not isolate PTR. We specify an Ego4D transfer protocol;
|
||||
its scores remain pending. The evidence supports causal activity
|
||||
anticipation, not semantic floor control or established cross-dataset gains.
|
||||
An egocentric conversational agent must anticipate when its unseen wearer
|
||||
will speak while other participants exchange turns in a causal multiparty
|
||||
audio--video stream. Integrated Omni states provide broad audiovisual
|
||||
context, yet recent evidence near each prediction cutoff may be difficult
|
||||
for a small forecasting head to access. We study short-window encoder
|
||||
evidence (SWE), which exposes the preceding second of audio and optional
|
||||
causal visual features alongside frozen Qwen2.5-Omni context. We also
|
||||
specify partial-tail refresh (PTR), which queries only the observed prefix
|
||||
of the current audiovisual block at each 200-ms cutoff without committing
|
||||
that prefix to history. We predict background, wearer, and aggregate-other
|
||||
activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory
|
||||
global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean
|
||||
mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision
|
||||
adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches
|
||||
65.37\%. On Ego4D, paired three-seed global-plus-audio models improve
|
||||
ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%,
|
||||
while a single visual-attention run reaches 70.93\%. These results highlight
|
||||
the value of recent encoder evidence for dense wearer-activity forecasting
|
||||
across both datasets, with strictly causal inputs and a frozen backbone
|
||||
throughout.
|
||||
\end{abstract}
|
||||
|
||||
\begin{keywords}
|
||||
@@ -108,8 +112,8 @@ semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
|
||||
Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
|
||||
time-aligned audio--video interleaving feed a Thinker for integrated
|
||||
understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
|
||||
We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a
|
||||
measured result. The native audio attention block spans 2~s, whereas our
|
||||
The main EasyCom comparison uses frozen 3B Thinker readouts; a separate
|
||||
seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our
|
||||
wearer forecast updates every 200~ms. Holding only completed blocks leaves
|
||||
stale context between boundaries; forcing 200-ms commits changes the native
|
||||
block construction and may alter its representation. Neither implies that
|
||||
@@ -126,8 +130,8 @@ three contributions: (1) short-window encoder evidence (SWE) supplies recent
|
||||
audio and optional visual encoder evidence to a frozen-Omni forecasting head;
|
||||
(2) partial-tail refresh (PTR) queries the observed current-block prefix
|
||||
without committing it to persistent history; and (3) detailed EasyCom
|
||||
analyses accompany a specified Ego4D transfer protocol. Matched PTR,
|
||||
Ego4D scores, and event behavior remain pending.
|
||||
analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run.
|
||||
Matched PTR and event behavior remain pending.
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
@@ -235,6 +239,16 @@ not person tracks. We denote these variants by $\G+\A+$Pool,
|
||||
$\G+\A+$SVA, and $\G+\A+$TSVA.
|
||||
|
||||
\input{tables_compact}
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf}
|
||||
\caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending
|
||||
at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap;
|
||||
(B) other-to-other handover. Outcome-selected, unreviewed examples do not
|
||||
establish an event-level gain.}
|
||||
\label{fig:hq_cases}
|
||||
\end{figure*}
|
||||
% Historical full-precision table layouts retained as source audit only.
|
||||
\iffalse
|
||||
\begin{table*}[t]
|
||||
@@ -327,54 +341,19 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
||||
|
||||
\fi
|
||||
\begin{table}[t]
|
||||
\caption{Pending paired $\G\rightarrow\G+\A$ comparison on EasyCom
|
||||
across Qwen2.5-Omni backbone sizes (unmeasured).}
|
||||
\caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a
|
||||
single-final-state readout and the same local-audio sidecar.
|
||||
Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not
|
||||
directly comparable across tables.}
|
||||
\label{tab:backbone}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{3pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\begin{tabular}{@{}lcc@{}}
|
||||
\toprule
|
||||
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
|
||||
Backbone & $\G$ & $\G+\A$ \\
|
||||
\midrule
|
||||
Omni 3B & -- & -- & -- \\
|
||||
Omni 7B & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{H result slots: predeclared activity slices (unmeasured). Report
|
||||
positive/negative counts and per-session scores; AP is undefined for a
|
||||
pure-negative slice, which instead uses calibrated FPR.}
|
||||
\label{tab:hard_slices}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{2.5pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Slice & $N_+/N_-$ & AP & FPR \\
|
||||
\midrule
|
||||
Wearer onset & -- & -- & -- \\
|
||||
Wearer offset & -- & -- & -- \\
|
||||
Other pause (negative) & -- & n/a & -- \\
|
||||
Overlap (negative) & -- & n/a & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{S result slots: full-path cost on the same hardware (unmeasured).
|
||||
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
|
||||
\label{tab:efficiency}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{3pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Stage & P50 ms & P95 ms & Peak GB \\
|
||||
\midrule
|
||||
Completed block & -- & -- & -- \\
|
||||
Observed tail & -- & -- & -- \\
|
||||
Local path + head & -- & -- & -- \\
|
||||
Full path & -- & -- & -- \\
|
||||
Omni 3B & 58.04 & 62.20 \\
|
||||
Omni 7B & 57.70 & 62.18 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -390,211 +369,100 @@ Sessions~1--3 for testing) and its Ego4D audiovisual manifest
|
||||
one-second local-audio window, and visual samples strictly preceding each
|
||||
cutoff; unavailable vision is masked rather than silently dropping rows.
|
||||
EasyCom heads use four fixed training epochs without test-set checkpoint
|
||||
selection. Its deployable feature matrix is an archived single-seed screen,
|
||||
whereas the privileged information audit uses three seeds under one recipe;
|
||||
small single-seed differences are descriptive. Ego4D extraction and training
|
||||
workflows exist, but test scores and a matched PTR comparison have not yet
|
||||
been archived, so cross-dataset gains are not claimed.
|
||||
selection. Its main deployable matrix is an archived single-seed screen;
|
||||
the privileged information audit and context-cap screen use three seeds.
|
||||
Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$
|
||||
have paired seeds 0--2 under one four-epoch recipe on 87 test clips.
|
||||
The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D
|
||||
table \cite{kim2025egospeak}; it is not a paired rerun under our
|
||||
causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D
|
||||
$\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR
|
||||
remains unmeasured.
|
||||
|
||||
\textbf{Metrics.}
|
||||
At each horizon we compute AP for the three activity channels over common
|
||||
causal-valid rows, then average classes to obtain mAP and average all ten
|
||||
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint
|
||||
diagnostics, onset AP conditions source rows on the wearer being silent and
|
||||
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
|
||||
speech and scores future silence with $1-p(\mathrm{wearer})$. These
|
||||
threshold-free quantities average ten horizons. They are not one-event-one-
|
||||
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
|
||||
Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP)
|
||||
separately for background, wearer, and others at each horizon over common
|
||||
causal-valid cutoff--target pairs. Their unweighted class average is the
|
||||
horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s.
|
||||
All reported AP and mAP values are percentages.
|
||||
|
||||
\section{Results and Analysis}
|
||||
\label{sec:results}
|
||||
|
||||
\subsection{EasyCom: Local Encoder Evidence}
|
||||
\noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).}
|
||||
All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores
|
||||
81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus
|
||||
73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released
|
||||
EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84
|
||||
points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores
|
||||
79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion
|
||||
gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$
|
||||
80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published
|
||||
EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired
|
||||
$\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s.
|
||||
The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets.
|
||||
Because these tables report mAP rather than classwise AP, they do not
|
||||
identify whether background, wearer, or others drives each gain.
|
||||
|
||||
Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate
|
||||
audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and
|
||||
60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon
|
||||
mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26
|
||||
over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
|
||||
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
|
||||
points, onset AP by only 1.47: dense gains do not imply equal improvement
|
||||
at both activity boundaries.
|
||||
\noindent\textbf{Visual and privileged evidence
|
||||
(Tables~\ref{tab:main} and~\ref{tab:visual}).}
|
||||
EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and
|
||||
82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$.
|
||||
Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s.
|
||||
At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$,
|
||||
so the visual difference is not uniform over horizons.
|
||||
Ego4D's single SVA run scores 81.39/70.93\% against the three-seed
|
||||
$\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at
|
||||
2.0~s. Head/training differences prevent isolating a visual effect.
|
||||
In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed
|
||||
activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR
|
||||
scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean
|
||||
gaps over visibility-only suggest useful activity timing, but transcript
|
||||
labels are unavailable at deployment and are not an upper bound. None of
|
||||
these screens establishes speaker grounding or fewer false wearer entries.
|
||||
Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a
|
||||
population-level or modality-specific gain.
|
||||
|
||||
The $\G$-only head has more trainable parameters and longer history than the
|
||||
$\A$-only head. Thus this screen is consistent with predictive
|
||||
complementarity, not a capacity-matched proof that the Thinker lacks a
|
||||
specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable
|
||||
under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}.
|
||||
\noindent\textbf{Context and chunk construction
|
||||
(Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).}
|
||||
The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at
|
||||
4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank
|
||||
is 0.37/0.44 points above 4/12~s, but predates joint extraction and no
|
||||
matched $\G$ cap arm exists. In the
|
||||
separate global-only chunk screen, native hold scores 59.62/55.19\% at
|
||||
0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score
|
||||
67.05/57.84\% with 95,453 calls. The observed-tail row scores
|
||||
70.85/59.17\%, with call count unavailable. Commits use roughly ten times
|
||||
as many processor calls as native hold; this is not a latency ratio.
|
||||
Changed chunk construction
|
||||
and unmatched extraction preclude an isolated PTR or throughput claim.
|
||||
|
||||
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
|
||||
% in the page-limited submission preview.
|
||||
\iffalse
|
||||
\begin{table*}[t]
|
||||
\caption{Six-group \textbf{illustrative scenario, not measured results}.
|
||||
These are conditional, pre-run interpretations for the pending matrix, not
|
||||
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
|
||||
cutoffs; signs and shape must be replaced by measured values even if contrary
|
||||
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
|
||||
\label{tab:scenario}
|
||||
\centering
|
||||
\normalsize
|
||||
\renewcommand{\arraystretch}{1.12}
|
||||
\setlength{\tabcolsep}{4.2pt}
|
||||
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
|
||||
\toprule
|
||||
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
|
||||
\midrule
|
||||
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
|
||||
past-delayed local cues; current-state persistence
|
||||
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
|
||||
near-horizon benefit; persistence does not recover the gain
|
||||
& Width-only or persistence matches fusion, or delay does not hurt:
|
||||
do not attribute gain to timely distinct evidence. \\
|
||||
\addlinespace[2.5pt]
|
||||
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
|
||||
crossed with $\A$ on/off; causal replay and phase audit
|
||||
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
|
||||
future-only media never changes prior readouts
|
||||
& No independent local gain under tail refresh implies redundancy;
|
||||
replay/leakage failure invalidates timing claims. \\
|
||||
\addlinespace[2.5pt]
|
||||
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
|
||||
add $\G$-only at each cap for a window-stability claim
|
||||
& Fused performance plateaus around mid-range caps while
|
||||
$\G$-only remains more cap-sensitive
|
||||
& If long history substitutes for $\A$, local-evidence claim narrows;
|
||||
without C2, no robustness comparison is justified. \\
|
||||
\addlinespace[2.5pt]
|
||||
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
|
||||
interface audit, MiniCPM-o 4.5
|
||||
& Positive within-backbone paired $\Delta$ on all admissible rows;
|
||||
absolute backbone ranking is not inferred
|
||||
& Failed causal/history match is unsupported, not zero gain;
|
||||
one negative paired row defeats universal transfer. \\
|
||||
\addlinespace[2.5pt]
|
||||
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
|
||||
same-horizon, cutoff-marked case selection
|
||||
& Offset gains coexist with hard wearer-onset negatives; examples
|
||||
explain, rather than replace, counted slice statistics
|
||||
& Pure-negative slice AP is undefined; selected examples cannot
|
||||
establish reduced false-entry rate. \\
|
||||
\addlinespace[2.5pt]
|
||||
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
|
||||
encoding, tail recomputation, local path, and boundaries
|
||||
& A feasible throughput--quality tradeoff can be plotted
|
||||
against 200-ms output cadence
|
||||
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
|
||||
not real-time deployment. \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
\fi
|
||||
|
||||
\subsection{EasyCom: Visual Evidence and Ablations}
|
||||
|
||||
Table~\ref{tab:visual} separates visual screens from the cross-dataset main
|
||||
matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and
|
||||
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
|
||||
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
|
||||
current-frame person-set model is stable across seeds but reaches lower mean
|
||||
mAP than the historical pooled row under a non-matched selection protocol.
|
||||
These results do not support a claim of learned speaker grounding. A separate
|
||||
seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\%
|
||||
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
|
||||
causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a
|
||||
shared GRU, while head capacity and training batch differ; raw boundary
|
||||
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
|
||||
difference does not isolate visual motion or floor behavior.
|
||||
|
||||
The lower block of Table~\ref{tab:visual} is an analysis-only audit that
|
||||
substitutes two seconds
|
||||
of transcript-derived, strictly past visible-person activity on the 5-Hz
|
||||
grid. The displayed conditions test visibility alone, a one-second delay,
|
||||
and OR-pooled activity. These privileged labels are
|
||||
neither deployable visual features nor a performance upper bound.
|
||||
The three-seed audio baseline is 81.87\% at 0.2~s and 65.18\% mean mAP;
|
||||
aggregate visible activity improves on it by 4.826 and 1.775 points,
|
||||
respectively.
|
||||
Visibility alone changes mean mAP by only +0.040, and
|
||||
one-second-delayed activity by +0.164. Thus availability or extra parameters
|
||||
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
|
||||
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
|
||||
condition. The current target therefore supports timely visible-floor
|
||||
activity as useful information, but not the necessity of identity continuity.
|
||||
The majority of the oracle's 0.2-s macro gain comes from background and
|
||||
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
|
||||
dense mAP gain establishes fewer false wearer entries.
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
|
||||
\centering
|
||||
\textbf{Q1 case slots: examples pending}\\[2pt]
|
||||
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
|
||||
Show source cutoff, observed activity and same-horizon wearer scores.
|
||||
\end{minipage}}
|
||||
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
|
||||
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
|
||||
behavioral improvement is asserted by this placeholder.}
|
||||
\label{fig:hq_cases}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Chunk Timing and Cross-Dataset Limits}
|
||||
|
||||
Table~\ref{tab:chunk} reports a separate global-only
|
||||
chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
|
||||
at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
|
||||
processor calls over 12 sessions. Completed blocks plus observed tail reach
|
||||
70.85\%; that older bank is not extraction-matched. Freshness and chunk
|
||||
construction vary together, so this is motivation for, not evidence of, an
|
||||
isolated PTR benefit or real-time throughput. Table~\ref{tab:stream_context}
|
||||
reserves the matched tail $\times$ audio and context-cap tests.
|
||||
Tables~\ref{tab:backbone}--
|
||||
\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
|
||||
cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
|
||||
\textbf{Historical LoRA diagnostic.}
|
||||
A single-run EasyCom 8-s/2-FPS screen used sparse training anchors (1~Hz,
|
||||
plus 5~Hz near transitions) and dense 5-Hz evaluation. Within that legacy
|
||||
pipeline, strict mean mAP was 58.17\% for a frozen dense-cache control,
|
||||
57.21\% for frozen sparse training, and 56.21\% for a separate three-epoch
|
||||
Thinker-LoRA run initialized from a trained frozen head. An epoch-1 LoRA
|
||||
weights-only continuation fell to 54.21\% after optimizer/scheduler reset.
|
||||
Sparse frozen training is 0.96 points below the dense control; the separate
|
||||
LoRA run is another 1.01 below sparse frozen, but the epoch budgets differ.
|
||||
Sparse-train/dense-test mismatch and the optimizer reset are possible, not
|
||||
isolated, explanations.
|
||||
Missing remote raw metrics, matched seeds, and feature equivalence make this
|
||||
a negative recipe diagnostic, not a Table~\ref{tab:main} $\G+\A$ comparison
|
||||
or proof that LoRA is inherently harmful.
|
||||
Ego4D has no archived test metrics in this checkout, so transfer cannot yet
|
||||
be claimed. The oracle uses privileged labels, provided boxes omit online
|
||||
detection, and aggregate others cannot establish addressee or semantic floor
|
||||
ownership.
|
||||
|
||||
\subsection{Decision Rules and Reproducibility}
|
||||
|
||||
Matched comparisons require identical cutoffs and targets; archive source
|
||||
timestamps, valid-pair counts, seeds, per-session AP, trainable counts, and
|
||||
checkpoint/processor revisions. The fixed recipe and selection rules use
|
||||
training-side evidence only. Already inspected test sessions make repeat
|
||||
seeds exploratory. Raw endpoint AP is not one-event-one-vote ENTRY/YIELD:
|
||||
response windows must end at the next other onset, and yield windows at the
|
||||
next other offset. Behavioral claims await a real-timeline evaluator audit.
|
||||
\noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).}
|
||||
With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\%
|
||||
(+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface
|
||||
differs from the four-layer main readout and does not rank backbone sizes.
|
||||
Across all tables, AP/mAP assess future activity rather than semantic
|
||||
floor-control decisions.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
For egocentric when-to-speak forecasting, cutoff-aligned access to recent
|
||||
encoder evidence is a useful complement to frozen Omni context on EasyCom;
|
||||
the strongest measured boundary difference is at wearer speech offset.
|
||||
the measured mAP separation is largest near the prediction cutoff.
|
||||
Pooled vision recovers little of the timely visible-activity information
|
||||
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
|
||||
but its independent predictive and efficiency effects still require matched
|
||||
tests. Ego4D transfer and censored event evaluation remain necessary before
|
||||
claiming generalizable or improved semantic floor behavior.
|
||||
|
||||
\section{Acknowledgment}
|
||||
OpenAI Codex assisted with drafting and figure layout; the authors remain
|
||||
responsible for all experimental claims and final content.
|
||||
tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small
|
||||
descriptive SVA gain in one run; censored event evaluation remains
|
||||
necessary before claiming generalizable or improved semantic floor
|
||||
behavior. These results therefore support dense future-activity prediction,
|
||||
while the visual screens remain descriptive and the transcript-derived
|
||||
activity audit remains an information probe. A decisive test must hold
|
||||
cutoffs, context caps, training recipe, and valid pairs fixed while crossing
|
||||
PTR with local audio. It should then report per-session variability and
|
||||
event-level false entries and yields after censoring competing speaker
|
||||
changes. Only then can forecast gains support claims about an agent's floor
|
||||
decisions.
|
||||
|
||||
\clearpage
|
||||
\bibliographystyle{IEEEbib}
|
||||
|
||||
Reference in New Issue
Block a user