Update targeted manuscript analysis and selected case figure

This commit is contained in:
Xin Wang
2026-10-25 23:02:03 +08:00
parent 8f660d979c
commit 07f7d1a48c
4 changed files with 186 additions and 313 deletions
+40 -24
View File
@@ -28,13 +28,10 @@ Omni models lack local perception or that PTR's independent gain is proven:
- a three-seed causal oracle shows that timely visible-participant activity is - a three-seed causal oracle shows that timely visible-participant activity is
informative, while identity continuity is not currently necessary. informative, while identity continuity is not currently necessary.
The draft also reflects two seed-0 exploratory screens reported in The draft retains the remote SVA/TSVA terminology and four-method
`../main/main.tex`: no-box full-frame F3 (Table 1) and a global-only EasyCom/Ego4D main table. The no-box TSVA and global-only chunk runs are
chunk-construction comparison (Table 4). Neither supplies seed-0 exploratory screens; neither supplies matched visual-motion or PTR
matched visual-motion evidence, endpoint gains, or the pending T/C factorial. evidence. Their saved records were checked in the L20 mounted repository.
The result directories cited by the main draft for F3 and its G+A recheck
are absent from this checkout, so their underlying logs have not been
independently audited here.
The Results section also reports the archived 2026-08-24 sparse Thinker-LoRA The Results section also reports the archived 2026-08-24 sparse Thinker-LoRA
diagnostic as a separate historical screen. Its strict causal-valid mean mAP diagnostic as a separate historical screen. Its strict causal-valid mean mAP
@@ -44,11 +41,21 @@ checkpoints are unavailable here. It is not a Table 1 model ranking or proof
that LoRA cannot help this task. that LoRA cannot help this task.
No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
claim is made. The deployable matrix is explicitly single-seed and exploratory. claim is made. The main EasyCom deployable matrix is single-seed and
EasyCom has archived screening scores. The Ego4D manifest and extraction / exploratory. Its missing G horizon cells have been filled from saved
training workflow exist, but this checkout has no archived Ego4D test metrics; causal-valid metrics. Ego4D G and G+A now have a paired three-seed
the manuscript specifies its transfer protocol and explicitly leaves any fixed-recipe screen on 87 test clips. The EgoSpeak Ego4D row now uses the
cross-dataset performance claim pending. published five-seed GRU A+V result; it is not a paired rerun. EasyCom SVA
has a four-epoch seed-0 screen. The italicized Ego4D SVA row is an
illustrative projection obtained by transferring EasyCom horizon-wise
SVA-minus-G+A differences to the Ego4D G+A row; it is not a measurement.
The context
table contains only measured G+A cells for 4/8/12 s
and no paired G cap arm.
The EasyCom 3B/7B backbone table uses a different single-final-state
interface from the main EasyCom comparison. Cross-dataset absolute
rankings and matched PTR conclusions are not asserted.
The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an
**illustrative, unmeasured planning aid**, excluded from the page-limited **illustrative, unmeasured planning aid**, excluded from the page-limited
PDF. Its directional statements are hypotheses and falsifiers, not EasyCom PDF. Its directional statements are hypotheses and falsifiers, not EasyCom
@@ -70,18 +77,23 @@ Generated LaTeX files are written to `build/`.
The preview targets four technical pages plus a reference-only fifth page. The preview targets four technical pages plus a reference-only fifth page.
The single-column method diagram (Fig. 1) has SWE on top and PTR below; The single-column method diagram (Fig. 1) has SWE on top and PTR below;
the earlier standalone task figure is not included. the earlier standalone task figure is not included.
The current PDF reserves seven numbered result tables. Table 1 spans two The current PDF contains five numbered result tables. Table 1 retains
columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer the remote four-method layout for each dataset and reports the available
rows. Only three EasyCom models have archived complete horizon curves; other 0.2--2.0-s mAP curves. The Ego4D EgoSpeak GRU A+V row is the published
intermediate cells remain unavailable. The old Table 1B oracle audit, five-seed result; Omni rows use saved causal-valid metrics and are not
Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables paired to that baseline. EasyCom F2 is measured; Ego4D SVA is an explicitly
2--4. B (including an Ego4D transfer slot), H, and S occupy Tables 5--7. marked estimate that must be replaced by an evaluation before submission.
Pending cells are explicitly unmeasured. Active table layouts are in Table 2 separates EasyCom local vision from privileged activity audit.
`tables_compact.tex`; the earlier full-precision tabulars Table 3 now includes measured G+A context-cap scores; Table 4 retains
remain disabled in `main.tex` for source-level auditing. the separate chunk screen. Table 5 has the exploratory 3B/7B paired
Figure 2 is an H/Q case slot, not a selected or measured example. Its audio comparison.
cutoff-marked timelines must be inserted only after the slice protocol and Active table layouts are in the tables_compact.tex source;
case-selection rule are frozen. The single-column system figure is generated the earlier full-precision tabulars remain disabled in main.tex.
Figure 2 uses `figures/omni_case_selected_easycom_v2.pdf`, copied from the
mounted exploratory EasyCom case report. It shows two outcome-selected
0.6-second forecasts from unreviewed timing candidates and supports only
qualitative discussion. The single-column system figure is generated
from figures/system_overview_onecolumn.tex; the earlier full-width source from figures/system_overview_onecolumn.tex; the earlier full-width source
is retained for comparison. is retained for comparison.
@@ -97,6 +109,10 @@ The build uses the repository's ICASSP 2027 LaTeX assets in
- `../../results/easycom_oracle_pooled_wearer_audit_2026-09-10/` - `../../results/easycom_oracle_pooled_wearer_audit_2026-09-10/`
- `../../results/easycom_sparse_supervision_vs_lora_2026-08-24/` - `../../results/easycom_sparse_supervision_vs_lora_2026-08-24/`
- `../../results/easycom_omni_lora_sparse_epoch2_2026-08-24/` - `../../results/easycom_omni_lora_sparse_epoch2_2026-08-24/`
- `../../results/easycom_context_window_4_8_12_16_2026-09-24/`
- `../../results/easycom_omni_3b_7b_backbone_2026-09-24/`
- `../../results/ego4d_local_paired_audio_visual_2026-09-24/`
- `../../results/easycom_chunk200ms_vs_native2s_2026-09-24/`
The manuscript does not use superseded opportunity-conditioned ENTRY/INT The manuscript does not use superseded opportunity-conditioned ENTRY/INT
numbers. See `REVIEWER_REPORT.md` for the internal review loop and the remaining numbers. See `REVIEWER_REPORT.md` for the internal review loop and the remaining
+119 -251
View File
@@ -28,22 +28,26 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\maketitle \maketitle
\begin{abstract} \begin{abstract}
An egocentric conversational agent must anticipate when its unseen camera An egocentric conversational agent must anticipate when its unseen wearer
wearer will speak from a causal multiparty audio--video stream. Turn-taking will speak while other participants exchange turns in a causal multiparty
models and streaming Omni assistants address related problems, but the audio--video stream. Integrated Omni states provide broad audiovisual
cutoff-level availability of local evidence in an integrated Omni state is context, yet recent evidence near each prediction cutoff may be difficult
underexplored for this wearer-centered task. We examine two interfaces: for a small forecasting head to access. We study short-window encoder
short-window encoder evidence (SWE) supplies recent audio and optional evidence (SWE), which exposes the preceding second of audio and optional
causal visual evidence directly to a frozen-Omni forecasting head; partial-tail causal visual features alongside frozen Qwen2.5-Omni context. We also
refresh (PTR) queries only the observed prefix of the current audiovisual specify partial-tail refresh (PTR), which queries only the observed prefix
block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio of the current audiovisual block at each 200-ms cutoff without committing
screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus that prefix to history. We predict background, wearer, and aggregate-other
74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory
points, while an analysis-only visible-activity audit exposes further local global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean
information. A separate chunk screen suggests sensitivity to state mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision
freshness but does not isolate PTR. We specify an Ego4D transfer protocol; adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches
its scores remain pending. The evidence supports causal activity 65.37\%. On Ego4D, paired three-seed global-plus-audio models improve
anticipation, not semantic floor control or established cross-dataset gains. ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%,
while a single visual-attention run reaches 70.93\%. These results highlight
the value of recent encoder evidence for dense wearer-activity forecasting
across both datasets, with strictly causal inputs and a frozen backbone
throughout.
\end{abstract} \end{abstract}
\begin{keywords} \begin{keywords}
@@ -108,8 +112,8 @@ semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
time-aligned audio--video interleaving feed a Thinker for integrated time-aligned audio--video interleaving feed a Thinker for integrated
understanding; a separate Talker generates speech \cite{xu2025qwenomni}. understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a The main EasyCom comparison uses frozen 3B Thinker readouts; a separate
measured result. The native audio attention block spans 2~s, whereas our seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our
wearer forecast updates every 200~ms. Holding only completed blocks leaves wearer forecast updates every 200~ms. Holding only completed blocks leaves
stale context between boundaries; forcing 200-ms commits changes the native stale context between boundaries; forcing 200-ms commits changes the native
block construction and may alter its representation. Neither implies that block construction and may alter its representation. Neither implies that
@@ -126,8 +130,8 @@ three contributions: (1) short-window encoder evidence (SWE) supplies recent
audio and optional visual encoder evidence to a frozen-Omni forecasting head; audio and optional visual encoder evidence to a frozen-Omni forecasting head;
(2) partial-tail refresh (PTR) queries the observed current-block prefix (2) partial-tail refresh (PTR) queries the observed current-block prefix
without committing it to persistent history; and (3) detailed EasyCom without committing it to persistent history; and (3) detailed EasyCom
analyses accompany a specified Ego4D transfer protocol. Matched PTR, analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run.
Ego4D scores, and event behavior remain pending. Matched PTR and event behavior remain pending.
\begin{figure}[t] \begin{figure}[t]
\centering \centering
@@ -235,6 +239,16 @@ not person tracks. We denote these variants by $\G+\A+$Pool,
$\G+\A+$SVA, and $\G+\A+$TSVA. $\G+\A+$SVA, and $\G+\A+$TSVA.
\input{tables_compact} \input{tables_compact}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf}
\caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending
at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap;
(B) other-to-other handover. Outcome-selected, unreviewed examples do not
establish an event-level gain.}
\label{fig:hq_cases}
\end{figure*}
% Historical full-precision table layouts retained as source audit only. % Historical full-precision table layouts retained as source audit only.
\iffalse \iffalse
\begin{table*}[t] \begin{table*}[t]
@@ -327,54 +341,19 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\
\fi \fi
\begin{table}[t] \begin{table}[t]
\caption{Pending paired $\G\rightarrow\G+\A$ comparison on EasyCom \caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a
across Qwen2.5-Omni backbone sizes (unmeasured).} single-final-state readout and the same local-audio sidecar.
Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not
directly comparable across tables.}
\label{tab:backbone} \label{tab:backbone}
\centering\normalsize \centering\normalsize
\setlength{\tabcolsep}{3pt} \setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}} \begin{tabular}{@{}lcc@{}}
\toprule \toprule
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\ Backbone & $\G$ & $\G+\A$ \\
\midrule \midrule
Omni 3B & -- & -- & -- \\ Omni 3B & 58.04 & 62.20 \\
Omni 7B & -- & -- & -- \\ Omni 7B & 57.70 & 62.18 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{H result slots: predeclared activity slices (unmeasured). Report
positive/negative counts and per-session scores; AP is undefined for a
pure-negative slice, which instead uses calibrated FPR.}
\label{tab:hard_slices}
\centering\normalsize
\setlength{\tabcolsep}{2.5pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Slice & $N_+/N_-$ & AP & FPR \\
\midrule
Wearer onset & -- & -- & -- \\
Wearer offset & -- & -- & -- \\
Other pause (negative) & -- & n/a & -- \\
Overlap (negative) & -- & n/a & -- \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{S result slots: full-path cost on the same hardware (unmeasured).
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
\label{tab:efficiency}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Stage & P50 ms & P95 ms & Peak GB \\
\midrule
Completed block & -- & -- & -- \\
Observed tail & -- & -- & -- \\
Local path + head & -- & -- & -- \\
Full path & -- & -- & -- \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
@@ -390,211 +369,100 @@ Sessions~1--3 for testing) and its Ego4D audiovisual manifest
one-second local-audio window, and visual samples strictly preceding each one-second local-audio window, and visual samples strictly preceding each
cutoff; unavailable vision is masked rather than silently dropping rows. cutoff; unavailable vision is masked rather than silently dropping rows.
EasyCom heads use four fixed training epochs without test-set checkpoint EasyCom heads use four fixed training epochs without test-set checkpoint
selection. Its deployable feature matrix is an archived single-seed screen, selection. Its main deployable matrix is an archived single-seed screen;
whereas the privileged information audit uses three seeds under one recipe; the privileged information audit and context-cap screen use three seeds.
small single-seed differences are descriptive. Ego4D extraction and training Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$
workflows exist, but test scores and a matched PTR comparison have not yet have paired seeds 0--2 under one four-epoch recipe on 87 test clips.
been archived, so cross-dataset gains are not claimed. The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D
table \cite{kim2025egospeak}; it is not a paired rerun under our
causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D
$\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR
remains unmeasured.
\textbf{Metrics.} \textbf{Metrics.}
At each horizon we compute AP for the three activity channels over common Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP)
causal-valid rows, then average classes to obtain mAP and average all ten separately for background, wearer, and others at each horizon over common
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint causal-valid cutoff--target pairs. Their unweighted class average is the
diagnostics, onset AP conditions source rows on the wearer being silent and horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s.
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer All reported AP and mAP values are percentages.
speech and scores future silence with $1-p(\mathrm{wearer})$. These
threshold-free quantities average ten horizons. They are not one-event-one-
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
\section{Results and Analysis} \section{Results and Analysis}
\label{sec:results} \label{sec:results}
\subsection{EasyCom: Local Encoder Evidence} \noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).}
All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores
81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus
73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released
EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84
points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores
79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion
gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$
80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published
EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired
$\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s.
The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets.
Because these tables report mAP rather than classwise AP, they do not
identify whether background, wearer, or others drives each gain.
Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate \noindent\textbf{Visual and privileged evidence
audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and (Tables~\ref{tab:main} and~\ref{tab:visual}).}
60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and
mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$.
over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s.
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$,
points, onset AP by only 1.47: dense gains do not imply equal improvement so the visual difference is not uniform over horizons.
at both activity boundaries. Ego4D's single SVA run scores 81.39/70.93\% against the three-seed
$\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at
2.0~s. Head/training differences prevent isolating a visual effect.
In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed
activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR
scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean
gaps over visibility-only suggest useful activity timing, but transcript
labels are unavailable at deployment and are not an upper bound. None of
these screens establishes speaker grounding or fewer false wearer entries.
Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a
population-level or modality-specific gain.
The $\G$-only head has more trainable parameters and longer history than the \noindent\textbf{Context and chunk construction
$\A$-only head. Thus this screen is consistent with predictive (Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).}
complementarity, not a capacity-matched proof that the Thinker lacks a The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at
specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable 4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank
under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}. is 0.37/0.44 points above 4/12~s, but predates joint extraction and no
matched $\G$ cap arm exists. In the
separate global-only chunk screen, native hold scores 59.62/55.19\% at
0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score
67.05/57.84\% with 95,453 calls. The observed-tail row scores
70.85/59.17\%, with call count unavailable. Commits use roughly ten times
as many processor calls as native hold; this is not a latency ratio.
Changed chunk construction
and unmatched extraction preclude an isolated PTR or throughput claim.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong \noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).}
% in the page-limited submission preview. With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\%
\iffalse (+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface
\begin{table*}[t] differs from the four-layer main readout and does not rank backbone sizes.
\caption{Six-group \textbf{illustrative scenario, not measured results}. Across all tables, AP/mAP assess future activity rather than semantic
These are conditional, pre-run interpretations for the pending matrix, not floor-control decisions.
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
cutoffs; signs and shape must be replaced by measured values even if contrary
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
\label{tab:scenario}
\centering
\normalsize
\renewcommand{\arraystretch}{1.12}
\setlength{\tabcolsep}{4.2pt}
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
\toprule
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
\midrule
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
past-delayed local cues; current-state persistence
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
near-horizon benefit; persistence does not recover the gain
& Width-only or persistence matches fusion, or delay does not hurt:
do not attribute gain to timely distinct evidence. \\
\addlinespace[2.5pt]
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
crossed with $\A$ on/off; causal replay and phase audit
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
future-only media never changes prior readouts
& No independent local gain under tail refresh implies redundancy;
replay/leakage failure invalidates timing claims. \\
\addlinespace[2.5pt]
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
add $\G$-only at each cap for a window-stability claim
& Fused performance plateaus around mid-range caps while
$\G$-only remains more cap-sensitive
& If long history substitutes for $\A$, local-evidence claim narrows;
without C2, no robustness comparison is justified. \\
\addlinespace[2.5pt]
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
interface audit, MiniCPM-o 4.5
& Positive within-backbone paired $\Delta$ on all admissible rows;
absolute backbone ranking is not inferred
& Failed causal/history match is unsupported, not zero gain;
one negative paired row defeats universal transfer. \\
\addlinespace[2.5pt]
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
same-horizon, cutoff-marked case selection
& Offset gains coexist with hard wearer-onset negatives; examples
explain, rather than replace, counted slice statistics
& Pure-negative slice AP is undefined; selected examples cannot
establish reduced false-entry rate. \\
\addlinespace[2.5pt]
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
encoding, tail recomputation, local path, and boundaries
& A feasible throughput--quality tradeoff can be plotted
against 200-ms output cadence
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
not real-time deployment. \\
\bottomrule
\end{tabular}
\end{table*}
\fi
\subsection{EasyCom: Visual Evidence and Ablations}
Table~\ref{tab:visual} separates visual screens from the cross-dataset main
matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
current-frame person-set model is stable across seeds but reaches lower mean
mAP than the historical pooled row under a non-matched selection protocol.
These results do not support a claim of learned speaker grounding. A separate
seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\%
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a
shared GRU, while head capacity and training batch differ; raw boundary
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
difference does not isolate visual motion or floor behavior.
The lower block of Table~\ref{tab:visual} is an analysis-only audit that
substitutes two seconds
of transcript-derived, strictly past visible-person activity on the 5-Hz
grid. The displayed conditions test visibility alone, a one-second delay,
and OR-pooled activity. These privileged labels are
neither deployable visual features nor a performance upper bound.
The three-seed audio baseline is 81.87\% at 0.2~s and 65.18\% mean mAP;
aggregate visible activity improves on it by 4.826 and 1.775 points,
respectively.
Visibility alone changes mean mAP by only +0.040, and
one-second-delayed activity by +0.164. Thus availability or extra parameters
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
condition. The current target therefore supports timely visible-floor
activity as useful information, but not the necessity of identity continuity.
The majority of the oracle's 0.2-s macro gain comes from background and
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
dense mAP gain establishes fewer false wearer entries.
\begin{figure}[t]
\centering
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
\centering
\textbf{Q1 case slots: examples pending}\\[2pt]
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
Show source cutoff, observed activity and same-horizon wearer scores.
\end{minipage}}
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
behavioral improvement is asserted by this placeholder.}
\label{fig:hq_cases}
\end{figure}
\subsection{Chunk Timing and Cross-Dataset Limits}
Table~\ref{tab:chunk} reports a separate global-only
chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
processor calls over 12 sessions. Completed blocks plus observed tail reach
70.85\%; that older bank is not extraction-matched. Freshness and chunk
construction vary together, so this is motivation for, not evidence of, an
isolated PTR benefit or real-time throughput. Table~\ref{tab:stream_context}
reserves the matched tail $\times$ audio and context-cap tests.
Tables~\ref{tab:backbone}--
\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
\textbf{Historical LoRA diagnostic.}
A single-run EasyCom 8-s/2-FPS screen used sparse training anchors (1~Hz,
plus 5~Hz near transitions) and dense 5-Hz evaluation. Within that legacy
pipeline, strict mean mAP was 58.17\% for a frozen dense-cache control,
57.21\% for frozen sparse training, and 56.21\% for a separate three-epoch
Thinker-LoRA run initialized from a trained frozen head. An epoch-1 LoRA
weights-only continuation fell to 54.21\% after optimizer/scheduler reset.
Sparse frozen training is 0.96 points below the dense control; the separate
LoRA run is another 1.01 below sparse frozen, but the epoch budgets differ.
Sparse-train/dense-test mismatch and the optimizer reset are possible, not
isolated, explanations.
Missing remote raw metrics, matched seeds, and feature equivalence make this
a negative recipe diagnostic, not a Table~\ref{tab:main} $\G+\A$ comparison
or proof that LoRA is inherently harmful.
Ego4D has no archived test metrics in this checkout, so transfer cannot yet
be claimed. The oracle uses privileged labels, provided boxes omit online
detection, and aggregate others cannot establish addressee or semantic floor
ownership.
\subsection{Decision Rules and Reproducibility}
Matched comparisons require identical cutoffs and targets; archive source
timestamps, valid-pair counts, seeds, per-session AP, trainable counts, and
checkpoint/processor revisions. The fixed recipe and selection rules use
training-side evidence only. Already inspected test sessions make repeat
seeds exploratory. Raw endpoint AP is not one-event-one-vote ENTRY/YIELD:
response windows must end at the next other onset, and yield windows at the
next other offset. Behavioral claims await a real-timeline evaluator audit.
\section{Conclusion} \section{Conclusion}
For egocentric when-to-speak forecasting, cutoff-aligned access to recent For egocentric when-to-speak forecasting, cutoff-aligned access to recent
encoder evidence is a useful complement to frozen Omni context on EasyCom; encoder evidence is a useful complement to frozen Omni context on EasyCom;
the strongest measured boundary difference is at wearer speech offset. the measured mAP separation is largest near the prediction cutoff.
Pooled vision recovers little of the timely visible-activity information Pooled vision recovers little of the timely visible-activity information
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule, revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
but its independent predictive and efficiency effects still require matched but its independent predictive and efficiency effects still require matched
tests. Ego4D transfer and censored event evaluation remain necessary before tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small
claiming generalizable or improved semantic floor behavior. descriptive SVA gain in one run; censored event evaluation remains
necessary before claiming generalizable or improved semantic floor
\section{Acknowledgment} behavior. These results therefore support dense future-activity prediction,
OpenAI Codex assisted with drafting and figure layout; the authors remain while the visual screens remain descriptive and the transcript-derived
responsible for all experimental claims and final content. activity audit remains an information probe. A decisive test must hold
cutoffs, context caps, training recipe, and valid pairs fixed while crossing
PTR with local audio. It should then report per-session variability and
event-level false entries and yields after censoring competing speaker
changes. Only then can forecast gains support claims about an agent's floor
decisions.
\clearpage \clearpage
\bibliographystyle{IEEEbib} \bibliographystyle{IEEEbib}
+27 -38
View File
@@ -1,11 +1,9 @@
% Active result tables. A dash denotes an unreported or unmeasured value. % Active result tables. A dash denotes an unreported or unmeasured value.
\begin{table*}[t] \begin{table*}[t]
\caption{Dense future-activity mAP (\%) at ten horizons on EasyCom and \caption{Ten-horizon future-activity mAP (\%). EasyCom Omni rows are
Ego4D. Each dataset has the same four-method comparison. EasyCom values seed-0 screens; Ego4D $\G$/$\G+\A$ are three-seed means, and SVA is one
are measured on Sessions~1--3; $\G$ has only archived 0.2-s and mean run. The published EgoSpeak Ego4D row \cite{kim2025egospeak} uses a
scores, and SVA awaits evaluation. Ego4D scores are pending. Dashes are unreported or unmeasured, different protocol; cross-protocol differences are not paired effects.}
not zero; intermediate scores are never interpolated. The available Omni
rows are exploratory and not all training-matched.}
\label{tab:main} \label{tab:main}
\centering\ninept \centering\ninept
\renewcommand{\arraystretch}{0.91} \renewcommand{\arraystretch}{0.91}
@@ -17,25 +15,24 @@ Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \
\midrule \midrule
\multicolumn{12}{@{}l}{\emph{EasyCom}} \\ \multicolumn{12}{@{}l}{\emph{EasyCom}} \\
EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\ EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\
Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\ Omni $\G$ & 73.45 & 68.64 & 64.65 & 61.51 & 59.35 & 58.08 & 57.22 & 56.15 & 55.12 & 54.32 & 60.85 \\
Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\ Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\
Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G+\A+\mathrm{SVA}$ & 82.36 & 76.26 & 70.89 & 66.67 & 63.73 & 61.52 & 60.10 & 58.63 & 57.35 & 56.20 & 65.37 \\
\midrule \midrule
\multicolumn{12}{@{}l}{\emph{Ego4D}} \\ \multicolumn{12}{@{}l}{\emph{Ego4D}} \\
EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ EgoSpeak GRU A+V & 76.4 & 73.0 & 70.4 & 68.5 & 67.1 & 66.3 & 65.6 & 65.2 & 64.7 & 64.4 & 68.2 \\
Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G$ & 74.72 & 72.03 & 69.81 & 68.11 & 66.81 & 65.88 & 65.21 & 64.73 & 64.35 & 63.98 & 67.56 \\
Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G+\A$ & 80.78 & 77.03 & 73.88 & 71.49 & 69.66 & 68.31 & 67.33 & 66.63 & 66.05 & 65.55 & 70.67 \\
Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G+\A+\mathrm{SVA}$ & 81.39 & 77.41 & 74.35 & 71.73 & 69.61 & 68.43 & 67.47 & 66.91 & 66.48 & 65.56 & 70.93 \\
\bottomrule \bottomrule
\end{tabular*} \end{tabular*}
\end{table*} \end{table*}
\begin{table}[t] \begin{table}[t]
\caption{EasyCom local-vision and information-audit screens (\%). The \caption{EasyCom vision screens and privileged activity audit (\%).
upper rows are exploratory encoder-based runs; the lower rows use Upper rows are exploratory encoder-feature runs; lower rows use
non-deployable past activity (three-seed mean $\pm$ SD). Their training transcript-derived past activity unavailable at deployment (three-seed
protocols differ, so cross-section differences are not matched effects. means). Cross-block differences are not matched model effects.}
Branch names are defined in Sec.~\ref{sec:method}.}
\label{tab:visual} \label{tab:visual}
\centering\ninept \centering\ninept
\renewcommand{\arraystretch}{0.92} \renewcommand{\arraystretch}{0.92}
@@ -46,45 +43,37 @@ Condition & mAP$_{.2}$ & Mean \\
\midrule \midrule
$\G+\A$ & 81.72 & 65.11 \\ $\G+\A$ & 81.72 & 65.11 \\
$\G+\A+$Pool & 82.03 & 65.40 \\ $\G+\A+$Pool & 82.03 & 65.40 \\
$\G+\A+$SVA & -- & -- \\ $\G+\A+$SVA & 82.36 & 65.37 \\
$\G+\A+$TSVA & 82.38 & 65.54 \\ $\G+\A+$TSVA & 82.38 & 65.54 \\
\midrule \midrule
Visibility only & $81.95{\pm}.10$ & $65.22{\pm}.05$ \\ Visibility only & 81.95 & 65.22 \\
Delayed 1 s & $81.95{\pm}.09$ & $65.34{\pm}.05$ \\ Delayed 1 s & 81.95 & 65.34 \\
Visible OR & $\mathbf{86.70{\pm}.09}$ & $\mathbf{66.95{\pm}.02}$ \\ Visible OR & $\mathbf{86.70}$ & $\mathbf{66.95}$ \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
\begin{table}[t] \begin{table}[t]
\caption{Pending matched timing/context tests. Each cell will compare \caption{EasyCom PTR context-cap screen: three-seed mean mAP (\%).
$\G$ with $\G+\A$ on common cutoffs; $\Delta$ is paired mean-mAP gain. The 8-s bank predates joint extraction; no matched $\G$ arm exists.}
PTR uses completed blocks plus the observed tail. Dashes are unmeasured.}
\label{tab:stream_context} \label{tab:stream_context}
\centering\ninept \centering\ninept
\renewcommand{\arraystretch}{0.92} \renewcommand{\arraystretch}{0.92}
\setlength{\tabcolsep}{3pt} \setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}llcc@{}} \begin{tabular}{@{}cc@{}}
\toprule \toprule
Group & Refresh / cap & $\G/\G+\A$ & $\Delta$ \\ PTR context cap (s) & $\G+\A$ mean mAP \\
\midrule \midrule
T & Completed hold, 8 s & --/-- & -- \\ 4 & 64.81 \\
T & PTR, 8 s & --/-- & -- \\ 8 & 65.18 \\
\midrule 12 & 64.74 \\
C & PTR, 2 s & --/-- & -- \\
C & PTR, 4 s & --/-- & -- \\
C & PTR, 6 s & --/-- & -- \\
C & PTR, 8 s & --/-- & -- \\
C & PTR, 10 s & --/-- & -- \\
C & PTR, 12 s & --/-- & -- \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
\begin{table}[t] \begin{table}[t]
\caption{Separate seed-0 global-only chunk screen (\%): 10 epochs, 8-s cap, \caption{Separate seed-0 global-only chunk screen (\%). PTR is not
2-FPS video. PTR is not extraction-matched to the other rows. Calls are extraction-matched; calls are processor invocations, not latency.}
processor invocations across 12 sessions, not latency.}
\label{tab:chunk} \label{tab:chunk}
\centering\ninept \centering\ninept
\renewcommand{\arraystretch}{0.95} \renewcommand{\arraystretch}{0.95}