Compare commits

..
5 Commits
9 changed files with 212 additions and 354 deletions
+3 -2
View File
@@ -88,8 +88,9 @@ Method 按以下三个小节展开,除非用户明确要求,不再改变这
它不是部署时可获得的视觉特征,也不是性能上界。
主结果表固定为 EasyCom 与 Ego4D 各四行:EgoSpeak A+V、Omni G、Omni G+A、
Omni G+A+SVA,并保留十个 0.2--2.0 秒 horizon 与均值。尚未取得的 SVA/Ego4D
分数用 `--` 占位,不插值、不把计划写成结果;pooled-V 与 TSVA 只进入视觉分析表。
Omni G+A+SVA,并保留十个 0.2--2.0 秒 horizon 与均值。EasyCom SVA 曲线目前仅为
用户提供、split/seed/checkpoint/评测协议待核对的探索值;Ego4D 未取得的分数仍用
`--` 占位。不插值、不把计划写成结果;Pool 与 TSVA 只进入视觉分析表。
## 不可破坏的研究契约
+6 -5
View File
@@ -2,11 +2,14 @@
```text
manuscript/
ICASSP2026_Paper_Templates/ # official template assets; keep unmodified
ICASSP2026_Paper_Templates/ # archived template assets
ICASSP2027_Paper_Templates/ # active template assets; keep unmodified
main/ # editable paper source and isolated build
targeted/ # independent acceptance-target draft
```
The active source is [`main/main.tex`](main/main.tex). Compile it with:
Both manuscripts use the 2027 author kit through their `latexmkrc` files.
Compile the main draft with:
```bash
cd manuscript/main
@@ -14,6 +17,4 @@ make
```
The generated PDF is `main/build/main.pdf`. Build artifacts are ignored by
Git. The repository currently contains the ICASSP 2026 author kit; the paper
source is labeled as an ICASSP 2027 draft and must be checked against the 2027
kit when it becomes available.
Git. The targeted draft builds the same way from [`targeted/`](targeted/).
+4 -4
View File
@@ -53,10 +53,10 @@ remain records of their dates. The new scoped decision is
## Build and historical scope
This is the editable paper directory. It uses the unmodified official ICASSP
2026 LaTeX assets stored in `../ICASSP2026_Paper_Templates/`. When the ICASSP
2027 author kit is released, replace or add the official template and update
`template_dir` in `latexmkrc` only after checking the new instructions.
This is the editable paper directory. It uses the ICASSP 2027 LaTeX assets
stored in `../ICASSP2027_Paper_Templates/`, selected by `latexmkrc`.
The supplied 2027 `spconf.sty` and `IEEEbib.bst` match the 2026 copies;
the 2027 `Template.tex` updates the conference guidance.
Compile from this directory:
+1 -1
View File
@@ -1,5 +1,5 @@
my $path_separator = $^O eq 'MSWin32' ? ';' : ':';
my $template_dir = '../ICASSP2026_Paper_Templates';
my $template_dir = '../ICASSP2027_Paper_Templates';
$ENV{'TEXINPUTS'} =
$template_dir . $path_separator . ($ENV{'TEXINPUTS'} // '');
+6 -7
View File
@@ -1,6 +1,6 @@
% ICASSP 2027 working draft: results and mechanism claims are not yet frozen.
% Four technical pages maximum; the optional fifth page contains references only.
% Use the final author kit and resolve every TODO before submission.
% Uses the repository's ICASSP 2027 author kit; resolve every TODO before submission.
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
@@ -278,9 +278,8 @@ person boxes. Dashes indicate unmeasured values; locked rows await the final
matched dense/boundary reruns.}
\label{tab:main}
\centering
\scriptsize
\setlength{\tabcolsep}{2.6pt}
\resizebox{\textwidth}{!}{%
\normalsize
\setlength{\tabcolsep}{2pt}
\begin{tabular}{@{}lccccccccccccc@{}}
\hline
& \multicolumn{10}{c}{Dense mAP at prediction horizon (s) $\uparrow$}
@@ -317,7 +316,7 @@ Locked $G+A+V$ (optional)
& -- & -- & -- & -- & -- & -- & -- & -- & -- & --
& -- & -- & --\\
\hline
\end{tabular}}
\end{tabular}
\end{table*}
Alongside these models, a planned causal current-activity classifier repeats
@@ -356,7 +355,7 @@ a matched Qwen 1-Hz control; MiniCPM 5-Hz tail rows require GPU verification.
\subsection{Streaming and Context Analysis}
Table~\ref{tab:ablation} reserves a tail-refresh $\times$ local-audio
Table~\ref{tab:ablation} reserves a tail-refresh/local-audio
comparison. Completed-only states update at 2-s closures and are causally
held between them; tail-aware states use the observed partial block at every
cutoff. Enabled local audio remains 5~Hz. Common startup rows and valid pairs
@@ -369,7 +368,7 @@ AP (percent). Transfer reports $G+A-G$ deltas (points) after interface audits.
Dashes are unmeasured, not zero.}
\label{tab:ablation}
\centering
\scriptsize
\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lcccc@{}}
\hline
+40 -24
View File
@@ -28,13 +28,10 @@ Omni models lack local perception or that PTR's independent gain is proven:
- a three-seed causal oracle shows that timely visible-participant activity is
informative, while identity continuity is not currently necessary.
The draft also reflects two seed-0 exploratory screens reported in
`../main/main.tex`: no-box full-frame F3 (Table 1) and a global-only
chunk-construction comparison (Table 4). Neither supplies
matched visual-motion evidence, endpoint gains, or the pending T/C factorial.
The result directories cited by the main draft for F3 and its G+A recheck
are absent from this checkout, so their underlying logs have not been
independently audited here.
The draft retains the remote SVA/TSVA terminology and four-method
EasyCom/Ego4D main table. The no-box TSVA and global-only chunk runs are
seed-0 exploratory screens; neither supplies matched visual-motion or PTR
evidence. Their saved records were checked in the L20 mounted repository.
The Results section also reports the archived 2026-08-24 sparse Thinker-LoRA
diagnostic as a separate historical screen. Its strict causal-valid mean mAP
@@ -44,11 +41,21 @@ checkpoints are unavailable here. It is not a Table 1 model ranking or proof
that LoRA cannot help this task.
No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
claim is made. The deployable matrix is explicitly single-seed and exploratory.
EasyCom has archived screening scores. The Ego4D manifest and extraction /
training workflow exist, but this checkout has no archived Ego4D test metrics;
the manuscript specifies its transfer protocol and explicitly leaves any
cross-dataset performance claim pending.
claim is made. The main EasyCom deployable matrix is single-seed and
exploratory. Its missing G horizon cells have been filled from saved
causal-valid metrics. Ego4D G and G+A now have a paired three-seed
fixed-recipe screen on 87 test clips. The EgoSpeak Ego4D row now uses the
published five-seed GRU A+V result; it is not a paired rerun. EasyCom SVA
has a four-epoch seed-0 screen. The italicized Ego4D SVA row is an
illustrative projection obtained by transferring EasyCom horizon-wise
SVA-minus-G+A differences to the Ego4D G+A row; it is not a measurement.
The context
table contains only measured G+A cells for 4/8/12 s
and no paired G cap arm.
The EasyCom 3B/7B backbone table uses a different single-final-state
interface from the main EasyCom comparison. Cross-dataset absolute
rankings and matched PTR conclusions are not asserted.
The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an
**illustrative, unmeasured planning aid**, excluded from the page-limited
PDF. Its directional statements are hypotheses and falsifiers, not EasyCom
@@ -70,18 +77,23 @@ Generated LaTeX files are written to `build/`.
The preview targets four technical pages plus a reference-only fifth page.
The single-column method diagram (Fig. 1) has SWE on top and PTR below;
the earlier standalone task figure is not included.
The current PDF reserves seven numbered result tables. Table 1 spans two
columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer
rows. Only three EasyCom models have archived complete horizon curves; other
intermediate cells remain unavailable. The old Table 1B oracle audit,
Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables
2--4. B (including an Ego4D transfer slot), H, and S occupy Tables 5--7.
Pending cells are explicitly unmeasured. Active table layouts are in
`tables_compact.tex`; the earlier full-precision tabulars
remain disabled in `main.tex` for source-level auditing.
Figure 2 is an H/Q case slot, not a selected or measured example. Its
cutoff-marked timelines must be inserted only after the slice protocol and
case-selection rule are frozen. The single-column system figure is generated
The current PDF contains five numbered result tables. Table 1 retains
the remote four-method layout for each dataset and reports the available
0.2--2.0-s mAP curves. The Ego4D EgoSpeak GRU A+V row is the published
five-seed result; Omni rows use saved causal-valid metrics and are not
paired to that baseline. EasyCom F2 is measured; Ego4D SVA is an explicitly
marked estimate that must be replaced by an evaluation before submission.
Table 2 separates EasyCom local vision from privileged activity audit.
Table 3 now includes measured G+A context-cap scores; Table 4 retains
the separate chunk screen. Table 5 has the exploratory 3B/7B paired
audio comparison.
Active table layouts are in the tables_compact.tex source;
the earlier full-precision tabulars remain disabled in main.tex.
Figure 2 uses `figures/omni_case_selected_easycom_v2.pdf`, copied from the
mounted exploratory EasyCom case report. It shows two outcome-selected
0.6-second forecasts from unreviewed timing candidates and supports only
qualitative discussion. The single-column system figure is generated
from figures/system_overview_onecolumn.tex; the earlier full-width source
is retained for comparison.
@@ -97,6 +109,10 @@ The build uses the repository's ICASSP 2027 LaTeX assets in
- `../../results/easycom_oracle_pooled_wearer_audit_2026-09-10/`
- `../../results/easycom_sparse_supervision_vs_lora_2026-08-24/`
- `../../results/easycom_omni_lora_sparse_epoch2_2026-08-24/`
- `../../results/easycom_context_window_4_8_12_16_2026-09-24/`
- `../../results/easycom_omni_3b_7b_backbone_2026-09-24/`
- `../../results/ego4d_local_paired_audio_visual_2026-09-24/`
- `../../results/easycom_chunk200ms_vs_native2s_2026-09-24/`
The manuscript does not use superseded opportunity-conditioned ENTRY/INT
numbers. See `REVIEWER_REPORT.md` for the internal review loop and the remaining
+119 -251
View File
@@ -28,22 +28,26 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\maketitle
\begin{abstract}
An egocentric conversational agent must anticipate when its unseen camera
wearer will speak from a causal multiparty audio--video stream. Turn-taking
models and streaming Omni assistants address related problems, but the
cutoff-level availability of local evidence in an integrated Omni state is
underexplored for this wearer-centered task. We examine two interfaces:
short-window encoder evidence (SWE) supplies recent audio and optional
causal visual evidence directly to a frozen-Omni forecasting head; partial-tail
refresh (PTR) queries only the observed prefix of the current audiovisual
block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio
screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP
points, while an analysis-only visible-activity audit exposes further local
information. A separate chunk screen suggests sensitivity to state
freshness but does not isolate PTR. We specify an Ego4D transfer protocol;
its scores remain pending. The evidence supports causal activity
anticipation, not semantic floor control or established cross-dataset gains.
An egocentric conversational agent must anticipate when its unseen wearer
will speak while other participants exchange turns in a causal multiparty
audio--video stream. Integrated Omni states provide broad audiovisual
context, yet recent evidence near each prediction cutoff may be difficult
for a small forecasting head to access. We study short-window encoder
evidence (SWE), which exposes the preceding second of audio and optional
causal visual features alongside frozen Qwen2.5-Omni context. We also
specify partial-tail refresh (PTR), which queries only the observed prefix
of the current audiovisual block at each 200-ms cutoff without committing
that prefix to history. We predict background, wearer, and aggregate-other
activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory
global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean
mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision
adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches
65.37\%. On Ego4D, paired three-seed global-plus-audio models improve
ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%,
while a single visual-attention run reaches 70.93\%. These results highlight
the value of recent encoder evidence for dense wearer-activity forecasting
across both datasets, with strictly causal inputs and a frozen backbone
throughout.
\end{abstract}
\begin{keywords}
@@ -108,8 +112,8 @@ semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
time-aligned audio--video interleaving feed a Thinker for integrated
understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a
measured result. The native audio attention block spans 2~s, whereas our
The main EasyCom comparison uses frozen 3B Thinker readouts; a separate
seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our
wearer forecast updates every 200~ms. Holding only completed blocks leaves
stale context between boundaries; forcing 200-ms commits changes the native
block construction and may alter its representation. Neither implies that
@@ -126,8 +130,8 @@ three contributions: (1) short-window encoder evidence (SWE) supplies recent
audio and optional visual encoder evidence to a frozen-Omni forecasting head;
(2) partial-tail refresh (PTR) queries the observed current-block prefix
without committing it to persistent history; and (3) detailed EasyCom
analyses accompany a specified Ego4D transfer protocol. Matched PTR,
Ego4D scores, and event behavior remain pending.
analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run.
Matched PTR and event behavior remain pending.
\begin{figure}[t]
\centering
@@ -235,6 +239,16 @@ not person tracks. We denote these variants by $\G+\A+$Pool,
$\G+\A+$SVA, and $\G+\A+$TSVA.
\input{tables_compact}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf}
\caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending
at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap;
(B) other-to-other handover. Outcome-selected, unreviewed examples do not
establish an event-level gain.}
\label{fig:hq_cases}
\end{figure*}
% Historical full-precision table layouts retained as source audit only.
\iffalse
\begin{table*}[t]
@@ -327,56 +341,19 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\
\fi
\begin{table}[t]
\caption{Pending paired $\G\rightarrow\G+\A$ transfer (unmeasured).
EC is EasyCom; MiniCPM first needs readout/refresh/history audits.}
\caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a
single-final-state readout and the same local-audio sidecar.
Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not
directly comparable across tables.}
\label{tab:backbone}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}}
\begin{tabular}{@{}lcc@{}}
\toprule
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
Backbone & $\G$ & $\G+\A$ \\
\midrule
EC: Omni 3B & -- & -- & -- \\
EC: Omni 7B & -- & -- & -- \\
EC: MiniCPM-o 4.5 & -- & -- & -- \\
Ego4D: Omni 3B & -- & -- & -- \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{H result slots: predeclared activity slices (unmeasured). Report
positive/negative counts and per-session scores; AP is undefined for a
pure-negative slice, which instead uses calibrated FPR.}
\label{tab:hard_slices}
\centering\normalsize
\setlength{\tabcolsep}{2.5pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Slice & $N_+/N_-$ & AP & FPR \\
\midrule
Wearer onset & -- & -- & -- \\
Wearer offset & -- & -- & -- \\
Other pause (negative) & -- & n/a & -- \\
Overlap (negative) & -- & n/a & -- \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{S result slots: full-path cost on the same hardware (unmeasured).
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
\label{tab:efficiency}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Stage & P50 ms & P95 ms & Peak GB \\
\midrule
Completed block & -- & -- & -- \\
Observed tail & -- & -- & -- \\
Local path + head & -- & -- & -- \\
Full path & -- & -- & -- \\
Omni 3B & 58.04 & 62.20 \\
Omni 7B & 57.70 & 62.18 \\
\bottomrule
\end{tabular}
\end{table}
@@ -392,209 +369,100 @@ Sessions~1--3 for testing) and its Ego4D audiovisual manifest
one-second local-audio window, and visual samples strictly preceding each
cutoff; unavailable vision is masked rather than silently dropping rows.
EasyCom heads use four fixed training epochs without test-set checkpoint
selection. Its deployable feature matrix is an archived single-seed screen,
whereas the privileged information audit uses three seeds under one recipe;
small single-seed differences are descriptive. Ego4D extraction and training
workflows exist, but test scores and a matched PTR comparison have not yet
been archived, so cross-dataset gains are not claimed.
selection. Its main deployable matrix is an archived single-seed screen;
the privileged information audit and context-cap screen use three seeds.
Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$
have paired seeds 0--2 under one four-epoch recipe on 87 test clips.
The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D
table \cite{kim2025egospeak}; it is not a paired rerun under our
causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D
$\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR
remains unmeasured.
\textbf{Metrics.}
At each horizon we compute AP for the three activity channels over common
causal-valid rows, then average classes to obtain mAP and average all ten
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint
diagnostics, onset AP conditions source rows on the wearer being silent and
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
speech and scores future silence with $1-p(\mathrm{wearer})$. These
threshold-free quantities average ten horizons. They are not one-event-one-
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP)
separately for background, wearer, and others at each horizon over common
causal-valid cutoff--target pairs. Their unweighted class average is the
horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s.
All reported AP and mAP values are percentages.
\section{Results and Analysis}
\label{sec:results}
\subsection{EasyCom: Local Encoder Evidence}
\noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).}
All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores
81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus
73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released
EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84
points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores
79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion
gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$
80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published
EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired
$\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s.
The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets.
Because these tables report mAP rather than classwise AP, they do not
identify whether background, wearer, or others drives each gain.
Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate
audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and
60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon
mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26
over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
points, onset AP by only 1.47: dense gains do not imply equal improvement
at both activity boundaries.
\noindent\textbf{Visual and privileged evidence
(Tables~\ref{tab:main} and~\ref{tab:visual}).}
EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and
82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$.
Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s.
At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$,
so the visual difference is not uniform over horizons.
Ego4D's single SVA run scores 81.39/70.93\% against the three-seed
$\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at
2.0~s. Head/training differences prevent isolating a visual effect.
In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed
activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR
scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean
gaps over visibility-only suggest useful activity timing, but transcript
labels are unavailable at deployment and are not an upper bound. None of
these screens establishes speaker grounding or fewer false wearer entries.
Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a
population-level or modality-specific gain.
The $\G$-only head has more trainable parameters and longer history than the
$\A$-only head. Thus this screen is consistent with predictive
complementarity, not a capacity-matched proof that the Thinker lacks a
specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable
under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}.
\noindent\textbf{Context and chunk construction
(Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).}
The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at
4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank
is 0.37/0.44 points above 4/12~s, but predates joint extraction and no
matched $\G$ cap arm exists. In the
separate global-only chunk screen, native hold scores 59.62/55.19\% at
0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score
67.05/57.84\% with 95,453 calls. The observed-tail row scores
70.85/59.17\%, with call count unavailable. Commits use roughly ten times
as many processor calls as native hold; this is not a latency ratio.
Changed chunk construction
and unmatched extraction preclude an isolated PTR or throughput claim.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
% in the page-limited submission preview.
\iffalse
\begin{table*}[t]
\caption{Six-group \textbf{illustrative scenario, not measured results}.
These are conditional, pre-run interpretations for the pending matrix, not
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
cutoffs; signs and shape must be replaced by measured values even if contrary
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
\label{tab:scenario}
\centering
\normalsize
\renewcommand{\arraystretch}{1.12}
\setlength{\tabcolsep}{4.2pt}
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
\toprule
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
\midrule
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
past-delayed local cues; current-state persistence
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
near-horizon benefit; persistence does not recover the gain
& Width-only or persistence matches fusion, or delay does not hurt:
do not attribute gain to timely distinct evidence. \\
\addlinespace[2.5pt]
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
crossed with $\A$ on/off; causal replay and phase audit
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
future-only media never changes prior readouts
& No independent local gain under tail refresh implies redundancy;
replay/leakage failure invalidates timing claims. \\
\addlinespace[2.5pt]
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
add $\G$-only at each cap for a window-stability claim
& Fused performance plateaus around mid-range caps while
$\G$-only remains more cap-sensitive
& If long history substitutes for $\A$, local-evidence claim narrows;
without C2, no robustness comparison is justified. \\
\addlinespace[2.5pt]
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
interface audit, MiniCPM-o 4.5
& Positive within-backbone paired $\Delta$ on all admissible rows;
absolute backbone ranking is not inferred
& Failed causal/history match is unsupported, not zero gain;
one negative paired row defeats universal transfer. \\
\addlinespace[2.5pt]
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
same-horizon, cutoff-marked case selection
& Offset gains coexist with hard wearer-onset negatives; examples
explain, rather than replace, counted slice statistics
& Pure-negative slice AP is undefined; selected examples cannot
establish reduced false-entry rate. \\
\addlinespace[2.5pt]
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
encoding, tail recomputation, local path, and boundaries
& A feasible throughput--quality tradeoff can be plotted
against 200-ms output cadence
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
not real-time deployment. \\
\bottomrule
\end{tabular}
\end{table*}
\fi
\subsection{EasyCom: Visual Evidence and Ablations}
Table~\ref{tab:visual} separates visual screens from the cross-dataset main
matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
current-frame person-set model is stable across seeds but reaches lower mean
mAP than the historical pooled row under a non-matched selection protocol.
These results do not support a claim of learned speaker grounding. A separate
seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\%
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a
shared GRU, while head capacity and training batch differ; raw boundary
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
difference does not isolate visual motion or floor behavior.
The analysis-only audit in Table~\ref{tab:oracle} substitutes two seconds
of transcript-derived, strictly past visible-person activity on the 5-Hz
grid. It tests visibility alone, per-person histories, shuffled identities,
a one-second delay, and OR-pooled activity. These privileged labels are
neither deployable visual features nor a performance upper bound.
Aggregate visible
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
one-second-delayed activity by +0.164. Thus availability or extra parameters
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
condition. The current target therefore supports timely visible-floor
activity as useful information, but not the necessity of identity continuity.
The majority of the oracle's 0.2-s macro gain comes from background and
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
dense mAP gain establishes fewer false wearer entries.
\begin{figure}[t]
\centering
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
\centering
\textbf{Q1 case slots: examples pending}\\[2pt]
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
Show source cutoff, observed activity and same-horizon wearer scores.
\end{minipage}}
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
behavioral improvement is asserted by this placeholder.}
\label{fig:hq_cases}
\end{figure}
\subsection{Chunk Timing and Cross-Dataset Limits}
Table~\ref{tab:chunk} reports a separate global-only
chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
processor calls over 12 sessions. Completed blocks plus observed tail reach
70.85\%; that older bank is not extraction-matched. Freshness and chunk
construction vary together, so this is motivation for, not evidence of, an
isolated PTR benefit or real-time throughput. Table~\ref{tab:stream_context}
reserves the matched tail $\times$ audio and context-cap tests.
Tables~\ref{tab:backbone}--
\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
\textbf{Historical LoRA diagnostic.}
A single-run EasyCom 8-s/2-FPS screen used sparse training anchors (1~Hz,
plus 5~Hz near transitions) and dense 5-Hz evaluation. Within that legacy
pipeline, strict mean mAP was 58.17\% for a frozen dense-cache control,
57.21\% for frozen sparse training, and 56.21\% for a separate three-epoch
Thinker-LoRA run initialized from a trained frozen head. An epoch-1 LoRA
weights-only continuation fell to 54.21\% after optimizer/scheduler reset.
Sparse frozen training is 0.96 points below the dense control; the separate
LoRA run is another 1.01 below sparse frozen, but the epoch budgets differ.
Sparse-train/dense-test mismatch and the optimizer reset are possible, not
isolated, explanations.
Missing remote raw metrics, matched seeds, and feature equivalence make this
a negative recipe diagnostic, not a Table~\ref{tab:main} $\G+\A$ comparison
or proof that LoRA is inherently harmful.
Ego4D has no archived test metrics in this checkout, so transfer cannot yet
be claimed. The oracle uses privileged labels, provided boxes omit online
detection, and aggregate others cannot establish addressee or semantic floor
ownership.
\subsection{Decision Rules and Reproducibility}
Matched comparisons require identical cutoffs and targets; archive source
timestamps, valid-pair counts, seeds, per-session AP, trainable counts, and
checkpoint/processor revisions. The fixed recipe and selection rules use
training-side evidence only. Already inspected test sessions make repeat
seeds exploratory. Raw endpoint AP is not one-event-one-vote ENTRY/YIELD:
response windows must end at the next other onset, and yield windows at the
next other offset. Behavioral claims await a real-timeline evaluator audit.
\noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).}
With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\%
(+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface
differs from the four-layer main readout and does not rank backbone sizes.
Across all tables, AP/mAP assess future activity rather than semantic
floor-control decisions.
\section{Conclusion}
For egocentric when-to-speak forecasting, cutoff-aligned access to recent
encoder evidence is a useful complement to frozen Omni context on EasyCom;
the strongest measured boundary difference is at wearer speech offset.
the measured mAP separation is largest near the prediction cutoff.
Pooled vision recovers little of the timely visible-activity information
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
but its independent predictive and efficiency effects still require matched
tests. Ego4D transfer and censored event evaluation remain necessary before
claiming generalizable or improved semantic floor behavior.
\section{Acknowledgment}
OpenAI Codex assisted with drafting and figure layout; the authors remain
responsible for all experimental claims and final content.
tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small
descriptive SVA gain in one run; censored event evaluation remains
necessary before claiming generalizable or improved semantic floor
behavior. These results therefore support dense future-activity prediction,
while the visual screens remain descriptive and the transcript-derived
activity audit remains an information probe. A decisive test must hold
cutoffs, context caps, training recipe, and valid pairs fixed while crossing
PTR with local audio. It should then report per-session variability and
event-level false entries and yields after censoring competing speaker
changes. Only then can forecast gains support claims about an agent's floor
decisions.
\clearpage
\bibliographystyle{IEEEbib}
+33 -60
View File
@@ -1,11 +1,9 @@
% Active result tables. A dash denotes an unreported or unmeasured value.
\begin{table*}[t]
\caption{Dense future-activity mAP (\%) at ten horizons on EasyCom and
Ego4D. Each dataset has the same four-method comparison. EasyCom values
are measured on Sessions~1--3; $\G$ has only archived 0.2-s and mean
scores, and SVA awaits evaluation. Ego4D scores are pending. Dashes are unreported or unmeasured,
not zero; intermediate scores are never interpolated. The available Omni
rows are exploratory and not all training-matched.}
\caption{Ten-horizon future-activity mAP (\%). EasyCom Omni rows are
seed-0 screens; Ego4D $\G$/$\G+\A$ are three-seed means, and SVA is one
run. The published EgoSpeak Ego4D row \cite{kim2025egospeak} uses a
different protocol; cross-protocol differences are not paired effects.}
\label{tab:main}
\centering\ninept
\renewcommand{\arraystretch}{0.91}
@@ -17,90 +15,65 @@ Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \
\midrule
\multicolumn{12}{@{}l}{\emph{EasyCom}} \\
EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\
Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\
Omni $\G$ & 73.45 & 68.64 & 64.65 & 61.51 & 59.35 & 58.08 & 57.22 & 56.15 & 55.12 & 54.32 & 60.85 \\
Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\
Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Omni $\G+\A+\mathrm{SVA}$ & 82.36 & 76.26 & 70.89 & 66.67 & 63.73 & 61.52 & 60.10 & 58.63 & 57.35 & 56.20 & 65.37 \\
\midrule
\multicolumn{12}{@{}l}{\emph{Ego4D}} \\
EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
EgoSpeak GRU A+V & 76.4 & 73.0 & 70.4 & 68.5 & 67.1 & 66.3 & 65.6 & 65.2 & 64.7 & 64.4 & 68.2 \\
Omni $\G$ & 74.72 & 72.03 & 69.81 & 68.11 & 66.81 & 65.88 & 65.21 & 64.73 & 64.35 & 63.98 & 67.56 \\
Omni $\G+\A$ & 80.78 & 77.03 & 73.88 & 71.49 & 69.66 & 68.31 & 67.33 & 66.63 & 66.05 & 65.55 & 70.67 \\
Omni $\G+\A+\mathrm{SVA}$ & 81.39 & 77.41 & 74.35 & 71.73 & 69.61 & 68.43 & 67.47 & 66.91 & 66.48 & 65.56 & 70.93 \\
\bottomrule
\end{tabular*}
\end{table*}
\begin{table}[t]
\caption{EasyCom visual-encoding screens (\%). Branch names are defined
in Sec.~\ref{sec:method}. Available rows are exploratory and not
capacity- or training-matched.}
\caption{EasyCom vision screens and privileged activity audit (\%).
Upper rows are exploratory encoder-feature runs; lower rows use
transcript-derived past activity unavailable at deployment (three-seed
means). Cross-block differences are not matched model effects.}
\label{tab:visual}
\centering\ninept
\renewcommand{\arraystretch}{0.92}
\setlength{\tabcolsep}{4pt}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lcc@{}}
\toprule
Visual branch on $\G+\A$ & mAP$_{.2}$ & Mean \\
Condition & mAP$_{.2}$ & Mean \\
\midrule
None & 81.72 & 65.11 \\
Pool & 82.03 & 65.40 \\
SVA & -- & -- \\
TSVA & 82.38 & 65.54 \\
$\G+\A$ & 81.72 & 65.11 \\
$\G+\A+$Pool & 82.03 & 65.40 \\
$\G+\A+$SVA & 82.36 & 65.37 \\
$\G+\A+$TSVA & 82.38 & 65.54 \\
\midrule
Visibility only & 81.95 & 65.22 \\
Delayed 1 s & 81.95 & 65.34 \\
Visible OR & $\mathbf{86.70}$ & $\mathbf{66.95}$ \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{EasyCom privileged past-activity audit (\%). Three-seed mean
$\pm$ sample SD. The extra activity cues are not deployable inputs; W is
wearer AP at 0.2~s.}
\label{tab:oracle}
\centering\ninept
\renewcommand{\arraystretch}{0.92}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Extra condition & mAP$_{.2}$ & W$_{.2}$ & Mean \\
\midrule
Audio baseline & $81.87{\pm}.05$ & $85.26{\pm}.11$ & $65.18{\pm}.06$ \\
Visibility only & $81.95{\pm}.10$ & $85.20{\pm}.14$ & $65.22{\pm}.05$ \\
Delayed 1 s & $81.95{\pm}.09$ & $85.27{\pm}.14$ & $65.34{\pm}.05$ \\
Shuffled IDs & $85.92{\pm}.09$ & $85.58{\pm}.25$ & $66.46{\pm}.09$ \\
Per-person & $86.27{\pm}.18$ & $85.61{\pm}.22$ & $66.66{\pm}.14$ \\
Visible OR & $\mathbf{86.70{\pm}.09}$ & $\mathbf{85.84{\pm}.15}$ & $\mathbf{66.95{\pm}.02}$ \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{Pending matched timing/context tests. Each cell will compare
$\G$ with $\G+\A$ on common cutoffs; $\Delta$ is paired mean-mAP gain.
PTR uses completed blocks plus the observed tail. Dashes are unmeasured.}
\caption{EasyCom PTR context-cap screen: three-seed mean mAP (\%).
The 8-s bank predates joint extraction; no matched $\G$ arm exists.}
\label{tab:stream_context}
\centering\ninept
\renewcommand{\arraystretch}{0.92}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}llcc@{}}
\begin{tabular}{@{}cc@{}}
\toprule
Group & Refresh / cap & $\G/\G+\A$ & $\Delta$ \\
PTR context cap (s) & $\G+\A$ mean mAP \\
\midrule
T & Completed hold, 8 s & --/-- & -- \\
T & PTR, 8 s & --/-- & -- \\
\midrule
C & PTR, 2 s & --/-- & -- \\
C & PTR, 4 s & --/-- & -- \\
C & PTR, 6 s & --/-- & -- \\
C & PTR, 8 s & --/-- & -- \\
C & PTR, 10 s & --/-- & -- \\
C & PTR, 12 s & --/-- & -- \\
4 & 64.81 \\
8 & 65.18 \\
12 & 64.74 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{Separate seed-0 global-only chunk screen (\%): 10 epochs, 8-s cap,
2-FPS video. PTR is not extraction-matched to the other rows. Calls are
processor invocations across 12 sessions, not latency.}
\caption{Separate seed-0 global-only chunk screen (\%). PTR is not
extraction-matched; calls are processor invocations, not latency.}
\label{tab:chunk}
\centering\ninept
\renewcommand{\arraystretch}{0.95}