wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure targeted comparison tables and method names

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 18:38:51 +08:00
parent bdb0397b75
commit 80f270a30a
2 changed files with 71 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/targeted/main.tex
+40 -28
View File
@@ -34,7 +34,7 @@ models and streaming Omni assistants address related problems, but the
cutoff-level availability of local evidence in an integrated Omni state is cutoff-level availability of local evidence in an integrated Omni state is
underexplored for this wearer-centered task. We examine two interfaces: underexplored for this wearer-centered task. We examine two interfaces:
short-window encoder evidence (SWE) supplies recent audio and optional short-window encoder evidence (SWE) supplies recent audio and optional
person-crop vision directly to a frozen-Omni forecasting head; partial-tail causal visual evidence directly to a frozen-Omni forecasting head; partial-tail
refresh (PTR) queries only the observed prefix of the current audiovisual refresh (PTR) queries only the observed prefix of the current audiovisual
block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio
screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
@@ -123,7 +123,7 @@ evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
suggest that global context and local audio are complementary; gains from suggest that global context and local audio are complementary; gains from
pooled vision and isolated PTR effects remain unproven. This study makes pooled vision and isolated PTR effects remain unproven. This study makes
three contributions: (1) short-window encoder evidence (SWE) supplies recent three contributions: (1) short-window encoder evidence (SWE) supplies recent
audio and optional person-crop vision to a frozen-Omni forecasting head; audio and optional visual encoder evidence to a frozen-Omni forecasting head;
(2) partial-tail refresh (PTR) queries the observed current-block prefix (2) partial-tail refresh (PTR) queries the observed current-block prefix
without committing it to persistent history; and (3) detailed EasyCom without committing it to persistent history; and (3) detailed EasyCom
analyses accompany a specified Ego4D transfer protocol. Matched PTR, analyses accompany a specified Ego4D transfer protocol. Matched PTR,
@@ -133,7 +133,7 @@ Ego4D scores, and event behavior remain pending.
\centering \centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context \caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
with recent audio and optional pooled person-crop evidence in the forecasting with recent audio and optional local visual evidence in the forecasting
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
the observed current-block tail at each 200-ms cutoff. The tail is not the observed current-block tail at each 200-ms cutoff. The tail is not
committed to persistent history.} committed to persistent history.}
@@ -212,30 +212,39 @@ pending, so PTR is a causal access rule rather than a demonstrated gain.
\subsection{Short-Window Feature Extraction} \subsection{Short-Window Feature Extraction}
We define every local observation by its prediction cutoff, before applying We define every local observation by its prediction cutoff, before applying
any encoder-specific preprocessing. The default audio branch encodes the any encoder-specific preprocessing. We denote the default Qwen-audio branch
by $\A$: it encodes the
mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni
audio encoder and averages valid output tokens. A wav2vec~2.0 convolutional audio encoder and averages valid output tokens. We denote by $\A_{\rm E}$
audio representation, as used by EgoSpeak \cite{kim2025egospeak}, is an the wav2vec~2.0 convolutional representation used by EgoSpeak
encoder-choice control; it must use the same one-second input interval and \cite{kim2025egospeak}. This encoder-choice control must use the same
one-second input interval and
forecasting head before differences can be attributed to the encoder. It is forecasting head before differences can be attributed to the encoder. It is
not the source of the reported $\G+\A$ result. not the source of the reported $\G+\A$ result.
For vision, a single-frame variant uses the latest frame strictly before We denote three visual branches by \emph{Pool}, \emph{SVA}, and \emph{TSVA}.
$t_n$; the pooled person-crop variant averages frozen Qwen vision tokens Pool averages frozen Qwen vision tokens
within provided boxes, across visible people, and over valid 2-FPS frames within provided boxes, across visible people, and over valid 2-FPS frames
in the preceding second. These pooled features summarize appearance but in the preceding second. These pooled features summarize appearance but
discard person identity and explicit motion. Boxes are dataset-provided, discard person identity and explicit motion. Boxes are dataset-provided,
not predictions from an evaluated online detector. Two no-box full-frame not predictions from an evaluated online detector. Two no-box full-frame
alternatives retain the post-merger spatial-token grid: F2 lets the alternatives retain the post-merger spatial-token grid. Spatial visual
global--audio query attend to the latest causal frame; F3 first encodes up attention (SVA) lets the global--audio query attend to the latest frame
strictly before $t_n$; temporal spatial visual attention (TSVA) first encodes up
to two observations at each matching grid location in chronological order to two observations at each matching grid location in chronological order
with a shared GRU, then applies the same query to the resulting grid. F3 with a shared GRU, then applies the same query to the resulting grid. TSVA
links image coordinates, not people or tracks. Its difference from F2 links image coordinates, not people or tracks. Its difference from SVA
confounds additional history and temporal capacity, so it cannot alone confounds additional history and temporal capacity, so it cannot alone
establish a motion or speaker-grounding mechanism. Empty or unavailable establish a motion or speaker-grounding mechanism. Empty or unavailable
local evidence is indicated by the validity masks, never filled with a local evidence is indicated by the validity masks, never filled with a
future observation. future observation.
The shorthand $\G$, $\G+\A$, and $\G+\A+$SVA identifies the three Omni
rows in the cross-dataset table; $\G+\A+$Pool, $\G+\A+$TSVA, and
$\G+\A_{\rm E}$ isolate visual or audio choices. Released EgoSpeak A+V
is an external baseline, not an encoder choice within our frozen-Omni head.
An unmeasured label specifies a comparison, not a result.
As a distinct, privileged short-window feature, we also derive visible As a distinct, privileged short-window feature, we also derive visible
non-wearer speech activity from transcript annotations \emph{before} each non-wearer speech activity from transcript annotations \emph{before} each
cutoff. Each visible participant contributes a two-second binary history cutoff. Each visible participant contributes a two-second binary history
@@ -253,7 +262,7 @@ or a performance upper bound.
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B: \caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
three-seed privileged past-activity audit, which is not deployable. Mean is three-seed privileged past-activity audit, which is not deployable. Mean is
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events. ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
F3 is a no-box, two-frame causal full-frame screen; it and the historical TSVA is a no-box, two-frame causal full-frame screen; it and the historical
deployable rows are exploratory seed-0 runs with unmatched head/training deployable rows are exploratory seed-0 runs with unmatched head/training
details. F1 has three seeds. Dashes are unavailable, not zero.} details. F1 has three seeds. Dashes are unavailable, not zero.}
\label{tab:main} \label{tab:main}
@@ -272,7 +281,7 @@ Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\ Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086 Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
& 65.400 & 26.019 & 45.371 \\\\ & 65.400 & 26.019 & 45.371 \\\\
No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\ No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\
Current-frame person set F1 & $81.777{\pm}0.156$ & -- Current-frame person set F1 & $81.777{\pm}0.156$ & --
& $64.951{\pm}0.095$ & -- & -- \\\\ & $64.951{\pm}0.095$ & -- & -- \\\\
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\ Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
@@ -425,12 +434,13 @@ threshold-free quantities average ten horizons. They are not one-event-one-
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding. vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
\textbf{Comparisons.} \textbf{Comparisons.}
We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni The cross-dataset main matrix contains released EgoSpeak A+V and the three
$\G$; $\G+\A$; and $\G+\A+$pooled-$\V$. A three-seed current-frame Omni variants defined in Sec.~\ref{sec:method}. Missing SVA and Ego4D
frame-set attention screen tests whether a richer person aggregation is cells remain blank until measured. EasyCom-only encoder choices serve as
already sufficient. Because that screen used Session~12 model selection diagnostic ablations rather than extra cross-dataset main rows. A three-seed
whereas the historical pooled row did not, their difference is diagnostic, current-frame person-set attention
not a matched causal attribution. screen used Session~12 model selection whereas the historical pooled row did
not; their difference is not a matched causal attribution.
\textbf{Pre-specified analyses.} \textbf{Pre-specified analyses.}
We freeze six control families: (L) capacity and recency, (T) tail We freeze six control families: (L) capacity and recency, (T) tail
@@ -444,10 +454,11 @@ one run. The source retains unmeasured hypotheses and falsifiers.
\subsection{EasyCom: Local Encoder Evidence} \subsection{EasyCom: Local Encoder Evidence}
Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate
0.2~s while integrated $\G$ is slightly stronger on the ten-horizon mean. audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and
Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over 60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon
$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26
over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
points, onset AP by only 1.47: dense gains do not imply equal improvement points, onset AP by only 1.47: dense gains do not imply equal improvement
at both activity boundaries. at both activity boundaries.
@@ -524,15 +535,16 @@ not real-time deployment. \\
\subsection{EasyCom: Visual Evidence and Ablations} \subsection{EasyCom: Visual Evidence and Ablations}
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and Table~\ref{tab:visual} separates visual screens from the cross-dataset main
matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when +0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
current-frame person-set model is stable across seeds but reaches lower mean current-frame person-set model is stable across seeds but reaches lower mean
mAP than the historical pooled row under a non-matched selection protocol. mAP than the historical pooled row under a non-matched selection protocol.
These results do not support a claim of learned speaker grounding. A separate These results do not support a claim of learned speaker grounding. A separate
seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\% seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\%
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a
shared GRU, while head capacity and training batch differ; raw boundary shared GRU, while head capacity and training batch differ; raw boundary
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
difference does not isolate visual motion or floor behavior. difference does not isolate visual motion or floor behavior.
manuscript/targeted/tables_compact.tex
+31 -14
View File
@@ -1,13 +1,11 @@
% Active result tables. A dash denotes an unreported or unmeasured value. % Active result tables. A dash denotes an unreported or unmeasured value.
\begin{table*}[t] \begin{table*}[t]
\caption{Dense future-activity mAP (\%) by prediction horizon. EasyCom \caption{Dense future-activity mAP (\%) at ten horizons on EasyCom and
scores use Sessions~1--3; the complete ten-horizon curves are reported for Ego4D. Each dataset has the same four-method comparison. EasyCom values
EgoSpeak, $\G+\A$, and F3. Other archived screens provide only 0.2-s and are measured on Sessions~1--3; $\G$ has only archived 0.2-s and mean
ten-horizon mean scores, so intermediate cells are not interpolated. scores, and SVA awaits evaluation. Ego4D scores are pending. Dashes are unreported or unmeasured,
Ego4D transfer is pending. Mean averages all ten horizons, including where not zero; intermediate scores are never interpolated. The available Omni
the individual values are unavailable. F3 and the fusion rows are exploratory rows are exploratory and not all training-matched.}
seed-0 runs; F1 reports a three-seed mean. These rows are not all
capacity- or training-matched.}
\label{tab:main} \label{tab:main}
\centering\ninept \centering\ninept
\renewcommand{\arraystretch}{0.91} \renewcommand{\arraystretch}{0.91}
@@ -19,20 +17,39 @@ Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \
\midrule \midrule
\multicolumn{12}{@{}l}{\emph{EasyCom (measured)}} \\ \multicolumn{12}{@{}l}{\emph{EasyCom (measured)}} \\
EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\ EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\
Recent $\A$ & 79.69 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.51 \\
Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\ Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\
$\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\ Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\
$\G+\A+$pooled-$\V$ & 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 65.40 \\ Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Full-frame F3 & 82.38 & 76.42 & 70.78 & 66.35 & 63.93 & 61.87 & 60.41 & 59.11 & 57.61 & 56.53 & 65.54 \\
Person-set F1 & 81.78 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 64.95 \\
\midrule \midrule
\multicolumn{12}{@{}l}{\emph{Ego4D (transfer pending)}} \\ \multicolumn{12}{@{}l}{\emph{Ego4D (transfer pending)}} \\
EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
$\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
\bottomrule \bottomrule
\end{tabular*} \end{tabular*}
\end{table*} \end{table*}
\begin{table}[t]
\caption{EasyCom visual-encoding screens (\%). Branch names are defined
in Sec.~\ref{sec:method}. Available rows are exploratory and not
capacity- or training-matched.}
\label{tab:visual}
\centering\ninept
\renewcommand{\arraystretch}{0.92}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{@{}lcc@{}}
\toprule
Visual branch on $\G+\A$ & mAP$_{.2}$ & Mean \\
\midrule
None & 81.72 & 65.11 \\
Pool & 82.03 & 65.40 \\
SVA & -- & -- \\
TSVA & 82.38 & 65.54 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t] \begin{table}[t]
\caption{EasyCom privileged past-activity audit (\%). Three-seed mean \caption{EasyCom privileged past-activity audit (\%). Three-seed mean
$\pm$ sample SD. The extra activity cues are not deployable inputs; W is $\pm$ sample SD. The extra activity cues are not deployable inputs; W is