wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Polish targeted manuscript framing and attribution

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 15:44:25 +08:00
parent e09ee1c432
commit 15f1b779b3
2 changed files with 97 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/targeted/README.md
+4
View File
@@ -15,6 +15,10 @@ interaction, then studies two cutoff-aligned interfaces:
- **partial-tail refresh (PTR)**: query only the observed prefix of the current - **partial-tail refresh (PTR)**: query only the observed prefix of the current
two-second block at each 200-ms cutoff, reusing completed payloads. two-second block at each 200-ms cutoff, reusing completed payloads.
SWE names the short temporal support of encoder evidence, not a shallow
network layer. The two-second chunk refers to this Qwen extraction setup,
not a universal Omni-model setting.
The evidence remains a causal global--local **audit**, not a claim that all The evidence remains a causal global--local **audit**, not a claim that all
Omni models lack local perception or that PTR's independent gain is proven: Omni models lack local perception or that PTR's independent gain is proven:
manuscript/targeted/main.tex
+89 -78
View File
@@ -19,30 +19,31 @@
\renewcommand{\dblfloatpagefraction}{0.75} \renewcommand{\dblfloatpagefraction}{0.75}
\title{Egocentric Speech Anticipation with Omni Models:\\ \title{Egocentric Speech Anticipation with Omni Models:\\
Short-Window Evidence and Partial-Tail Refresh} Short-Window Evidence and Partial-Tail Refresh}
\name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\ \name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}} Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom} \address{China Telecom Corporation Limited Shanghai Branch}
\begin{document} \begin{document}
\ninept \ninept
\maketitle \maketitle
\begin{abstract} \begin{abstract}
An egocentric conversational agent must anticipate when its invisible wearer An egocentric conversational agent must anticipate when its unseen camera
will speak while observing a causal multiparty audio--video stream. Although wearer will speak from a causal multiparty audio--video stream. Turn-taking
turn-taking prediction and omni-modal assistants have advanced, the temporal models and streaming Omni assistants address related problems, but the
accessibility of local evidence in an integrated Omni state remains unclear. cutoff-level availability of local evidence in an integrated Omni state is
We study two interfaces: a short-window encoder-evidence path that exposes underexplored for this wearer-centered task. We examine two interfaces:
recent audio and optional person-crop vision to a frozen-Omni forecasting short-window encoder evidence (SWE) supplies recent audio and optional
head, and partial-tail refresh that queries the observed prefix of an person-crop vision directly to a frozen-Omni forecasting head; partial-tail
incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the refresh (PTR) queries only the observed prefix of the current audiovisual
exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\% block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio
ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
vision adds only 0.29 mean-mAP points, whereas an analysis-only causal 74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP
visible-activity oracle reveals additional information. A separate chunk points, while an analysis-only visible-activity audit exposes further local
screen suggests sensitivity to state freshness, but does not isolate the information. A separate chunk screen suggests sensitivity to state
partial-tail effect. These results motivate finer local evidence access; freshness but does not isolate PTR. We specify an Ego4D transfer protocol;
they do not establish semantic floor control or cross-dataset transfer. its scores remain pending. The evidence supports causal activity
anticipation, not semantic floor control or established cross-dataset gains.
\end{abstract} \end{abstract}
\begin{keywords} \begin{keywords}
@@ -53,16 +54,17 @@ streaming, voice activity projection, conversational agents
\section{Introduction} \section{Introduction}
\label{sec:intro} \label{sec:intro}
An egocentric conversational agent must decide \emph{when} to speak while its An egocentric conversational agent must decide \emph{when} to speak even
camera wearer is usually invisible. It sees other participants and hears a though its camera wearer is usually invisible. It hears a mixed conversation
mixed conversation, but must predict the wearer's own future speech before a and sees other participants, but must anticipate its own speech from the
turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models observed stream before a turn boundary is known (Fig.~\ref{fig:task}).
and voice-activity projection (VAP) show the value of anticipatory rather than Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap, extensions, backchannel prediction, online end-of-turn detection, and event
elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap, benchmarks address increasingly realistic timing decisions
jiang2026turnbench}. Yet these tasks differ in sensor placement, visible \cite{ekstedt2022vap,russell2025mmvap,elmers2025triadic,
speakers, dialogue roles, and whether the target is a future activity state lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
or a semantic intervention. Their input viewpoints and target decisions, however, are not interchangeable
with forecasting the future activity of an unseen wearer.
\begin{figure}[H] \begin{figure}[H]
\centering \centering
@@ -75,40 +77,42 @@ not semantic floor ownership.}
\end{figure} \end{figure}
\textbf{Egocentric audiovisual conversation.} \textbf{Egocentric audiovisual conversation.}
EgoSpeak forecasts wearer speech from first-person RGB and audio using EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
specialized encoders and a temporal fusion model \cite{kim2025egospeak}; RGB features with specialized encoders, fuses their temporal evidence, and
EasyCom and Ego4D supply complementary egocentric conversation settings forecasts wearer speech on EasyCom and Ego4D \cite{kim2025egospeak,
\cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker donley2021easycom,grauman2022ego4d}. Other first-person work addresses
anticipation, egocentric speaking/listening graphs, and addressee detection multi-view next-speaker prediction, speaking/listening graphs, or addressee
study related but different targets \cite{serj2024threem,jia2024avconv, detection \cite{serj2024threem,jia2024avconv,kong2024sicnet}.
kong2024sicnet}. Egocentric active-speaker localization and detection Egocentric active-speaker methods \cite{jiang2022asl,huh2025egoasd,
\cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person clarke2025asd} and long--short person modeling \cite{wang2024loconet}
modeling \cite{wang2024loconet} estimate who is speaking now; they do not by recover present speaker evidence; audiovisual cues can also help turn
themselves forecast an unseen wearer's future activity. Visual cues can help prediction in noise \cite{oconnorrussell2025noise}. These studies make a
turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled broad ``first audiovisual turn-taking'' claim untenable. What remains
vision conveys the relevant local floor evidence here is empirical. less examined is how a single causal first-person stream should expose
recent audio--visual evidence to forecast the invisible wearer's activity.
\textbf{Omni and local evidence.} \textbf{Streaming Omni and temporal access.}
Omni systems integrate modalities and streaming interaction Omni systems integrate modalities and support streaming interaction
\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug, \cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side
audio--visual adaptation offers another route to accessible modality evidence audio--visual adaptation offers another route to accessible modality evidence
\cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither \cite{lin2023lavish,rouditchenko2024whisperflamingo}. Qwen2.5-Omni uses
large-context integration nor a nominally streaming interface guarantees a block-wise audio processing \cite{xu2025qwenomni}; in our extraction, a 2-s
fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni block is long relative to a 200-ms decision interval. A streaming interface
extraction, completed 2-s blocks and an incomplete current block have alone therefore does not specify how fresh the integrated representation is
different update rules. We therefore test two distinct interfaces: at each forecast cutoff. We study two complementary, cutoff-aligned
\emph{short-window encoder evidence} (SWE) exposes recent modality features interventions: \emph{short-window encoder evidence} (SWE) directly exposes
outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR) recent modality features outside the Thinker state, while \emph{partial-tail
queries only the observed current-block prefix. Neither term implies that refresh} (PTR) queries the observed prefix of the incomplete current block.
all Omni models use the same chunk duration or lack local perception. SWE denotes temporal locality, not shallow encoder layers; the argument
concerns this pipeline, not an intrinsic deficit of every Omni model.
We contribute a causal implementation of SWE and PTR for wearer activity Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity
anticipation; an EasyCom audit separating global, local-audio, pooled-visual, anticipation; an EasyCom analysis separating global, recent-audio,
and privileged information conditions; and a protocol for EasyCom and Ego4D pooled-visual, and privileged visible-activity information; and a specified
transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics.
scores are still pending, so our measured claims remain confined to the Only EasyCom screening scores are currently available: matched PTR effects,
available EasyCom screens. Ego4D transfer, and event-level behavior remain open tests.
\begin{figure}[t] \begin{figure}[t]
\centering \centering
@@ -125,8 +129,9 @@ temporarily queries the observed current-block prefix while reusing completed
\subsection{Task and Short-Window Encoder Evidence} \subsection{Task and Short-Window Encoder Evidence}
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each $t_n$ and predicts activity at $t_n+h$, where
$h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
horizon has three channels in fixed order: background, camera wearer, and horizon has three channels in fixed order: background, camera wearer, and
aggregate others. Wearer and others may overlap; no output identifies an aggregate others. Wearer and others may overlap; no output identifies an
individual speaker, addressee, or semantic end of turn. individual speaker, addressee, or semantic end of turn.
@@ -134,12 +139,13 @@ individual speaker, addressee, or semantic end of turn.
The global feature $\vect g_n$ concatenates the final four frozen The global feature $\vect g_n$ concatenates the final four frozen
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
already contains audio and video context; $\G$ is not audio-free. SWE already contains audio and video context; $\G$ is not audio-free. SWE
adds encoder-side features without requiring the Thinker state to preserve provides a separate short-window path so the head need not recover all
their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed cutoff-local detail from the integrated state. Local audio $\vect a_n$ crops
the latest one-second mixed
waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and
averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens
inside provided person boxes, then across people and valid causal frames in inside provided person boxes, then across people and valid causal 2-FPS
the latest second. This pooling has no identity or mouth-motion trajectory; frames in the latest second. This pooling has no identity or mouth-motion trajectory;
provided boxes do not imply an online detector. provided boxes do not imply an online detector.
Each enabled local feature is layer-normalized, projected to 256 dimensions, Each enabled local feature is layer-normalized, projected to 256 dimensions,
@@ -166,8 +172,10 @@ completed modality payloads, we crop the tail at $t_n$ and query it in a
temporary cache. The tail and query are not committed. Only completed temporary cache. The tail and query are not committed. Only completed
blocks enter persistent history; session and source discontinuities reset blocks enter persistent history; session and source discontinuities reset
caches. Thus each 5-Hz readout sees the observed prefix without future caches. Thus each 5-Hz readout sees the observed prefix without future
backfill. PTR is a causal access rule, not by itself a proven accuracy or backfill. To test whether access policy matters, the matched design crosses
real-time latency improvement. completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap;
the separate 2--12-s cap sweep tests context length. Those comparisons are
pending, so PTR is a causal access rule rather than a demonstrated gain.
\subsection{Visible-Activity Information Audit} \subsection{Visible-Activity Information Audit}
@@ -333,7 +341,7 @@ Full path & -- & -- & -- \\
\label{sec:experiments} \label{sec:experiments}
\textbf{Data and protocol.} \textbf{Data and protocol.}
We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training We use EasyCom with the EgoSpeak official split: Sessions~4--12 for training
and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads
are trained for four fixed epochs on all training sessions without test-time are trained for four fixed epochs on all training sessions without test-time
checkpoint selection. The deployable feature matrix is an archived checkpoint selection. The deployable feature matrix is an archived
@@ -341,7 +349,7 @@ single-seed screen; the information audit uses seeds 0, 1, and 2 with an
identical fixed recipe. We therefore report standard deviations only for the identical fixed recipe. We therefore report standard deviations only for the
latter and treat small differences in the former descriptively. latter and treat small differences in the former descriptively.
\textbf{Ego4D transfer protocol.} \textbf{Ego4D transfer protocol (pending results).}
Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak
manifest and targets, with 346 training and 87 test clips. The same 5-Hz manifest and targets, with 346 training and 87 test clips. The same 5-Hz
cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and
@@ -368,8 +376,8 @@ already sufficient. Because that screen used Session~12 model selection
whereas the historical pooled row did not, their difference is diagnostic, whereas the historical pooled row did not, their difference is diagnostic,
not a matched causal attribution. not a matched causal attribution.
\textbf{Six pre-specified control families.} \textbf{Pre-specified analyses.}
We freeze six pending control families: (L) capacity and recency, (T) tail We freeze six control families: (L) capacity and recency, (T) tail
refresh crossed with local audio, (C) six context caps, (B) paired backbone refresh crossed with local audio, (C) six context caps, (B) paired backbone
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency. transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
They share the official split and physical cutoffs; the 8-s T/C row reuses They share the official split and physical cutoffs; the 8-s T/C row reuses
@@ -381,7 +389,7 @@ one run. The source retains unmeasured hypotheses and falsifiers.
\subsection{EasyCom: Local Encoder Evidence} \subsection{EasyCom: Local Encoder Evidence}
Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at
0.2~s while integrated $\G$ is slightly stronger across all horizons. 0.2~s while integrated $\G$ is slightly stronger on the ten-horizon mean.
Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over
$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
@@ -389,8 +397,10 @@ points, onset AP by only 1.47: dense gains do not imply equal improvement
at both activity boundaries. at both activity boundaries.
The $\G$-only head has more trainable parameters and longer history than the The $\G$-only head has more trainable parameters and longer history than the
$\A$-only head. Thus this screen shows predictive complementarity, not a $\A$-only head. Thus this screen is consistent with predictive
capacity-matched proof that the Thinker lacks a specific local cue. complementarity, not a capacity-matched proof that the Thinker lacks a
specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable
under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong % Retained as a planning aid; illustrative, unmeasured patterns do not belong
% in the page-limited submission preview. % in the page-limited submission preview.
@@ -541,13 +551,14 @@ next other offset. Behavioral claims await a real-timeline evaluator audit.
\section{Conclusion} \section{Conclusion}
On EasyCom, a short-window audio-encoder path complements the frozen Omni For egocentric when-to-speak forecasting, cutoff-aligned access to recent
readout for future wearer activity, most clearly at speech offset. Pooled encoder evidence is a useful complement to frozen Omni context on EasyCom;
vision recovers little of the timely visible-activity information exposed by the strongest measured boundary difference is at wearer speech offset.
the analysis-only audit. Partial-tail refresh provides a causal 5-Hz Pooled vision recovers little of the timely visible-activity information
interface, but its independent predictive and efficiency effects await revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
matched tests. Ego4D transfer and censored event evaluation remain necessary but its independent predictive and efficiency effects still require matched
before claiming generalizable or improved semantic floor behavior. tests. Ego4D transfer and censored event evaluation remain necessary before
claiming generalizable or improved semantic floor behavior.
\section{Acknowledgment} \section{Acknowledgment}
OpenAI Codex assisted with drafting and figure layout; the authors remain OpenAI Codex assisted with drafting and figure layout; the authors remain