diff --git a/manuscript/targeted/README.md b/manuscript/targeted/README.md index 3467bed..cd5e515 100644 --- a/manuscript/targeted/README.md +++ b/manuscript/targeted/README.md @@ -15,6 +15,10 @@ interaction, then studies two cutoff-aligned interfaces: - **partial-tail refresh (PTR)**: query only the observed prefix of the current two-second block at each 200-ms cutoff, reusing completed payloads. +SWE names the short temporal support of encoder evidence, not a shallow +network layer. The two-second chunk refers to this Qwen extraction setup, +not a universal Omni-model setting. + The evidence remains a causal global--local **audit**, not a claim that all Omni models lack local perception or that PTR's independent gain is proven: diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index 7e04b3b..aef35a1 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -19,30 +19,31 @@ \renewcommand{\dblfloatpagefraction}{0.75} \title{Egocentric Speech Anticipation with Omni Models:\\ Short-Window Evidence and Partial-Tail Refresh} -\name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\ +\name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}} -\address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom} +\address{China Telecom Corporation Limited Shanghai Branch} \begin{document} \ninept \maketitle \begin{abstract} -An egocentric conversational agent must anticipate when its invisible wearer -will speak while observing a causal multiparty audio--video stream. Although -turn-taking prediction and omni-modal assistants have advanced, the temporal -accessibility of local evidence in an integrated Omni state remains unclear. -We study two interfaces: a short-window encoder-evidence path that exposes -recent audio and optional person-crop vision to a frozen-Omni forecasting -head, and partial-tail refresh that queries the observed prefix of an -incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the -exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\% -ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled -vision adds only 0.29 mean-mAP points, whereas an analysis-only causal -visible-activity oracle reveals additional information. A separate chunk -screen suggests sensitivity to state freshness, but does not isolate the -partial-tail effect. These results motivate finer local evidence access; -they do not establish semantic floor control or cross-dataset transfer. +An egocentric conversational agent must anticipate when its unseen camera +wearer will speak from a causal multiparty audio--video stream. Turn-taking +models and streaming Omni assistants address related problems, but the +cutoff-level availability of local evidence in an integrated Omni state is +underexplored for this wearer-centered task. We examine two interfaces: +short-window encoder evidence (SWE) supplies recent audio and optional +person-crop vision directly to a frozen-Omni forecasting head; partial-tail +refresh (PTR) queries only the observed prefix of the current audiovisual +block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio +screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus +74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP +points, while an analysis-only visible-activity audit exposes further local +information. A separate chunk screen suggests sensitivity to state +freshness but does not isolate PTR. We specify an Ego4D transfer protocol; +its scores remain pending. The evidence supports causal activity +anticipation, not semantic floor control or established cross-dataset gains. \end{abstract} \begin{keywords} @@ -53,16 +54,17 @@ streaming, voice activity projection, conversational agents \section{Introduction} \label{sec:intro} -An egocentric conversational agent must decide \emph{when} to speak while its -camera wearer is usually invisible. It sees other participants and hears a -mixed conversation, but must predict the wearer's own future speech before a -turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models -and voice-activity projection (VAP) show the value of anticipatory rather than -silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap, -elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap, -jiang2026turnbench}. Yet these tasks differ in sensor placement, visible -speakers, dialogue roles, and whether the target is a future activity state -or a semantic intervention. +An egocentric conversational agent must decide \emph{when} to speak even +though its camera wearer is usually invisible. It hears a mixed conversation +and sees other participants, but must anticipate its own speech from the +observed stream before a turn boundary is known (Fig.~\ref{fig:task}). +Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic +extensions, backchannel prediction, online end-of-turn detection, and event +benchmarks address increasingly realistic timing decisions +\cite{ekstedt2022vap,russell2025mmvap,elmers2025triadic, +lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}. +Their input viewpoints and target decisions, however, are not interchangeable +with forecasting the future activity of an unseen wearer. \begin{figure}[H] \centering @@ -75,40 +77,42 @@ not semantic floor ownership.} \end{figure} \textbf{Egocentric audiovisual conversation.} -EgoSpeak forecasts wearer speech from first-person RGB and audio using -specialized encoders and a temporal fusion model \cite{kim2025egospeak}; -EasyCom and Ego4D supply complementary egocentric conversation settings -\cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker -anticipation, egocentric speaking/listening graphs, and addressee detection -study related but different targets \cite{serj2024threem,jia2024avconv, -kong2024sicnet}. Egocentric active-speaker localization and detection -\cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person -modeling \cite{wang2024loconet} estimate who is speaking now; they do not by -themselves forecast an unseen wearer's future activity. Visual cues can help -turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled -vision conveys the relevant local floor evidence here is empirical. +EgoSpeak is the closest wearer-centered predecessor: it extracts audio and +RGB features with specialized encoders, fuses their temporal evidence, and +forecasts wearer speech on EasyCom and Ego4D \cite{kim2025egospeak, +donley2021easycom,grauman2022ego4d}. Other first-person work addresses +multi-view next-speaker prediction, speaking/listening graphs, or addressee +detection \cite{serj2024threem,jia2024avconv,kong2024sicnet}. +Egocentric active-speaker methods \cite{jiang2022asl,huh2025egoasd, +clarke2025asd} and long--short person modeling \cite{wang2024loconet} +recover present speaker evidence; audiovisual cues can also help turn +prediction in noise \cite{oconnorrussell2025noise}. These studies make a +broad ``first audiovisual turn-taking'' claim untenable. What remains +less examined is how a single causal first-person stream should expose +recent audio--visual evidence to forecast the invisible wearer's activity. -\textbf{Omni and local evidence.} -Omni systems integrate modalities and streaming interaction +\textbf{Streaming Omni and temporal access.} +Omni systems integrate modalities and support streaming interaction \cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug, yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side audio--visual adaptation offers another route to accessible modality evidence -\cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither -large-context integration nor a nominally streaming interface guarantees a -fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni -extraction, completed 2-s blocks and an incomplete current block have -different update rules. We therefore test two distinct interfaces: -\emph{short-window encoder evidence} (SWE) exposes recent modality features -outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR) -queries only the observed current-block prefix. Neither term implies that -all Omni models use the same chunk duration or lack local perception. +\cite{lin2023lavish,rouditchenko2024whisperflamingo}. Qwen2.5-Omni uses +block-wise audio processing \cite{xu2025qwenomni}; in our extraction, a 2-s +block is long relative to a 200-ms decision interval. A streaming interface +alone therefore does not specify how fresh the integrated representation is +at each forecast cutoff. We study two complementary, cutoff-aligned +interventions: \emph{short-window encoder evidence} (SWE) directly exposes +recent modality features outside the Thinker state, while \emph{partial-tail +refresh} (PTR) queries the observed prefix of the incomplete current block. +SWE denotes temporal locality, not shallow encoder layers; the argument +concerns this pipeline, not an intrinsic deficit of every Omni model. -We contribute a causal implementation of SWE and PTR for wearer activity -anticipation; an EasyCom audit separating global, local-audio, pooled-visual, -and privileged information conditions; and a protocol for EasyCom and Ego4D -transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D -scores are still pending, so our measured claims remain confined to the -available EasyCom screens. +Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity +anticipation; an EasyCom analysis separating global, recent-audio, +pooled-visual, and privileged visible-activity information; and a specified +EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics. +Only EasyCom screening scores are currently available: matched PTR effects, +Ego4D transfer, and event-level behavior remain open tests. \begin{figure}[t] \centering @@ -125,8 +129,9 @@ temporarily queries the observed current-block prefix while reusing completed \subsection{Task and Short-Window Encoder Evidence} -At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and -predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each +At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by +$t_n$ and predicts activity at $t_n+h$, where +$h\in\{0.2,0.4,\ldots,2.0\}$~s. Each horizon has three channels in fixed order: background, camera wearer, and aggregate others. Wearer and others may overlap; no output identifies an individual speaker, addressee, or semantic end of turn. @@ -134,12 +139,13 @@ individual speaker, addressee, or semantic end of turn. The global feature $\vect g_n$ concatenates the final four frozen Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It already contains audio and video context; $\G$ is not audio-free. SWE -adds encoder-side features without requiring the Thinker state to preserve -their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed +provides a separate short-window path so the head need not recover all +cutoff-local detail from the integrated state. Local audio $\vect a_n$ crops +the latest one-second mixed waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens -inside provided person boxes, then across people and valid causal frames in -the latest second. This pooling has no identity or mouth-motion trajectory; +inside provided person boxes, then across people and valid causal 2-FPS +frames in the latest second. This pooling has no identity or mouth-motion trajectory; provided boxes do not imply an online detector. Each enabled local feature is layer-normalized, projected to 256 dimensions, @@ -160,14 +166,16 @@ is frozen-backbone adaptation rather than end-to-end training. \subsection{Causal Partial-Tail Refresh} PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap -(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the -observed current tail are available; span varies with block phase. Reusing +(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the +observed current tail are available; span varies with block phase. Reusing completed modality payloads, we crop the tail at $t_n$ and query it in a -temporary cache. The tail and query are not committed. Only completed +temporary cache. The tail and query are not committed. Only completed blocks enter persistent history; session and source discontinuities reset -caches. Thus each 5-Hz readout sees the observed prefix without future -backfill. PTR is a causal access rule, not by itself a proven accuracy or -real-time latency improvement. +caches. Thus each 5-Hz readout sees the observed prefix without future +backfill. To test whether access policy matters, the matched design crosses +completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap; +the separate 2--12-s cap sweep tests context length. Those comparisons are +pending, so PTR is a causal access rule rather than a demonstrated gain. \subsection{Visible-Activity Information Audit} @@ -333,7 +341,7 @@ Full path & -- & -- & -- \\ \label{sec:experiments} \textbf{Data and protocol.} -We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training +We use EasyCom with the EgoSpeak official split: Sessions~4--12 for training and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads are trained for four fixed epochs on all training sessions without test-time checkpoint selection. The deployable feature matrix is an archived @@ -341,7 +349,7 @@ single-seed screen; the information audit uses seeds 0, 1, and 2 with an identical fixed recipe. We therefore report standard deviations only for the latter and treat small differences in the former descriptively. -\textbf{Ego4D transfer protocol.} +\textbf{Ego4D transfer protocol (pending results).} Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak manifest and targets, with 346 training and 87 test clips. The same 5-Hz cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and @@ -368,8 +376,8 @@ already sufficient. Because that screen used Session~12 model selection whereas the historical pooled row did not, their difference is diagnostic, not a matched causal attribution. -\textbf{Six pre-specified control families.} -We freeze six pending control families: (L) capacity and recency, (T) tail +\textbf{Pre-specified analyses.} +We freeze six control families: (L) capacity and recency, (T) tail refresh crossed with local audio, (C) six context caps, (B) paired backbone transfer, (H/Q) error slices and timelines, and (S) full-path efficiency. They share the official split and physical cutoffs; the 8-s T/C row reuses @@ -381,7 +389,7 @@ one run. The source retains unmeasured hypotheses and falsifiers. \subsection{EasyCom: Local Encoder Evidence} Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at -0.2~s while integrated $\G$ is slightly stronger across all horizons. +0.2~s while integrated $\G$ is slightly stronger on the ten-horizon mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at 0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 @@ -389,8 +397,10 @@ points, onset AP by only 1.47: dense gains do not imply equal improvement at both activity boundaries. The $\G$-only head has more trainable parameters and longer history than the -$\A$-only head. Thus this screen shows predictive complementarity, not a -capacity-matched proof that the Thinker lacks a specific local cue. +$\A$-only head. Thus this screen is consistent with predictive +complementarity, not a capacity-matched proof that the Thinker lacks a +specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable +under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}. % Retained as a planning aid; illustrative, unmeasured patterns do not belong % in the page-limited submission preview. @@ -541,13 +551,14 @@ next other offset. Behavioral claims await a real-timeline evaluator audit. \section{Conclusion} -On EasyCom, a short-window audio-encoder path complements the frozen Omni -readout for future wearer activity, most clearly at speech offset. Pooled -vision recovers little of the timely visible-activity information exposed by -the analysis-only audit. Partial-tail refresh provides a causal 5-Hz -interface, but its independent predictive and efficiency effects await -matched tests. Ego4D transfer and censored event evaluation remain necessary -before claiming generalizable or improved semantic floor behavior. +For egocentric when-to-speak forecasting, cutoff-aligned access to recent +encoder evidence is a useful complement to frozen Omni context on EasyCom; +the strongest measured boundary difference is at wearer speech offset. +Pooled vision recovers little of the timely visible-activity information +revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule, +but its independent predictive and efficiency effects still require matched +tests. Ego4D transfer and censored event evaluation remain necessary before +claiming generalizable or improved semantic floor behavior. \section{Acknowledgment} OpenAI Codex assisted with drafting and figure layout; the authors remain