% Acceptance-target ICASSP 2027 draft grounded only in archived measurements. % Uses the repository's ICASSP 2027 author kit. % Four technical pages plus an optional reference-only page. \documentclass{article} \usepackage{spconf} \usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float} \hypersetup{hidelinks} \newcommand{\vect}[1]{\boldsymbol{#1}} \newcommand{\G}{\mathrm{G}} \newcommand{\A}{\mathrm{A}} \newcommand{\V}{\mathrm{V}} \setlength{\emergencystretch}{2em} \setcounter{topnumber}{5} \setcounter{dbltopnumber}{5} \renewcommand{\topfraction}{0.95} \renewcommand{\dbltopfraction}{0.95} \renewcommand{\textfraction}{0.05} \renewcommand{\floatpagefraction}{0.75} \renewcommand{\dblfloatpagefraction}{0.75} \title{Egocentric Speech Anticipation with Omni Models:\\ Short-Window Evidence and Partial-Tail Refresh} \name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}} \address{China Telecom Corporation Limited Shanghai Branch} \begin{document} \ninept \maketitle \begin{abstract} An egocentric conversational agent must anticipate when its unseen wearer will speak while other participants exchange turns in a causal multiparty audio--video stream. Integrated Omni states provide broad audiovisual context, yet recent evidence near each prediction cutoff may be difficult for a small forecasting head to access. We study short-window encoder evidence (SWE), which exposes the preceding second of audio and optional causal visual features alongside frozen Qwen2.5-Omni context. We also specify partial-tail refresh (PTR), which queries only the observed prefix of the current audiovisual block at each 200-ms cutoff without committing that prefix to history. We predict background, wearer, and aggregate-other activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches 65.37\%. On Ego4D, paired three-seed global-plus-audio models improve ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%, while a single visual-attention run reaches 70.93\%. These results highlight the value of recent encoder evidence for dense wearer-activity forecasting across both datasets, with strictly causal inputs and a frozen backbone throughout. \end{abstract} \begin{keywords} egocentric speech anticipation, multimodal foundation models, causal streaming, voice activity projection, conversational agents \end{keywords} \section{Introduction} \label{sec:intro} An egocentric conversational agent could answer situated questions, supply context, and assist its wearer during face-to-face conversation. To participate naturally, it must decide not only \emph{what} to say but when to begin, wait, or stop speaking. This requires interpreting mixed audio, visible people's behavior, and linguistic content as the number and roles of interlocutors change. A pause is not necessarily an invitation: two people visible to the wearer may be talking to each other, and their turn exchange should not trigger the agent's response. Since the wearer is normally off-camera, the relevant target is not simply which visible face speaks now, but whether the wearer is about to speak or cease speaking. Both boundaries matter: the agent should withhold an entry during a third-party exchange and yield when another participant takes the floor. From a moving first-person view, face visibility and overlapping voices can change independently; neither a silence threshold nor one detected face provides a fixed map of who is interacting with the wearer. These timing errors arise before response content is considered. Prior work addresses complementary parts of this decision. Voice activity detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice activity projection (VAP) predicts future speaker activity \cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and linguistic evidence for spoken-agent turn detection \cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel prediction exploits visible cues but is commonly evaluated with two interlocutors \cite{russell2025mmvap,lin2025backchannel, oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed three-person conversations \cite{elmers2025triadic}; online three-party EOT has also used controlled multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of speakers using single-camera video and face tracks, but studies visible speakers from a static third-party view \cite{qi2026muvap}. Device-directed speech detection can combine acoustics, ASR output, and text to ask whether an utterance addresses an assistant, yet the cited model does not observe visual relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly forecasts the unseen wearer's future speech from causal first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak, donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction, first-person speaking/listening graphs, and addressee detection provide related but distinct targets \cite{serj2024threem,jia2024avconv, kong2024sicnet}. Egocentric active-speaker localization likewise concerns current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}. Together, these studies leave open how an integrated streaming model should retain cutoff-local evidence for forecasting the unseen wearer's activity in a natural multiparty, first-person stream. This is a representation-and-timing question, not a claim to first egocentric turn prediction. Our activity labels do not establish addressee or semantic floor ownership. Omni models couple modality encoders to a shared language-model state, so audio, video, and text can inform streaming interaction and longer-range semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}. Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and time-aligned audio--video interleaving feed a Thinker for integrated understanding; a separate Talker generates speech \cite{xu2025qwenomni}. The main EasyCom comparison uses frozen 3B Thinker readouts; a separate seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our wearer forecast updates every 200~ms. Holding only completed blocks leaves stale context between boundaries; forcing 200-ms commits changes the native block construction and may alter its representation. Neither implies that all Omni models lack local evidence. An integrated, multi-layer state can carry broader audiovisual and linguistic context, yet transient cues may be less accessible to a small head than encoder-side features themselves \cite{lin2023lavish,rouditchenko2024whisperflamingo}. We therefore retain the global Thinker state while exposing recent modality evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens suggest that global context and local audio are complementary; gains from pooled vision and isolated PTR effects remain unproven. This study makes three contributions: (1) short-window encoder evidence (SWE) supplies recent audio and optional visual encoder evidence to a frozen-Omni forecasting head; (2) partial-tail refresh (PTR) queries the observed current-block prefix without committing it to persistent history; and (3) detailed EasyCom analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run. Matched PTR and event behavior remain pending. \begin{figure}[t] \centering \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} \caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context with recent audio and optional local visual evidence in the forecasting head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only the observed current-block tail at each 200-ms cutoff. The tail is not committed to persistent history.} \label{fig:overview} \end{figure} \section{Method} \label{sec:method} \subsection{Task and Global--Local Fusion} At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each horizon has three channels in fixed order: background, camera wearer, and aggregate others. Wearer and others may overlap; no output identifies an individual speaker, addressee, or semantic end of turn. The global representation $\vect g_n$ concatenates the final four frozen Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It already integrates audio and video over the available context; $\G$ is not an audio-free baseline. Short-window encoder evidence (SWE) adds a direct, cutoff-aligned path for recent audio $\vect a_n$ and, when enabled, visual evidence $\vect v_n$. This separates the question of what the global state encodes from whether a small prediction head can access transient cues. In the pooled fusion, each available local vector is layer-normalized, projected to 256 dimensions, and accompanied by a validity bit. The concatenated global and local features are passed to a layer-normalized MLP: \begin{equation} \widehat{\vect Y}_{n,h,c}=\operatorname{softmax}_{c} F_{\theta}\!\left([\vect g_n;\widetilde{\vect a}_n;m_n^a; \widetilde{\vect v}_n;m_n^v]\right)_{h,c}, \label{eq:fusion} \end{equation} where a disabled branch is omitted and $F_\theta$ produces ten sets of three logits. For token-preserving visual variants, $\vect v_n$ denotes an attention-pooled context: a query formed from $\vect g_n$ and valid $\vect a_n$ attends to the causal visual token set (and a learned null token), rather than averaging it beforehand. The resulting context and validity bit enter the same prediction head. We train the projections, visual aggregator when present, and head with the EgoSpeak multi-positive cross-entropy over causal-valid source--horizon pairs. The principal SWE configuration freezes all feature encoders and the Thinker: this is frozen-backbone adaptation, not task-level end-to-end training. We separate \emph{where} parameters are updated from \emph{which} Thinker layers are read. The primary comparison trains only the layer-combination projection, local-fusion modules, and 30-logit prediction head on cached frozen features. Parameter-adaptation controls instead attach low-rank adapters to Thinker attention projections, either across the configured Thinker or restricted to its upper layers; the prediction head may be updated jointly or held fixed to isolate representation changes. These controls require the same causal media, source cutoffs, supervision, initialization, and model-selection protocol as the frozen-head condition. Archived LoRA runs, including an upper-eight-layer setting, are exploratory and not a matched layer-scope ablation. Last-four-layer \emph{readout} fusion changes the head input, not the LoRA update scope; neither its gain nor an unmatched LoRA score establishes an adaptation advantage. \subsection{Causal Partial-Tail Refresh} PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap (Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the observed current tail are available; span varies with block phase. Reusing completed modality payloads, we crop the tail at $t_n$ and query it in a temporary cache. The tail and query are not committed. Only completed blocks enter persistent history; session and source discontinuities reset caches. Thus each 5-Hz readout sees the observed prefix without future backfill. To test whether access policy matters, the matched design crosses completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap; the separate 2--12-s cap sweep tests context length. Those comparisons are pending, so PTR is a causal access rule rather than a demonstrated gain. \subsection{Short-Window Feature Extraction} Each local branch observes only media preceding the prediction cutoff $t_n$; unavailable observations retain explicit validity masks rather than borrowing later samples. All encoders are frozen. The notation below distinguishes encoder choice from fusion choice: $\G$ is the global Thinker state, and local branches are appended to the same forecasting head. The default audio feature $\A$ encodes the mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the Qwen2.5-Omni audio encoder and averages valid tokens. $\A_{\rm E}$ denotes EgoSpeak's wav2vec~2.0 convolutional representation \cite{kim2025egospeak}; comparing $\G+\A$ with $\G+\A_{\rm E}$ requires the same window and head. Released EgoSpeak A+V remains an external baseline, not this encoder substitution. For vision, Pool averages Qwen tokens within provided person boxes and across valid 2-FPS frames in the preceding second; the boxes are not outputs of an evaluated online detector. Spatial visual attention (SVA) instead lets $\G+\A$ query the unpooled spatial tokens of the latest strictly pre-cutoff full frame. Temporal spatial visual attention (TSVA) first applies a shared GRU to up to two causal tokens at each matching grid position, then uses the same query. TSVA links image coordinates, not person tracks. We denote these variants by $\G+\A+$Pool, $\G+\A+$SVA, and $\G+\A+$TSVA. \input{tables_compact} \begin{figure*}[t] \centering \includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf} \caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap; (B) other-to-other handover. Outcome-selected, unreviewed examples do not establish an event-level gain.} \label{fig:hq_cases} \end{figure*} % Historical full-precision table layouts retained as source audit only. \iffalse \begin{table*}[t] \caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B: three-seed privileged past-activity audit, which is not deployable. Mean is ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events. TSVA is a no-box, two-frame causal full-frame screen; it and the historical deployable rows are exploratory seed-0 runs with unmatched head/training details. F1 has three seeds. Dashes are unavailable, not zero.} \label{tab:main} \centering \normalsize \renewcommand{\arraystretch}{0.87} \setlength{\tabcolsep}{5.0pt} \begin{tabular}{@{}lccccc@{}} \toprule Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$ & Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\ \midrule EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\ Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\ Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\ Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\ Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086 & 65.400 & 26.019 & 45.371 \\\\ No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\ Current-frame person set F1 & $81.777{\pm}0.156$ & -- & $64.951{\pm}0.095$ & -- & -- \\\\ Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\ Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\ Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\ \bottomrule \end{tabular} \par\smallskip \emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip \normalsize \setlength{\tabcolsep}{3.2pt} \begin{tabular}{@{}lccc@{}} \toprule Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\ \midrule None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\ Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\ Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\ Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\ Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\ Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$ & $\mathbf{66.951{\pm}.022}$ \\\\ \bottomrule \end{tabular} \end{table*} \begin{table*}[t] \caption{Chunk and context audit. Panel A is an unmeasured matched T/C matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap, 2-FPS video); its rows are not entries in Panel A. Dashes are pending.} \label{tab:stream_context} \centering\normalsize \renewcommand{\arraystretch}{0.87} \setlength{\tabcolsep}{4pt} \begin{tabular}{@{}llccccc@{}} \toprule Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP & $\Delta$ On & $\Delta$ Off \\ \midrule T & Completed-only, 8 s & -- & -- & -- & -- & -- \\ T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\ \midrule C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\ C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\ C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\ C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\ C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\ C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\ \bottomrule \end{tabular} \par\smallskip \emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip \begin{tabular}{@{}lccc@{}} \toprule Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\ \midrule Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\ Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\ Completed blocks + observed tail & 70.85 & 59.17 & -- \\ \bottomrule \end{tabular} \end{table*} \fi \begin{table}[t] \caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a single-final-state readout and the same local-audio sidecar. Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not directly comparable across tables.} \label{tab:backbone} \centering\normalsize \setlength{\tabcolsep}{3pt} \begin{tabular}{@{}lcc@{}} \toprule Backbone & $\G$ & $\G+\A$ \\ \midrule Omni 3B & 58.04 & 62.20 \\ Omni 7B & 57.70 & 62.18 \\ \bottomrule \end{tabular} \end{table} \section{Experiments} \label{sec:experiments} \textbf{Data Preparation.} We use the EgoSpeak EasyCom split (Sessions~4--12 for training and Sessions~1--3 for testing) and its Ego4D audiovisual manifest \cite{donley2021easycom,grauman2022ego4d}, which contains 346 training and 87 test clips. Both datasets use 5-Hz causal cutoffs and targets, the same one-second local-audio window, and visual samples strictly preceding each cutoff; unavailable vision is masked rather than silently dropping rows. EasyCom heads use four fixed training epochs without test-set checkpoint selection. Its main deployable matrix is an archived single-seed screen; the privileged information audit and context-cap screen use three seeds. Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$ have paired seeds 0--2 under one four-epoch recipe on 87 test clips. The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D table \cite{kim2025egospeak}; it is not a paired rerun under our causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D $\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR remains unmeasured. \textbf{Metrics.} Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP) separately for background, wearer, and others at each horizon over common causal-valid cutoff--target pairs. Their unweighted class average is the horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s. All reported AP and mAP values are percentages. \section{Results and Analysis} \label{sec:results} \noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).} All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores 81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus 73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84 points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores 79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$ 80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired $\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s. The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets. Because these tables report mAP rather than classwise AP, they do not identify whether background, wearer, or others drives each gain. \noindent\textbf{Visual and privileged evidence (Tables~\ref{tab:main} and~\ref{tab:visual}).} EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and 82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$. Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s. At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$, so the visual difference is not uniform over horizons. Ego4D's single SVA run scores 81.39/70.93\% against the three-seed $\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at 2.0~s. Head/training differences prevent isolating a visual effect. In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean gaps over visibility-only suggest useful activity timing, but transcript labels are unavailable at deployment and are not an upper bound. None of these screens establishes speaker grounding or fewer false wearer entries. Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a population-level or modality-specific gain. \noindent\textbf{Context and chunk construction (Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).} The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at 4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank is 0.37/0.44 points above 4/12~s, but predates joint extraction and no matched $\G$ cap arm exists. In the separate global-only chunk screen, native hold scores 59.62/55.19\% at 0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score 67.05/57.84\% with 95,453 calls. The observed-tail row scores 70.85/59.17\%, with call count unavailable. Commits use roughly ten times as many processor calls as native hold; this is not a latency ratio. Changed chunk construction and unmatched extraction preclude an isolated PTR or throughput claim. \noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).} With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\% (+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface differs from the four-layer main readout and does not rank backbone sizes. Across all tables, AP/mAP assess future activity rather than semantic floor-control decisions. \section{Conclusion} For egocentric when-to-speak forecasting, cutoff-aligned access to recent encoder evidence is a useful complement to frozen Omni context on EasyCom; the measured mAP separation is largest near the prediction cutoff. Pooled vision recovers little of the timely visible-activity information revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule, but its independent predictive and efficiency effects still require matched tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small descriptive SVA gain in one run; censored event evaluation remains necessary before claiming generalizable or improved semantic floor behavior. These results therefore support dense future-activity prediction, while the visual screens remain descriptive and the transcript-derived activity audit remains an information probe. A decisive test must hold cutoffs, context caps, training recipe, and valid pairs fixed while crossing PTR with local audio. It should then report per-session variability and event-level false entries and yields after censoring competing speaker changes. Only then can forecast gains support claims about an agent's floor decisions. \clearpage \bibliographystyle{IEEEbib} \bibliography{references} \end{document}