% Acceptance-target ICASSP 2027 draft grounded only in archived measurements. % The official 2027 author kit should replace the 2026-compatible style assets % before submission. Four technical pages plus an optional reference-only page. \documentclass{article} \usepackage{spconf} \usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor} \hypersetup{hidelinks} \newcommand{\vect}[1]{\boldsymbol{#1}} \newcommand{\G}{\mathrm{G}} \newcommand{\A}{\mathrm{A}} \newcommand{\V}{\mathrm{V}} \setlength{\emergencystretch}{2em} \title{Does an Omni Model Know When the Wearer Will Speak?\\ A Causal Global--Local Audit of Egocentric Speech Anticipation} \name{Authors withheld in this working draft} \address{} \begin{document} \ninept \maketitle \begin{abstract} An egocentric conversational agent must anticipate the future speech of an invisible camera wearer from a causal multiparty audio--video stream. We ask whether a frozen general-purpose Omni state exposes the sub-second evidence needed by this task. Our cutoff-aligned model combines an 8-s Qwen2.5-Omni context with the most recent 1-s audio and predicts three activity channels at ten horizons from 0.2 to 2.0~s. In a single-seed EasyCom screen, this model reaches 81.72\% mAP at 0.2~s and 65.11\% averaged over horizons, compared with 74.20\% and 61.19\% for released EgoSpeak. The gain is larger for wearer offset anticipation (+7.23 points) than onset anticipation (+1.47). Pooled visual features add only 0.29 mean-mAP points. A separate three-seed, strictly causal information audit explains this gap: privileged past activity of visible participants adds 4.83 points at 0.2~s and 1.77 mean-mAP points, whereas visibility, one-second-delayed activity, and identity shuffling show that timeliness matters more than identity continuity. The results isolate an evidence-access bottleneck: recent acoustic evidence is already useful, while current generic visual pooling does not recover the available visible-floor signal. \end{abstract} \begin{keywords} egocentric speech anticipation, multimodal foundation models, causal streaming, voice activity projection, conversational agents \end{keywords} \section{Introduction} \label{sec:intro} Conversational agents must decide not only \emph{what} to say, but also \emph{when} to participate. This decision is particularly asymmetric in an egocentric stream: the camera wearer is normally invisible, yet the model must forecast that wearer's behavior from mixed audio and observations of other people. A useful predictor must update before a turn boundary is known and must not use media from after its decision cutoff. Voice Activity Projection (VAP) casts turn taking as continuous prediction of future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to untrimmed first-person video and defines the EasyCom protocol used here \cite{kim2025egospeak,donley2021easycom}. Other work predicts future speakers from synchronized egocentric sensors \cite{serj2024threem}, reconstructs egocentric conversational graphs \cite{jia2024avconv}, or grounds causal multiparty projections in visible face tracks \cite{qi2026muvap}. Those systems establish that neither multiparty audiovisual modeling nor first-person speech initiation is itself a novelty. Our question is narrower: does a general-purpose Omni readout make timely evidence accessible for forecasting the unseen wearer? Qwen2.5-Omni integrates long-form audio and video in a Thinker representation \cite{xu2025qwenomni}. Such integration is attractive for conversational context, but a blockwise state can blur or delay the short acoustic and visual cues needed at 200-ms intervals. We therefore separate an integrated global state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat their value as an empirical question rather than assuming that a larger backbone solves temporal access. This paper makes three contributions. First, we formulate a strictly causal global--local audit for invisible-wearer activity forecasting, with a common 5-Hz cutoff and ten future horizons. Second, we realize the audit with a bounded Omni stream that combines completed history with a temporary observed tail and a separate one-second audio path. Third, we distinguish deployable feature gains from an analysis-only visible-activity oracle. This reveals a specific negative result---generic pooled or current-frame person features do not recover most of the available local-floor information---and identifies timely activity estimation, rather than identity continuity, as the next modeling target. \begin{figure*}[t] \centering \includegraphics[width=0.97\textwidth]{figures/system_overview.pdf} \caption{Global--local causal audit. At each 200-ms cutoff, the prediction path fuses an integrated Omni state with explicit recent audio and optional person-crop vision. Completed 2-s blocks are reused while only the observed partial tail is temporarily recomputed. The purple branch is a privileged past-activity audit used to test information value; it is never a deployable input and contains no future activity.} \label{fig:overview} \end{figure*} \section{Method} \label{sec:method} \subsection{Task and Causal Inputs} At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each horizon has three channels in fixed order: background, camera wearer, and aggregate others. Wearer and others may overlap; no output identifies an individual speaker, addressee, or semantic end of turn. The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B. We concatenate the final four Thinker readout states and project them to 2048 dimensions. The readout contains both audio and video context; $\G$ is not an audio-free condition. The local audio feature $\vect a_n$ is obtained by cropping the most recent one-second mixed waveform \emph{before} preprocessing, encoding it with the frozen Qwen audio encoder, and averaging valid output tokens. An optional visual feature $\vect v_n$ averages spatial tokens inside provided person boxes, people within each causal frame, and valid frames from the most recent second. It preserves neither identity nor mouth-motion trajectories, and provided boxes do not constitute an online detector. Each enabled local feature is layer-normalized, projected to 256 dimensions, and paired with a validity bit. The trainable head is \begin{equation} \widehat{\vect Y}_{n}= \operatorname{softmax}_{c}F_{\theta} \bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a; \widetilde{\vect v}_n;m_n^v\bigr], \label{eq:fusion} \end{equation} where omitted modalities remove their terms and $F_\theta$ is a layer-normalized MLP with a 30-logit output. We train only the projections and head with the EgoSpeak multi-positive cross-entropy over causal-valid source--horizon pairs. The Omni and modality encoders remain frozen, so this is frozen-backbone adaptation rather than end-to-end training. \subsection{Cutoff-Aligned Streaming} The global stream uses source-aligned 2-s blocks and an 8-s maximum presentation budget. After warm-up, three completed blocks and the observed part of the current block are available; the exact span varies with block phase. Completed modality payloads are reused when the bounded language-model cache is rebuilt. At every cutoff, the incomplete current block is cropped at $t_n$, appended to a temporary cache, and queried for $\vect g_n$. Neither the temporary tail nor the query is committed. At a block boundary the completed payload is committed once and a new tail begins. Sessions and source discontinuities reset all caches. Thus the logical state is refreshed at 5~Hz without backfilling a completed block into earlier rows. \subsection{Visible-Activity Information Audit} To distinguish missing information from ineffective learned vision, we build analysis-only features from transcript-derived \emph{past} activity. At each cutoff, visible non-wearer participants receive two seconds of binary history sampled on the same 5-Hz grid. The preparation fails if its aggregate activity disagrees with the released target. We compare: (i) visibility with activity zeroed, (ii) per-person histories, (iii) identities shuffled while preserving active-person counts at every lag, (iv) histories delayed by one second, and (v) one OR-pooled aggregate-visible history. These conditions never expose future activity, but their speech labels make them privileged diagnostics, not deployable features or upper bounds. \section{Experiments} \label{sec:experiments} \textbf{Data and protocol.} We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads are trained for four fixed epochs on all training sessions without test-time checkpoint selection. The deployable feature matrix is an archived single-seed screen; the information audit uses seeds 0, 1, and 2 with an identical fixed recipe. We therefore report standard deviations only for the latter and treat small differences in the former descriptively. \textbf{Metrics.} At each horizon we compute AP for the three activity channels over common causal-valid rows, then average classes to obtain mAP and average all ten horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint diagnostics, onset AP conditions source rows on the wearer being silent and scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer speech and scores future silence with $1-p(\mathrm{wearer})$. These threshold-free quantities average ten horizons. They are not one-event-one- vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding. \textbf{Comparisons.} We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni $\G$; $\G+\A$; and $\G+\A+$pooled-$\V$. A three-seed current-frame frame-set attention screen tests whether a richer person aggregation is already sufficient. Because that screen used Session~12 model selection whereas the historical pooled row did not, their difference is diagnostic, not a matched causal attribution. \section{Results} \label{sec:results} \begin{table*}[t] \caption{EasyCom Sessions~1--3 results (\%). $\mathrm{mAP}_{.2}$ and $\mathrm{WAP}_{.2}$ are dense 0.2-s metrics; Mean averages dense mAP over ten horizons. On/Off are raw wearer endpoint AP, not semantic event scores. Deployable rows are archived single-seed screens; $\pm$ is shown only for the three-seed F1 screen. Dashes denote unavailable measurements.} \label{tab:main} \centering \scriptsize \setlength{\tabcolsep}{5.0pt} \begin{tabular}{@{}lccccc@{}} \toprule Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$ & Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\ \midrule EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\ Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\ Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\ Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\ Omni $\G+\A+$pooled-$\V$ & \textbf{82.027} & \textbf{85.086} & \textbf{65.400} & \textbf{26.019} & 45.371 \\\\ Current-frame person set F1 & $81.777{\pm}0.156$ & -- & $64.951{\pm}0.095$ & -- & -- \\\\ \bottomrule \end{tabular} \end{table*} \subsection{Global Context and Recent Audio Are Complementary} Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is slightly stronger when mAP is averaged to 2.0~s. Their fusion is better than $\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at 0.2~s and 4.47 at 2.0~s. It also improves every class when averaged over horizons (+8.72 background, +2.43 wearer, and +2.65 others). Relative to released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by 3.92 points. The endpoint effect is asymmetric: offset AP rises by 7.23 points, while onset AP rises by 1.47. Dense forecasting gains therefore do not imply equal progress at both behavior boundaries. The global-only head has more trainable parameters and a longer context than the local-only head, so this comparison demonstrates predictive complementarity, not that semantic reasoning alone causes the gain. A wider history-matched control is required for a representation-level causal claim. \subsection{Generic Vision Does Not Recover the Available Signal} Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and +0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when averaged over horizons; onset/offset changes are +0.059/+0.120. A richer current-frame person-set model is stable across seeds but reaches lower mean mAP than the historical pooled row under a non-matched selection protocol. These results do not support a claim of learned speaker grounding. \begin{table}[t] \caption{Three-seed privileged-information audit (mean $\pm$ sample standard deviation, \%). Every row includes $\G+\A$; activity conditions use only past labels. Bold marks the best diagnostic, not a deployable model.} \label{tab:oracle} \centering \scriptsize \setlength{\tabcolsep}{3.2pt} \begin{tabular}{@{}lccc@{}} \toprule Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\ \midrule None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\ Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\ Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\ Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\ Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\ Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$ & $\mathbf{66.951{\pm}.022}$ \\\\ \bottomrule \end{tabular} \end{table} Table~\ref{tab:oracle} locates the missing information. Aggregate visible activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775 mean-mAP points. Visibility alone changes mean mAP by only +0.040, and one-second-delayed activity by +0.164. Thus availability or extra parameters cannot explain the oracle gain. Per-person activity exceeds shuffled identity by only 0.200 mean-mAP points and is itself 0.291 below the aggregate condition. The current target therefore supports timely visible-floor activity as useful information, but not the necessity of identity continuity. The majority of the oracle's 0.2-s macro gain comes from background and others; wearer AP rises by only 0.572. This is why neither the oracle nor a dense mAP gain establishes fewer false wearer entries. \subsection{What the Audit Establishes} Three conclusions survive the controls. First, a general-purpose global state and a one-second acoustic readout provide different useful timescales. Second, generic visual pooling has not converted visible people into reliable local floor evidence. Third, perfect timely aggregate activity contains recoverable information, whereas identity contributes little under the current target. A dedicated causal active-speaker front end---designed for mouth motion, blur, and short audio--visual synchrony as in egocentric ASD \cite{huh2025egoasd}---is consequently a better-motivated next step than a larger identity-aware fusion module. The evidence remains bounded. EasyCom has only three official test sessions; the deployable matrix is single seed and has been examined repeatedly. The oracle uses labels and is not a performance upper bound. Provided boxes omit online detection cost, and logical 5-Hz output is not a measured throughput claim. Finally, aggregate others cannot supply addressee or semantic interruption ground truth. Confirmatory behavioral claims require a frozen training-side operating point, competing-risk event attribution, and multi-seed matched controls. \section{Conclusion} We presented a strictly causal global--local audit of egocentric wearer speech anticipation. Frozen Omni context and recent audio are strongly complementary, especially for future wearer cessation. In contrast, pooled or current-frame person features do not recover the local-floor signal revealed by a causal visible-activity oracle. The actionable conclusion is deliberately narrow: the next bottleneck is timely, learnable local activity evidence, not simply more multimodal capacity or persistent identity. This distinction provides a clearer target for future egocentric conversational agents and prevents dense forecasting gains from being mistaken for semantic floor control. \section{Acknowledgment} OpenAI Codex assisted with substantive drafting and figure layout in the abstract, introduction, and presentation of the archived analysis; the authors remain responsible for verification, experimental claims, and the final manuscript. \bibliographystyle{IEEEbib} \bibliography{references} \end{document}