Add targeted ICASSP manuscript draft

2026-09-23 07:54:36 +08:00
parent dcbf60a360
commit ba162f44a0
9 changed files with 643 additions and 0 deletions
@@ -0,0 +1,6 @@
 build/
 figures/*.aux
 figures/*.fdb_latexmk
 figures/*.fls
 figures/*.log
 figures/*.synctex.gz
@@ -0,0 +1,16 @@
 LATEXMK ?= latexmk
 .PHONY: all clean figure
 all: figures/system_overview.pdf
 	mkdir -p build
 	$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
 figure: figures/system_overview.pdf
 figures/system_overview.pdf: figures/system_overview.tex
 	$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
 clean:
 	$(LATEXMK) -C -outdir=build main.tex
 	$(LATEXMK) -c -outdir=figures figures/system_overview.tex
@@ -0,0 +1,44 @@
 # Targeted ICASSP 2027 manuscript
 This directory is an independent acceptance-target manuscript. It does not
 modify or replace `../main/`.
 ## Positioning
 The paper is deliberately framed as a causal global--local **audit**, because
 that is the strongest complete story supported by archived evidence:
 - global Omni context and recent audio are complementary across all horizons;
 - most measured benefit is dense forecasting and wearer-offset anticipation;
 - pooled and current-frame person vision do not establish speaker grounding;
 - a three-seed causal oracle shows that timely visible-participant activity is
  informative, while identity continuity is not currently necessary.
 No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
 claim is made. The deployable matrix is explicitly single-seed and exploratory.
 ## Build
 ```bash
 cd manuscript/targeted
 make
 ```
 The editable sources are `main.tex`, `references.bib`, and
 `figures/system_overview.tex`. Generated files are written to `build/`.
 The build currently uses the repository's unmodified ICASSP 2026 style assets
 as a compatibility preview. Replace the `template_dir` in `latexmkrc` with the
 official ICASSP 2027 author kit before submission and repeat all page/layout
 checks.
 ## Evidence sources
 - `results/easycom_encoder_only_1s_4ep_2026-08-28/`
 - `results/easycom_egospeak_three_level_eval_2026-09-16/`
 - `results/easycom_omni_floor_screen_2026-09-16/`
 - `results/easycom_oracle_pooled_wearer_audit_2026-09-10/`
 The manuscript does not use superseded opportunity-conditioned ENTRY/INT
 numbers. See `REVIEWER_REPORT.md` for the internal review loop and the remaining
 scientific submission gates.
@@ -0,0 +1,87 @@
 # Internal ICASSP-style review loop
 ## Review 1: reject in original method framing
 **Main concerns**
 1. The novelty claim was too broad. EgoSpeak already defines egocentric
   when-to-speak prediction, while MuVAP, AV-CONV, and 3M-Transformer cover
   adjacent multiparty audiovisual structure.
 2. The learned visual method did not support the intended speaker-grounding
   claim. Pooled vision gave only `+0.291` mean-mAP points over `G+A`, and the
   current-frame person-set screen did not exceed the historical pooled row.
 3. Dense mAP, raw endpoint AP, and semantic floor events were conflated.
 4. Planned tail, context, transfer, and behavior experiments occupied too much
   of a four-page argument despite being unmeasured.
 **Revision**
 The paper was reframed from a successful visual adapter to a causal
 global--local evidence audit. The claim is now that recent audio and global
 context are complementary, while generic visual access remains insufficient.
 All semantic event numbers and planned cross-backbone claims were removed.
 ## Review 2: weak reject after reframing
 **Strengths**
 - Clear task asymmetry: visible interlocutors condition prediction of an
  invisible wearer.
 - Strict causal cutoff and fixed 5-Hz / ten-horizon contract.
 - A coherent positive result for `G+A`, especially wearer-offset anticipation.
 - An unusually informative negative result: visible-activity labels help, but
  current generic visual features do not recover that signal.
 - Oracle controls distinguish visibility, timeliness, aggregation, and
  identity continuity.
 **Remaining concerns**
 1. The deployable main comparison is single seed; only the oracle has three
   seeds.
 2. Sessions 1--3 have been inspected repeatedly, so the current matrix is not
   a clean confirmatory test.
 3. `G+A` changes history, feature access, and trainable capacity relative to
   `A` or `G`; a matched wider/temporal control is missing.
 4. Raw onset/offset AP is not a one-event-one-vote behavioral metric.
 5. Checkpoint hashes, common valid-pair counts, and original predictions are
   not all archived on this host.
 **Revision**
 The abstract and results now identify single-seed versus three-seed evidence,
 avoid significance language, separate endpoint diagnostics from events, and
 state the capacity/history confound. The main paper no longer claims reduced
 false entry, semantic yield, speaker identity, or real-time throughput.
 ## Review 3: current verdict
 **Score: 5/10, borderline, structurally ready but not yet confirmatory.**
 The manuscript is substantially clearer and more defensible than a paper that
 claims a successful visual grounding method. Its strongest publishable insight
 is the controlled mismatch between oracle information value and learned visual
 access. However, writing quality cannot replace the missing confirmatory
 evidence. Calling the current artifact "accepted-paper ready" would overstate
 the science.
 ## Minimum gates for a credible accept recommendation
 1. Re-run `A`, `G`, `G+A`, and the selected visual condition for seeds 0/1/2
   under one frozen recipe and common valid pairs; archive predictions,
   checkpoint/config hashes, per-session values, and trainable parameter counts.
 2. Add a parameter- and history-matched control. The central `G+A` gain must
   remain larger than seed variance and the matched-capacity alternative.
 3. Run the corrected competing-risk evaluator with training-side calibration,
   manually audit real timelines, and report matched-recall false-entry and
   false-stop rates only if denominators and invalid counts agree.
 4. If visual grounding remains a contribution, replace generic pooling with a
   causal active-speaker/floor estimator and require correct-time evidence to
   beat delayed, removed, and capacity-matched controls. Otherwise retain the
   honest audio-local paper and remove visual-method language.
 5. Rebuild with the official 2027 kit, keep technical content within four
   pages, put only references on an optional fifth page, and retain the IEEE
   AI-assisted drafting disclosure.
 Passing gates 1--3 would support a solid ICASSP empirical paper. Passing gate
 4 with a meaningful matched-recall behavior gain would support the stronger
 method paper originally envisioned.
@@ -0,0 +1,92 @@
 \documentclass[tikz,border=2pt]{standalone}
 \usepackage{amsmath}
 \usetikzlibrary{arrows.meta,calc,positioning}
 \definecolor{navy}{HTML}{19324A}
 \definecolor{bluefill}{HTML}{EAF1F7}
 \definecolor{teal}{HTML}{0B7A75}
 \definecolor{tealfill}{HTML}{E5F4F2}
 \definecolor{orange}{HTML}{C96A12}
 \definecolor{orangefill}{HTML}{FFF0DF}
 \definecolor{purple}{HTML}{6E4B8B}
 \definecolor{purplefill}{HTML}{F1EBF6}
 \definecolor{grayline}{HTML}{697782}
 \begin{document}
 \begin{tikzpicture}[
 font=\sffamily\scriptsize, >=Latex,
 flow/.style={-{Latex[length=1.7mm]},line width=0.65pt,draw=navy},
 localflow/.style={flow,draw=teal},
 auditflow/.style={flow,draw=purple,dashed},
 box/.style={draw=navy,rounded corners=1.5pt,line width=0.65pt,
 fill=bluefill,minimum height=8mm,align=center,inner sep=2.2pt},
 localbox/.style={box,draw=teal,fill=tealfill},
 auditbox/.style={box,draw=purple,fill=purplefill,dashed},
 note/.style={font=\sffamily\tiny,text=grayline,align=center}
 ]
 \node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
 at (-0.1,3.2) {Prediction path: integrated history plus explicit recent evidence};
 \node[align=center,text=navy] (av) at (0.7,2.45)
 {observed full-scene\\audio--video prefix};
 \node[box] (omni) at (3.25,2.45) {frozen Qwen2.5-Omni\\8-s maximum context};
 \node[box] (global) at (5.95,2.45) {last-four-layer\\global state $G_n$};
 \node[align=center,text=teal] (audio) at (0.7,1.23)
 {most recent 1-s\\mixed audio};
 \node[localbox] (ae) at (3.25,1.23) {frozen audio encoder\\valid-token mean};
 \node[localbox] (ap) at (5.95,1.23) {local projection\\$\widetilde A_n$ + validity};
 \node[align=center,text=teal] (video) at (0.7,0.02)
 {recent causal\\person crops};
 \node[localbox,dashed] (ve) at (3.25,0.02)
 {frozen visual encoder\\crop / person pooling};
 \node[localbox,dashed] (vp) at (5.95,0.02)
 {optional projection\\$\widetilde V_n$ + validity};
 \node[box,minimum width=21mm] (fusion) at (8.65,1.23)
 {cutoff-aligned\\fusion head};
 \node[box,minimum width=24mm] (out) at (11.55,1.23)
 {$10\!\times\!3$ activity forecast\\0.2--2.0 s};
 \draw[flow] (av) -- (omni);
 \draw[flow] (omni) -- (global);
 \draw[localflow] (audio) -- (ae);
 \draw[localflow] (ae) -- (ap);
 \draw[localflow,dashed] (video) -- (ve);
 \draw[localflow,dashed] (ve) -- (vp);
 \draw[flow] (global.east) -- ++(0.30,0) |- (fusion.north);
 \draw[localflow] (ap) -- (fusion);
 \draw[localflow,dashed] (vp.east) -- ++(0.30,0) |- (fusion.south);
 \draw[flow] (fusion) -- (out);
 \node[note] at (11.45,0.18)
 {background / invisible wearer / aggregate others};
 \draw[grayline,line width=0.45pt] (-0.1,-0.62) -- (12.9,-0.62);
 \node[anchor=west,font=\sffamily\scriptsize\bfseries,text=navy]
 at (-0.1,-0.94) {Strictly causal stream at cutoff $t_n$};
 \node[anchor=east,text=navy] at (2.05,-1.62) {Omni media};
 \foreach \a/\b in {2.3/3.85,3.85/5.4,5.4/6.95} {
 \draw[fill=bluefill,draw=navy,line width=0.55pt]
 (\a,-1.84) rectangle (\b,-1.40);
 \node[text=navy,font=\sffamily\tiny] at ({(\a+\b)/2},-1.62) {completed 2 s};
 }
 \draw[fill=orangefill,draw=orange,line width=0.65pt]
 (6.95,-1.84) rectangle (8.45,-1.40);
 \node[text=orange,font=\sffamily\tiny] at (7.70,-1.62) {observed tail};
 \draw[dashed,draw=orange,line width=0.75pt]
 (8.45,-2.95) -- (8.45,-1.10);
 \node[anchor=south,text=orange,font=\sffamily\tiny\bfseries]
 at (8.45,-1.10) {cutoff $t_n$};
 \node[note,anchor=west] at (8.85,-1.62)
 {replay completed payloads; recompute only\\the temporary tail; exclude future media};
 \node[anchor=east,text=teal] at (2.05,-2.31) {local evidence};
 \draw[draw=teal,line width=1.8pt] (7.65,-2.31) -- (8.45,-2.31);
 \node[note,anchor=west,text=teal] at (8.85,-2.31)
 {one-second causal window, refreshed every 200 ms};
 \node[anchor=east,text=navy] at (2.05,-2.86) {decision grid};
 \draw[draw=navy,line width=0.6pt] (2.3,-2.86) -- (8.45,-2.86);
 \foreach \x in {2.42,2.58,...,8.34} {
 \draw[draw=navy,line width=0.5pt] (\x,-2.95) -- (\x,-2.77);
 }
 \node[auditbox,minimum width=42mm] (oracle) at (11.02,-2.78)
 {analysis only: past visible-person activity\\aggregate / per-person / delayed / shuffled};
 \draw[auditflow] (oracle.north) -- ++(0,0.32) -| (fusion.south);
 \node[note,text=purple] at (11.02,-3.35)
 {privileged information audit; never a deployable model input};
 \end{tikzpicture}
 \end{document}
@@ -0,0 +1,9 @@
 my $path_separator = $^O eq 'MSWin32' ? ';' : ':';
 my $template_dir = '../ICASSP2026_Paper_Templates';
 $ENV{'TEXINPUTS'} =
    $template_dir . $path_separator . ($ENV{'TEXINPUTS'} // '');
 $ENV{'BSTINPUTS'} =
    $template_dir . $path_separator . ($ENV{'BSTINPUTS'} // '');
 $out_dir = 'build';
@@ -0,0 +1,326 @@
 % Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
 % The official 2027 author kit should replace the 2026-compatible style assets
 % before submission. Four technical pages plus an optional reference-only page.
 \documentclass{article}
 \usepackage{spconf}
 \usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor}
 \hypersetup{hidelinks}
 \newcommand{\vect}[1]{\boldsymbol{#1}}
 \newcommand{\G}{\mathrm{G}}
 \newcommand{\A}{\mathrm{A}}
 \newcommand{\V}{\mathrm{V}}
 \setlength{\emergencystretch}{2em}
 \title{Does an Omni Model Know When the Wearer Will Speak?\\
 A Causal Global--Local Audit of Egocentric Speech Anticipation}
 \name{Authors withheld in this working draft}
 \address{}
 \begin{document}
 \ninept
 \maketitle
 \begin{abstract}
 An egocentric conversational agent must anticipate the future speech of an
 invisible camera wearer from a causal multiparty audio--video stream.  We ask
 whether a frozen general-purpose Omni state exposes the sub-second evidence
 needed by this task.  Our cutoff-aligned model combines an 8-s Qwen2.5-Omni
 context with the most recent 1-s audio and predicts three activity channels at
 ten horizons from 0.2 to 2.0~s.  In a single-seed EasyCom screen, this model reaches
 81.72\% mAP at 0.2~s and 65.11\% averaged over horizons, compared with
 74.20\% and 61.19\% for released EgoSpeak.  The gain is larger for wearer
 offset anticipation (+7.23 points) than onset anticipation (+1.47).  Pooled
 visual features add only 0.29 mean-mAP points.  A separate three-seed,
 strictly causal information audit explains this gap: privileged past activity
 of visible participants adds 4.83 points at 0.2~s and 1.77 mean-mAP points,
 whereas visibility, one-second-delayed activity, and identity shuffling show
 that timeliness matters more than identity continuity.  The results isolate an
 evidence-access bottleneck: recent acoustic evidence is already useful, while
 current generic visual pooling does not recover the available visible-floor
 signal.
 \end{abstract}
 \begin{keywords}
 egocentric speech anticipation, multimodal foundation models, causal
 streaming, voice activity projection, conversational agents
 \end{keywords}
 \section{Introduction}
 \label{sec:intro}
 Conversational agents must decide not only \emph{what} to say, but also
 \emph{when} to participate.  This decision is particularly asymmetric in an
 egocentric stream: the camera wearer is normally invisible, yet the model must
 forecast that wearer's behavior from mixed audio and observations of other
 people.  A useful predictor must update before a turn boundary is known and
 must not use media from after its decision cutoff.
 Voice Activity Projection (VAP) casts turn taking as continuous prediction of
 future activity \cite{ekstedt2022vap}.  EgoSpeak brings this objective to
 untrimmed first-person video and defines the EasyCom protocol used here
 \cite{kim2025egospeak,donley2021easycom}.  Other work predicts future speakers
 from synchronized egocentric sensors \cite{serj2024threem}, reconstructs
 egocentric conversational graphs \cite{jia2024avconv}, or grounds causal
 multiparty projections in visible face tracks \cite{qi2026muvap}.  Those
 systems establish that neither multiparty audiovisual modeling nor
 first-person speech initiation is itself a novelty.  Our question is narrower:
 does a general-purpose Omni readout make timely evidence accessible for
 forecasting the unseen wearer?
 Qwen2.5-Omni integrates long-form audio and video in a Thinker representation
 \cite{xu2025qwenomni}.  Such integration is attractive for conversational
 context, but a blockwise state can blur or delay the short acoustic and visual
 cues needed at 200-ms intervals.  We therefore separate an integrated global
 state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
 their value as an empirical question rather than assuming that a larger
 backbone solves temporal access.
 This paper makes three contributions.  First, we formulate a strictly causal
 global--local audit for invisible-wearer activity forecasting, with a common
 5-Hz cutoff and ten future horizons.  Second, we realize the audit with a
 bounded Omni stream that combines completed history with a temporary observed
 tail and a separate one-second audio path.  Third, we distinguish deployable
 feature gains from an analysis-only visible-activity oracle.  This reveals a
 specific negative result---generic pooled or current-frame person features do
 not recover most of the available local-floor information---and identifies
 timely activity estimation, rather than identity continuity, as the next
 modeling target.
 \begin{figure*}[t]
 \centering
 \includegraphics[width=0.97\textwidth]{figures/system_overview.pdf}
 \caption{Global--local causal audit.  At each 200-ms cutoff, the prediction
 path fuses an integrated Omni state with explicit recent audio and optional
 person-crop vision.  Completed 2-s blocks are reused while only the observed
 partial tail is temporarily recomputed.  The purple branch is a privileged
 past-activity audit used to test information value; it is never a deployable
 input and contains no future activity.}
 \label{fig:overview}
 \end{figure*}
 \section{Method}
 \label{sec:method}
 \subsection{Task and Causal Inputs}
 At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
 predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s.  Each
 horizon has three channels in fixed order: background, camera wearer, and
 aggregate others.  Wearer and others may overlap; no output identifies an
 individual speaker, addressee, or semantic end of turn.
 The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B.  We
 concatenate the final four Thinker readout states and project them to 2048
 dimensions.  The readout contains both audio and video context; $\G$ is not an
 audio-free condition.  The local audio feature $\vect a_n$ is obtained by
 cropping the most recent one-second mixed waveform \emph{before}
 preprocessing, encoding it with the frozen Qwen audio encoder, and averaging
 valid output tokens.  An optional visual feature $\vect v_n$ averages spatial
 tokens inside provided person boxes, people within each causal frame, and
 valid frames from the most recent second.  It preserves neither identity nor
 mouth-motion trajectories, and provided boxes do not constitute an online
 detector.
 Each enabled local feature is layer-normalized, projected to 256 dimensions,
 and paired with a validity bit.  The trainable head is
 \begin{equation}
 \widehat{\vect Y}_{n}=
 \operatorname{softmax}_{c}F_{\theta}
 \bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a;
       \widetilde{\vect v}_n;m_n^v\bigr],
 \label{eq:fusion}
 \end{equation}
 where omitted modalities remove their terms and $F_\theta$ is a
 layer-normalized MLP with a 30-logit output.  We train only the projections and
 head with the EgoSpeak multi-positive cross-entropy over causal-valid
 source--horizon pairs.  The Omni and modality encoders remain frozen, so this
 is frozen-backbone adaptation rather than end-to-end training.
 \subsection{Cutoff-Aligned Streaming}
 The global stream uses source-aligned 2-s blocks and an 8-s maximum
 presentation budget.  After warm-up, three completed blocks and the observed
 part of the current block are available; the exact span varies with block
 phase.  Completed modality payloads are reused when the bounded language-model
 cache is rebuilt.  At every cutoff, the incomplete current block is cropped at
 $t_n$, appended to a temporary cache, and queried for $\vect g_n$.  Neither
 the temporary tail nor the query is committed.  At a block boundary the
 completed payload is committed once and a new tail begins.  Sessions and
 source discontinuities reset all caches.  Thus the logical state is refreshed
 at 5~Hz without backfilling a completed block into earlier rows.
 \subsection{Visible-Activity Information Audit}
 To distinguish missing information from ineffective learned vision, we build
 analysis-only features from transcript-derived \emph{past} activity.  At each
 cutoff, visible non-wearer participants receive two seconds of binary history
 sampled on the same 5-Hz grid.  The preparation fails if its aggregate
 activity disagrees with the released target.  We compare: (i) visibility with
 activity zeroed, (ii) per-person histories, (iii) identities shuffled while
 preserving active-person counts at every lag, (iv) histories delayed by one
 second, and (v) one OR-pooled aggregate-visible history.  These conditions
 never expose future activity, but their speech labels make them privileged
 diagnostics, not deployable features or upper bounds.
 \section{Experiments}
 \label{sec:experiments}
 \textbf{Data and protocol.}
 We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training
 and Sessions~1--3 for testing.  Inputs and targets share a 5-Hz grid.  Heads
 are trained for four fixed epochs on all training sessions without test-time
 checkpoint selection.  The deployable feature matrix is an archived
 single-seed screen; the information audit uses seeds 0, 1, and 2 with an
 identical fixed recipe.  We therefore report standard deviations only for the
 latter and treat small differences in the former descriptively.
 \textbf{Metrics.}
 At each horizon we compute AP for the three activity channels over common
 causal-valid rows, then average classes to obtain mAP and average all ten
 horizons to obtain mean mAP.  We also report wearer AP at 0.2~s.  For endpoint
 diagnostics, onset AP conditions source rows on the wearer being silent and
 scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
 speech and scores future silence with $1-p(\mathrm{wearer})$.  These
 threshold-free quantities average ten horizons.  They are not one-event-one-
 vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
 \textbf{Comparisons.}
 We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni
 $\G$; $\G+\A$; and $\G+\A+$pooled-$\V$.  A three-seed current-frame
 frame-set attention screen tests whether a richer person aggregation is
 already sufficient.  Because that screen used Session~12 model selection
 whereas the historical pooled row did not, their difference is diagnostic,
 not a matched causal attribution.
 \section{Results}
 \label{sec:results}
 \begin{table*}[t]
 \caption{EasyCom Sessions~1--3 results (\%).  $\mathrm{mAP}_{.2}$ and
 $\mathrm{WAP}_{.2}$ are dense 0.2-s metrics; Mean averages dense mAP over ten
 horizons.  On/Off are raw wearer endpoint AP, not semantic event scores.
 Deployable rows are archived single-seed screens; $\pm$ is shown only for the
 three-seed F1 screen.  Dashes denote unavailable measurements.}
 \label{tab:main}
 \centering
 \scriptsize
 \setlength{\tabcolsep}{5.0pt}
 \begin{tabular}{@{}lccccc@{}}
 \toprule
 Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
 & Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
 \midrule
 EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
 Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
 Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
 Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
 Omni $\G+\A+$pooled-$\V$ & \textbf{82.027} & \textbf{85.086}
 & \textbf{65.400} & \textbf{26.019} & 45.371 \\\\
 Current-frame person set F1 & $81.777{\pm}0.156$ & --
 & $64.951{\pm}0.095$ & -- & -- \\\\
 \bottomrule
 \end{tabular}
 \end{table*}
 \subsection{Global Context and Recent Audio Are Complementary}
 Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
 slightly stronger when mAP is averaged to 2.0~s.  Their fusion is better than
 $\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at
 0.2~s and 4.47 at 2.0~s.  It also improves every class when averaged over
 horizons (+8.72 background, +2.43 wearer, and +2.65 others).  Relative to
 released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by
 3.92 points.  The endpoint effect is asymmetric: offset AP rises by 7.23
 points, while onset AP rises by 1.47.  Dense forecasting gains therefore do
 not imply equal progress at both behavior boundaries.
 The global-only head has more trainable parameters and a longer context than
 the local-only head, so this comparison demonstrates predictive
 complementarity, not that semantic reasoning alone causes the gain.  A wider
 history-matched control is required for a representation-level causal claim.
 \subsection{Generic Vision Does Not Recover the Available Signal}
 Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
 +0.291 mean mAP.  Its wearer increment is +0.020 at 0.2~s and +0.061 when
 averaged over horizons; onset/offset changes are +0.059/+0.120.  A richer
 current-frame person-set model is stable across seeds but reaches lower mean
 mAP than the historical pooled row under a non-matched selection protocol.
 These results do not support a claim of learned speaker grounding.
 \begin{table}[t]
 \caption{Three-seed privileged-information audit (mean $\pm$ sample standard
 deviation, \%).  Every row includes $\G+\A$; activity conditions use only
 past labels.  Bold marks the best diagnostic, not a deployable model.}
 \label{tab:oracle}
 \centering
 \scriptsize
 \setlength{\tabcolsep}{3.2pt}
 \begin{tabular}{@{}lccc@{}}
 \toprule
 Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
 \midrule
 None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
 Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
 Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
 Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
 Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
 Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
 & $\mathbf{66.951{\pm}.022}$ \\\\
 \bottomrule
 \end{tabular}
 \end{table}
 Table~\ref{tab:oracle} locates the missing information.  Aggregate visible
 activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
 mean-mAP points.  Visibility alone changes mean mAP by only +0.040, and
 one-second-delayed activity by +0.164.  Thus availability or extra parameters
 cannot explain the oracle gain.  Per-person activity exceeds shuffled identity
 by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
 condition.  The current target therefore supports timely visible-floor
 activity as useful information, but not the necessity of identity continuity.
 The majority of the oracle's 0.2-s macro gain comes from background and
 others; wearer AP rises by only 0.572.  This is why neither the oracle nor a
 dense mAP gain establishes fewer false wearer entries.
 \subsection{What the Audit Establishes}
 Three conclusions survive the controls.  First, a general-purpose global
 state and a one-second acoustic readout provide different useful timescales.
 Second, generic visual pooling has not converted visible people into reliable
 local floor evidence.  Third, perfect timely aggregate activity contains
 recoverable information, whereas identity contributes little under the
 current target.  A dedicated causal active-speaker front end---designed for
 mouth motion, blur, and short audio--visual synchrony as in egocentric ASD
 \cite{huh2025egoasd}---is consequently a better-motivated next step than a
 larger identity-aware fusion module.
 The evidence remains bounded.  EasyCom has only three official test sessions;
 the deployable matrix is single seed and has been examined repeatedly.  The
 oracle uses labels and is not a performance upper bound.  Provided boxes omit
 online detection cost, and logical 5-Hz output is not a measured throughput
 claim.  Finally, aggregate others cannot supply addressee or semantic
 interruption ground truth.  Confirmatory behavioral claims require a frozen
 training-side operating point, competing-risk event attribution, and
 multi-seed matched controls.
 \section{Conclusion}
 We presented a strictly causal global--local audit of egocentric wearer speech
 anticipation.  Frozen Omni context and recent audio are strongly complementary,
 especially for future wearer cessation.  In contrast, pooled or current-frame
 person features do not recover the local-floor signal revealed by a causal
 visible-activity oracle.  The actionable conclusion is deliberately narrow:
 the next bottleneck is timely, learnable local activity evidence, not simply
 more multimodal capacity or persistent identity.  This distinction provides a
 clearer target for future egocentric conversational agents and prevents dense
 forecasting gains from being mistaken for semantic floor control.
 \section{Acknowledgment}
 OpenAI Codex assisted with substantive drafting and figure layout in the
 abstract, introduction, and presentation of the archived analysis; the
 authors remain responsible for verification, experimental claims, and the
 final manuscript.
 \bibliographystyle{IEEEbib}
 \bibliography{references}
 \end{document}
@@ -0,0 +1,63 @@
@inproceedings{ekstedt2022vap,
  author    = {Erik Ekstedt and Gabriel Skantze},
  title     = {{Voice Activity Projection: Self-Supervised Learning of Turn-Taking Events}},
  booktitle = {Proc. Interspeech},
  pages     = {5190--5194},
  year      = {2022},
  doi       = {10.21437/Interspeech.2022-10955}
 }
@inproceedings{kim2025egospeak,
  author    = {Junhyeok Kim and Min Soo Kim and Jiwan Chung and Jungbin Cho and Jisoo Kim and Sungwoong Kim and Gyeongbo Sim and Youngjae Yu},
  title     = {{EgoSpeak}: Learning When to Speak for Egocentric Conversational Agents in the Wild},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL},
  pages     = {2990--3005},
  year      = {2025},
  doi       = {10.18653/v1/2025.findings-naacl.163}
 }
@article{donley2021easycom,
  author  = {Jacob Donley and Vladimir Tourbabin and Jung-Suk Lee and Mark Broyles and Hao Jiang and Jie Shen and Maja Pantic and Vamsi Krishna Ithapu and Ravish Mehra},
  title   = {{EasyCom}: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments},
  journal = {arXiv preprint arXiv:2107.04174},
  year    = {2021}
 }
@article{xu2025qwenomni,
  author  = {Jin Xu and Zhifang Guo and Jinzheng He and Hangrui Hu and Ting He and Shuai Bai and Keqin Chen and Jialin Wang and Yang Fan and Kai Dang and Bin Zhang and Xiong Wang and Yunfei Chu and Junyang Lin},
  title   = {{Qwen2.5-Omni Technical Report}},
  journal = {arXiv preprint arXiv:2503.20215},
  year    = {2025}
 }
@inproceedings{serj2024threem,
  author    = {Mehdi Fatan Serj and Emanuele Mincato and Dimitra Pintzou and Mariella Dimiccoli},
  title     = {{3M-Transformer}: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction},
  booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages     = {8050--8054},
  year      = {2024},
  doi       = {10.1109/ICASSP48485.2024.10448136}
 }
@inproceedings{jia2024avconv,
  author    = {Wenqi Jia and Miao Liu and Hao Jiang and Ishwarya Ananthabhotla and James M. Rehg and Vamsi Krishna Ithapu and Ruohan Gao},
  title     = {The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages     = {26396--26405},
  year      = {2024}
 }
@article{qi2026muvap,
  author  = {Haotian Qi and Gabriel Skantze},
  title   = {{MuVAP}: Multimodal Multiparty Voice Activity Projection for Turn-Taking Prediction in the Wild},
  journal = {arXiv preprint arXiv:2606.16731},
  year    = {2026}
 }
@inproceedings{huh2025egoasd,
  author    = {Jaesung Huh and Juan Azcarreta Ortiz and Anurag Kumar and Ashutosh Pandey and Ali Aroudi and Daniel D. E. Wong and Francesco Nesta and Buye Xu and Jacob Donley},
  title     = {Advancing Active Speaker Detection for Egocentric Videos},
  booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year      = {2025},
  doi       = {10.1109/ICASSP49660.2025.10888166}
 }