Document global-local Omni paper position and oracle evidence

manuscript/main/main.tex

@@ -10,8 +10,8 @@
 \newcommand{\vect}[1]{\boldsymbol{#1}}
 \setlength{\emergencystretch}{2em}
 
-\title{WHEN SHOULD AN EGOCENTRIC CONVERSATIONAL AGENT SPEAK? PROBING AND
-ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION}
+\title{GLOBAL OMNI CONTEXT AND CAUSAL LOCAL AUDIOVISUAL EVIDENCE FOR
+EGOCENTRIC MULTIPARTY FLOOR-ACTION FORECASTING}
 
 % TODO: Insert the final author list and affiliations after the ICASSP 2027
 % review policy and author order are finalized.
@@ -24,26 +24,28 @@ ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION}
 
 \begin{abstract}
 An egocentric conversational agent operating among several people must decide
-whether to speak or keep listening before it generates a response. EgoSpeak
-operationalizes this problem using recordings from human camera wearers:
-causal first-person audio--video forms the observation, and the wearer's future
-speaking behavior forms the action target. The wearer is thus a data-collection
-surrogate rather than the deployed system. EgoSpeak evaluates
-separately extracted audio--visual features with Transformer, GRU, and Mamba
-temporal models; whether end-to-end pretrained omni models that jointly process
-raw audio and video expose the
-fine-grained cues required for a decision every 200~ms remains unclear. We
-probe representative omni backbones using frozen readout,
-parameter-efficient tuning, and causal side branches that preserve recent
-audio and visible-person evidence. The study tests whether a global omni state
-underrepresents other-person activity and whether local evidence compensates
-without making the agent passive. An EasyCom event protocol measures false
-agent entries during multi-other exchanges at matched recall of wearer-derived
-onset targets. Exploratory Qwen2.5-Omni-3B results motivate this hypothesis but
-do not yet establish a cross-model limitation or determine the final adapter.
-\todo{After the train-side study is locked, name the selected backbones and
-adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction
-at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.}
+whether to speak or keep listening before generating a response. Following
+EgoSpeak, we use causal first-person audio--video as observation and the future
+speaking behavior of the human camera wearer as an imitation target for this
+agent decision. We ask whether a jointly pretrained Omni model exposes the
+fine-grained floor cues required every 200~ms. Our global--local formulation
+uses a causal global Omni readout for broad scene and conversational context,
+while a causal local branch preserves recent mixed audio and a variable set of
+track-organized person observations. Anonymous tracks are an organizational
+interface whose incremental value must be established, not an assumption that
+the model knows speaker identity. On EasyCom development data, a single-seed
+oracle analysis finds that perfectly aligned aggregate activity among visible
+non-wearers improves 0.2-s mAP by 6.97 points and ten-horizon mean mAP by 2.91
+points over global Omni plus recent audio. Person-resolved histories do not
+outperform an identity-shuffled control. This result motivates timely local
+floor evidence, while explicitly withholding a person-continuity claim. The
+full study compares frozen, parameter-efficient, and learned local variants
+and evaluates false agent entry at matched wearer-onset recall.
+\todo{After the train-side study is locked, replace the final two sentences
+with the selected learned local variant and insert: its oracle-gap recovery,
+multi-other false-entry reduction at 80\% onset recall, onset/yield
+non-degradation, and 95\% intervals. Mention track continuity only if its
+controls separate.}
 \end{abstract}
 
 \begin{keywords}
@@ -79,26 +81,39 @@ wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba
 temporal models, forecasting the camera wearer and aggregate other-person
 activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is
 not the task definition or the use of first-person data. The unresolved model
-question is whether an end-to-end pretrained audio--visual omni backbone,
+question is whether a jointly pretrained audio--visual Omni backbone,
 operating on causal raw media, exposes an adequate state for this dense
 floor-action controller. To our knowledge, such omni representations have not been
 systematically evaluated or adapted for the EgoSpeak task.
 
-Our working hypothesis is specific. An omni model may retain broad scene and
-speech semantics while its global causal state makes recent prosody, activity
-boundaries, and individual visible-speaker motion difficult for a shallow
-5-Hz head to recover. This would appear as inaccurate prediction of
-other-person activity and poorly timed agent entries, especially in
-multiparty exchanges. A causal local side branch may compensate by preserving
-recent audio and anonymous person-level evidence before fusion with the global
-state. This is a hypothesis to test, not an assumed omni-model deficiency:
-frozen, parameter-efficiently tuned, and locally augmented variants must be
-compared within each backbone.
+Our working hypothesis is a representational decomposition. A global causal
+Omni state may retain broad scene, speech, and semantic context while making
+recent prosody, activity boundaries, and localized person motion difficult for
+a shallow 5-Hz controller to recover. A causal local branch therefore
+preserves recent mixed audio and a variable-size set of person observations
+before fusing them back into the global state. The local set can be pooled
+within a frame or organized into anonymous tracks across time. Whether that
+continuity adds useful information beyond aggregate activity is itself a
+falsifiable question. Frozen, parameter-efficiently tuned, and locally
+augmented variants must consequently be compared within each backbone.
+
+An initial causal oracle screen sharpens this hypothesis. On held-out EasyCom
+Session~12, perfectly timed aggregate activity from currently visible
+non-wearers improves a matched global-plus-audio model by 6.97 points at 0.2~s
+and 2.91 points in mean mAP. Visibility alone and activity delayed by one
+second remain near the audio baseline. However, person-resolved histories are
+weaker than aggregate histories and indistinguishable from histories whose
+identity continuity is shuffled. Thus the current evidence supports the value
+of timely local floor information, not a claim that person identity or track
+continuity has already been exploited.
 
 Voice Activity Projection (VAP) provides the broader future-activity view of
 turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves
 compatibility with that view, but its aggregate ``others'' channel discards
 whether the same or different visible people produced recent activity.
+Our oracle result shows that this discarded distinction is not automatically
+useful under the dense target: identity continuity must beat aggregate and
+shuffled controls before it enters the method claim.
 Moreover, conventional per-frame average precision can hide the behavioral
 failure of entering during an exchange between others. We therefore retain the
 dense EgoSpeak target while adding event evaluation of agent entry and yield;
@@ -121,37 +136,40 @@ direct, continuous agent floor-action prediction and whether causal local
 evidence helps when the visible participant set changes.
 
 We call this task \emph{egocentric floor-action projection}. The study asks
-four questions: (RQ1) how accurately a frozen global omni state predicts the
-wearer-derived agent target and other-person activity; (RQ2) whether standard
-parameter-efficient tuning recovers missing performance; (RQ3) whether causal
-local audio and person-level visual evidence provide complementary
-information; and (RQ4) whether any aggregate gain reduces false agent entries
-without making the controller globally passive. EasyCom is suitable because
+four questions: (RQ1) how accurately a frozen global Omni state predicts the
+wearer-derived agent target and other-person activity; (RQ2) whether timely
+causal local audio and visible-person evidence complement that state; (RQ3)
+whether aggregate person-set evidence is sufficient or anonymous track
+continuity adds measurable value; and (RQ4) whether any dense gain reduces
+false agent entries without making the controller globally passive.
+Parameter-efficient tuning is a matched adaptation control rather than an
+assumed remedy. EasyCom is suitable because
 it provides synchronized egocentric audio--video, per-person tracks, and voice
 activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
 analysis strata only and are never model input or required output.
 
-The intended contributions are therefore: a controlled cross-backbone study
-of pretrained omni representations for EgoSpeak-style floor action; a
-lightweight local-evidence adaptation selected from
-pre-registered candidates; and a behavior-aligned event evaluation that maps
-wearer demonstrations to agent entry and yield decisions. Current
-Qwen2.5-Omni observations are motivation, not a claim that all omni models
-intrinsically fail. Likewise, track-consistent cross-attention becomes the
-final method only if it beats pooled, no-ID, and capacity-matched controls.
+The intended contributions are therefore: a controlled study of global Omni
+representations for EgoSpeak-style floor action; an oracle information audit
+that separates timely aggregate floor evidence from person continuity; a
+global--local adaptation selected from progressively structured candidates;
+and a behavior-aligned event evaluation that maps wearer demonstrations to
+agent entry and yield decisions. Current Qwen2.5-Omni observations are
+motivation, not a claim that all Omni models intrinsically fail. Likewise,
+track-consistent cross-attention becomes the final method only if it beats
+aggregate, shuffled/no-ID, and capacity-matched controls.
 
 \begin{figure}[t]
 \centering
 \fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
-\todo{After model selection, draw the final system figure. For now it must show
-the shared experiment interface: raw causal AV; online face/head detection and
-causal tracking; backbone-specific global readout; recent audio and a masked
-set of anonymous track states; the gated local adapter; and the common
-$10\times3$ head. Mark causal cutoffs, trainable modules, parameter budgets,
-NULL/OFFSCREEN handling, and source-boundary resets.}}}
-\caption{Planned omni-model study for egocentric conversational floor action.
-The local adapter remains a candidate family until the controlled train-side
-comparison is complete.}
+\todo{After model selection, draw the final system figure. Show the shared
+interface: raw causal AV to a global Omni readout; a parallel recent-audio and
+visible-person path; aggregate/no-ID/track-organized variants; residual fusion;
+and the common $10\times3$ head. Draw the transcript-derived oracle only as a
+dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus
+online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}}
+\caption{Global context and causal local evidence for egocentric floor action.
+Track organization is a controlled candidate, not a presupposed source of
+speaker identity.}
 \label{fig:method}
 \end{figure}
 
@@ -207,13 +225,14 @@ expose a comparable causal state, report it as unsupported rather than inventing
 a non-equivalent readout. Add future backbones only before the experiment
 protocol is locked.}
 
-\subsection{Candidate local-evidence adaptation}
+\subsection{Global context and causal local evidence}
 
-The current hypothesis is that global omni states preserve broad semantic
-context but may make recent prosody, activity boundaries, and individual
-visual motion difficult for a shallow floor head to recover. From the most
+The global state $\vect g^{(m)}_t$ is responsible for broad scene and
+conversational context, but is not assumed to retain every localized cue needed
+for floor control. From the most
 recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
-time $t$, an online detector and tracker produce a variable-size anonymous set
+time $t$, provided boxes during method isolation---and an online detector and
+tracker in the deployable condition---produce a variable-size anonymous set
 $\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
 where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
 $\vect r_{t,i}$ contains only causal reliability variables such as visibility,
@@ -231,8 +250,8 @@ current person set by masked cross-attention,
    W_q[\vect g^{(m)}_t;\vect a_t],
    [\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
  \right), \\
- \vect z^{(m)}_t &= \vect g^{(m)}_t+\tanh(\alpha)W_o\vect u_t,
- \qquad \alpha\vert_{\mathrm{init}}=0,
+ \vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t,
+ \qquad W_o\vert_{\mathrm{init}}=0,
  \label{eq:adapter}
 \end{align}
 where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
@@ -243,14 +262,23 @@ reported. The operator is permutation equivariant
 \cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
 identity semantics.
 
+Equation~\eqref{eq:track} is a candidate organization of local evidence, not a
+claim that recurrent identity is necessary. Simpler alternatives replace each
+$\vect h_{t,i}$ with a current-frame observation or pool the set into one
+aggregate vector. This distinction is important because the current oracle
+screen favors aggregate activity and does not separate correct from shuffled
+identity histories.
+
 This decomposition is informed by MuVAP: its global audio representation
 queries same-frame causal ASD track representations, followed by temporal
 modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
-useful separation between global floor dynamics and person-grounded evidence,
+useful separation between global floor dynamics and person-organized evidence,
 but not its role-relative output: our fixed target is the invisible wearer's
-entry/yield action, not the next visible speaker. The zero-initialized gate
-also follows the conservative adapter principle used to inject visual evidence
-into strong audio models \cite{rouditchenko2024whisperflamingo}. Local evidence
+entry/yield action, not the next visible speaker. The zero-initialized residual
+output follows the conservative adapter principle used to inject visual
+evidence without perturbing the base prediction at initialization
+\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits
+the residual output layer to receive a gradient immediately. Local evidence
 is derived from the same causal raw media, not close microphones, transcripts,
 speaker labels, global identity anchors, or future frames.
 
@@ -258,8 +286,10 @@ We evaluate candidates in increasing structural complexity: pooled recent
 audio/vision; a capacity-matched temporal branch; a per-frame person set with
 no cross-time identity; a shared track-wise temporal encoder; and person--audio
 cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
-candidate that improves the pre-registered event metric over simpler and
-capacity-matched controls becomes the final method. A limited latent
+candidate that improves the pre-registered event metric over simpler,
+aggregate, identity-shuffled, and capacity-matched controls becomes the final
+method. If person continuity does not separate from those controls, the final
+model remains a global-plus-local aggregate floor-evidence model. A limited latent
 bottleneck is tested only if large visible sets make full person attention a
 runtime bottleneck; it is not assumed to improve accuracy.
 
@@ -339,6 +369,18 @@ clip-local anonymous participant tracks; its per-person voice coverage on the
 433-clip EgoSpeak split must be audited before reporting the same multi-other
 event analysis.
 
+\textbf{Oracle information audit.}
+Before selecting a learned local module, we measure what information could in
+principle complement the global state. Analysis-only sidecars expose
+transcript-derived speech activity no later than the current cutoff under six
+conditions: local audio, visibility only, one-second-delayed activity, timely
+aggregate visible activity, per-person histories, and histories with their
+cross-time identity assignments shuffled. These labels never enter a
+deployable model. The audit separates three hypotheses that ordinary feature
+ablation conflates: the value of seeing people, the value of knowing current
+visible-other activity, and the incremental value of preserving who produced
+that activity.
+
 \textbf{Backbone comparison.}
 The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
 MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
@@ -383,9 +425,11 @@ confirmatory three-seed run.}
 For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
 plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
 This compact matrix tests the representation hypothesis before investing in a
-complex fusion module. If local vision survives three seeds and the event
-gate, we then compare a parameter-matched temporal branch without person
-continuity, a separately trained per-frame/no-ID set branch, and a
+complex fusion module. The first learned target after the oracle audit is a
+causal estimator of timely aggregate visible-person floor evidence. Only if
+person-organized features survive three seeds, visual time/person corruptions,
+and the event gate do we compare a parameter-matched temporal branch without
+person continuity, a separately trained per-frame/no-ID set branch, and a
 track-consistent masked cross-attention branch. Provided-track and online-track
 inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
 global renaming must leave outputs unchanged, whereas \emph{broken
@@ -419,6 +463,36 @@ after the architecture is frozen.}
 \section{Results and Analysis}
 \label{sec:results}
 
+\begin{table}[t]
+\caption{Causal oracle-information screen on EasyCom development Session~12
+(seed 0, four epochs). Oracle speech activity is analysis-only. Values are
+percentages under the causal-valid protocol.}
+\label{tab:oracle}
+\centering
+\small
+\begin{tabular}{lcc}
+\hline
+Input condition & 0.2-s mAP & Mean mAP\\
+\hline
+Global + local audio              & 72.58 & 58.77\\
+Visibility only                   & 71.95 & 58.86\\
+Activity delayed by 1 s           & 72.02 & 58.90\\
+Aggregate visible activity        & \textbf{79.55} & \textbf{61.68}\\
+Per-person activity histories     & 78.80 & 61.11\\
+Identity-shuffled histories       & 78.86 & 61.19\\
+\hline
+\end{tabular}
+\end{table}
+
+Table~\ref{tab:oracle} establishes an information opportunity, not a deployable
+result. Timely aggregate activity among visible non-wearers provides a large
+gain, while visibility alone and stale activity do not. However, preserving
+person-resolved history gives no advantage over destroying its cross-time
+identity assignment, and both trail the aggregate representation. The learned
+method must therefore first recover useful local floor evidence; person-track
+continuity is retained only if subsequent no-ID and broken-track controls
+separate.
+
 \begin{table}[t]
 \caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
 Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
@@ -441,9 +515,9 @@ $\Delta$ visual & +0.30 & +0.02 & +0.29\\
 
 Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
 Pooled vision gives a small gain, while immediate wearer AP is nearly
-unchanged. This is compatible with either weak but useful social grounding or
-generic visual regularization; only matched-recall event evaluation and
-track-continuity controls can distinguish them.
+unchanged. This is compatible with either useful generic local context or
+regularization; it is not evidence of person grounding. Only matched-recall
+event evaluation and visual-alignment controls can distinguish these cases.
 A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
 readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
 feature protocol, it is not combined numerically with
@@ -522,10 +596,11 @@ rejected PEFT recipes and fusion candidates to supplementary material unless
 they directly explain the selected method.}
 
 \textbf{Qualitative evidence.}
-For a multi-other exchange, visualize per-person speech activity and track IDs,
-wearer ground truth, the frozen-global and selected-local wearer-entry scores,
-and the single frozen threshold on one physical time axis. If the selected
-candidate is track-aware, also show its broken-continuity score. A useful
+For a multi-other exchange, visualize annotation-derived per-person activity
+as an analysis reference, provided track IDs when applicable, wearer ground
+truth, the frozen-global and selected-local wearer-entry scores, and the single
+frozen threshold on one physical time axis. If the selected candidate is
+track-aware, also show its broken-continuity score. A useful
 example shows the frozen baseline rising at an other-speaker boundary while
 the selected adapter remains below threshold; a companion positive example
 must show that a real wearer onset is not suppressed.
@@ -541,17 +616,21 @@ event result and cannot establish it alone.}
 \label{sec:conclusion}
 
 We study when an egocentric agent should enter or yield a multiparty floor, not
-merely whether someone will speak. The central question is model-agnostic:
-whether frozen or parameter-efficiently adapted omni representations expose
-the fine-grained floor cues required by a 5-Hz controller, and whether causal
-local evidence closes any gap. Dense activity forecasting supplies a common
-target, while matched-recall event evaluation tests whether an apparent gain
-actually prevents intrusions during multi-other exchanges. Current Qwen-3B
-results are exploratory motivation only. \todo{After the controlled study,
-state which backbone/adapter was selected, whether the representation pattern
-generalized beyond Qwen, the quantitative false-entry and onset/yield results,
-and the main deployment limitation. Do not infer success from aggregate mAP or
-selected examples alone.}
+merely whether someone will speak. The working decomposition is global Omni
+context plus causal local audiovisual floor evidence. The current
+Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate
+visible-other activity is informative beyond the global state and recent
+mixed audio. It does not support the stronger claim that participant identity
+or track continuity adds value, because per-person histories do not beat an
+identity-shuffled control. Dense activity forecasting supplies a comparable
+target, while matched-recall event evaluation must determine whether a learned
+local estimator actually prevents intrusions during multi-other exchanges
+without suppressing valid wearer entry or yield. \todo{After the controlled
+study, state which learned local variant was selected, how much of the oracle
+gap it recovered, whether the pattern generalized beyond Qwen, the
+quantitative false-entry and onset/yield results, and the main deployment
+limitation. If no learned variant passes these tests, conclude with the
+information audit rather than claiming a track-based solution.}
 
 \bibliographystyle{IEEEbib}
 \bibliography{references}