icassp2027-experiment/manuscript/main/main.tex

% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
% in this repository. Replace the class/style files when the 2027 author kit
% is released, and remove every visible \todo{} marker before submission.
% Compile from this directory with: make
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}

\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
\newcommand{\vect}[1]{\boldsymbol{#1}}
\setlength{\emergencystretch}{2em}

\title{Adapting Omni Models for Egocentric Multiparty Floor-Entry Forecasting}

% TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized.
\name{Internal Manuscript Draft}
\address{}

\begin{document}
\ninept
\maketitle

\begin{abstract}
An egocentric conversational agent operating among several people must decide
whether to speak or keep listening before generating a response. Following
EgoSpeak, we use causal first-person audio--video as observation and the future
speaking behavior of the human camera wearer as an imitation target for this
agent decision. We ask whether a jointly pretrained Omni model exposes the
fine-grained floor cues required every 200~ms. Our global--local formulation
uses a causal global Omni readout for broad scene and conversational context,
while a causal local branch preserves recent mixed audio and a variable set of
track-organized person observations. Anonymous tracks are an organizational
interface whose incremental value must be established, not an assumption that
the model knows speaker identity. On EasyCom development data, a single-seed
oracle analysis finds that perfectly aligned aggregate activity among visible
non-wearers improves 0.2-s mAP by 6.97 points and ten-horizon mean mAP by 2.91
points over global Omni plus recent audio. Person-resolved histories do not
outperform an identity-shuffled control. This result motivates timely local
floor evidence, while explicitly withholding a person-continuity claim. The
full study compares frozen, parameter-efficient, and learned local variants
and evaluates false agent entry at matched wearer-onset recall.
\todo{After the train-side study is locked, replace the final two sentences
with the selected learned local variant and insert: its oracle-gap recovery,
multi-other false-entry reduction at 80\% onset recall, onset/yield
non-degradation, and 95\% intervals. Mention track continuity only if its
controls separate.}
\end{abstract}

\begin{keywords}
conversational agents, turn-taking, speech initiation, egocentric video,
omni models
\end{keywords}

\section{Introduction}
\label{sec:intro}

We consider a conversational agent equipped with an egocentric camera and
microphone and deployed in a scene where several people may speak. Before
response generation and speech synthesis, its causal floor controller must
decide every 200~ms whether to continue listening, begin speaking, or yield. A
pause in dyadic dialogue may invite a response; in multiparty interaction it
may instead mark a handover between two other people. An agent that reacts only
to acoustic availability can therefore interrupt a conversation that it
should continue observing.

Existing datasets do not contain autonomous agents making these decisions at
scale. Following EgoSpeak \cite{kim2025egospeak}, we use human egocentric
recordings to construct supervision: the camera wearer defines the first-person
sensor origin, and the wearer's observed speaking behavior defines the desired
floor-action target. This is an imitation-learning proxy for a future agent
under comparable observations. Throughout the paper, \emph{wearer} names this
dataset role and target channel; it is neither the deployed agent nor a
semantic model input. Response content and closed-loop effects of deploying an
agent remain outside our scope.

EgoSpeak already establishes real-time speech-initiation prediction in causal,
untrimmed egocentric video. It combines separately pretrained ResNet-50 and
wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba
temporal models, forecasting the camera wearer and aggregate other-person
activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is
not the task definition or the use of first-person data. The unresolved model
question is whether a jointly pretrained audio--visual Omni backbone,
operating on causal raw media, exposes an adequate state for this dense
floor-action controller. To our knowledge, such omni representations have not been
systematically evaluated or adapted for the EgoSpeak task.

Our working hypothesis is a representational decomposition. A global causal
Omni state may retain broad scene, speech, and semantic context while making
recent prosody, activity boundaries, and localized person motion difficult for
a shallow 5-Hz controller to recover. A causal local branch therefore
preserves recent mixed audio and a variable-size set of person observations
before fusing them back into the global state. The local set can be pooled
within a frame or organized into anonymous tracks across time. Whether that
continuity adds useful information beyond aggregate activity is itself a
falsifiable question. Frozen, parameter-efficiently tuned, and locally
augmented variants must consequently be compared within each backbone.

An initial causal oracle screen sharpens this hypothesis. On held-out EasyCom
Session~12, perfectly timed aggregate activity from currently visible
non-wearers improves a matched global-plus-audio model by 6.97 points at 0.2~s
and 2.91 points in mean mAP. Visibility alone and activity delayed by one
second remain near the audio baseline. However, person-resolved histories are
weaker than aggregate histories and indistinguishable from histories whose
identity continuity is shuffled. Thus the current evidence supports the value
of timely local floor information, not a claim that person identity or track
continuity has already been exploited.

Voice Activity Projection (VAP) provides the broader future-activity view of
turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves
compatibility with that view, but its aggregate ``others'' channel discards
whether the same or different visible people produced recent activity.
Our oracle result shows that this discarded distinction is not automatically
useful under the dense target: identity continuity must beat aggregate and
shuffled controls before it enters the method claim.
Moreover, conventional per-frame average precision can hide the behavioral
failure of entering during an exchange between others. We therefore retain the
dense EgoSpeak target while adding event evaluation of agent entry and yield;
event-oriented work likewise emphasizes recall, false alarms, and latency
rather than a single frame-level score \cite{jiang2026turnbench}.

Related work supplies complementary parts of the problem. Egocentric
talking-to-me detection predicts whether a completed utterance addresses the
camera wearer and benefits from long-term social context
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap},
while AV-CONV reconstructs current person-level speaking/listening relations
from egocentric audio--video \cite{jia2024avconv}. The former talking-to-me
task is generally utterance-level rather than continuous anticipation;
MuVAP predicts visible participants from a static third-person view; and
AV-CONV estimates a current interaction graph rather than strictly causal
future wearer action. Our gap is therefore narrower than ``audio--visual
multiparty turn taking'': we study whether omni representations support
direct, continuous agent floor-action prediction and whether causal local
evidence helps when the visible participant set changes.

We call this task \emph{egocentric floor-action projection}. The study asks
four questions: (RQ1) how accurately a frozen global Omni state predicts the
wearer-derived agent target and other-person activity; (RQ2) whether timely
causal local audio and visible-person evidence complement that state; (RQ3)
whether aggregate person-set evidence is sufficient or anonymous track
continuity adds measurable value; and (RQ4) whether any dense gain reduces
false agent entries without making the controller globally passive.
Parameter-efficient tuning is a matched adaptation control rather than an
assumed remedy. EasyCom is suitable because
it provides synchronized egocentric audio--video, per-person tracks, and voice
activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
analysis strata only and are never model input or required output.

The intended contributions are therefore: a controlled study of global Omni
representations for EgoSpeak-style floor action; an oracle information audit
that separates timely aggregate floor evidence from person continuity; a
global--local adaptation selected from progressively structured candidates;
and a behavior-aligned event evaluation that maps wearer demonstrations to
agent entry and yield decisions. Current Qwen2.5-Omni observations are
motivation, not a claim that all Omni models intrinsically fail. Likewise,
track-consistent cross-attention becomes the final method only if it beats
aggregate, shuffled/no-ID, and capacity-matched controls.

\begin{figure}[t]
\centering
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
\todo{After model selection, draw the final system figure. Show the shared
interface: raw causal AV to a global Omni readout; a parallel recent-audio and
visible-person path; aggregate/no-ID/track-organized variants; residual fusion;
and the common $10\times3$ head. Draw the transcript-derived oracle only as a
dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus
online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}}
\caption{Global context and causal local evidence for egocentric floor action.
Track organization is a controlled candidate, not a presupposed source of
speaker identity.}
\label{fig:method}
\end{figure}

\section{Methods}
\label{sec:method}

\subsection{Wearer-relative activity projection}

At decision time $t$, the model observes only the causal stream
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
For bin $k$, the target
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed; hence this is not a three-way softmax. The
output is
\begin{equation}
  \hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
  \qquad q_t=\hat y^{\mathrm{w}}_{t,1},
  \label{eq:task}
\end{equation}
where $q_t$ is the immediate wearer-entry score used by the event protocol.
It is interpreted as an \emph{entry} score only in wearer-silent opportunity
windows; elsewhere it remains a future-activity probability. This retains the
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
signal.

\subsection{Omni backbones and representation protocols}

Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a
backbone-specific causal global readout at the same physical cutoff $t$.
Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family;
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5
\cite{cui2026minicpmo45} form the planned initial comparison. These models have
different native streaming APIs, temporal tokenization, hidden dimensions, and
adaptation interfaces. We therefore emphasize within-backbone deltas rather
than treating raw cross-model scores as a pure architecture ranking.

Every supported backbone is evaluated under three representation protocols:
(i) a frozen global readout with a dimension-matched prediction head;
(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the
same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak
targets, context duration, training examples, head capacity, and evaluation
are held fixed whenever the backbone permits it. A failure of one LoRA recipe
is reported as that recipe's result, not as proof that the backbone cannot
adapt.

\todo{Implement and audit one adapter for each planned backbone. For every
model, document model revision, processor, native audio/video rates, exact
hidden-state readout, causal/streaming semantics, context limit, dtype,
trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
expose a comparable causal state, report it as unsupported rather than inventing
a non-equivalent readout. Add future backbones only before the experiment
protocol is locked.}

\subsection{Global context and causal local evidence}

The global state $\vect g^{(m)}_t$ is responsible for broad scene and
conversational context, but is not assumed to retain every localized cue needed
for floor control. From the most
recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
time $t$, provided boxes during method isolation---and an online detector and
tracker in the deployable condition---produce a variable-size anonymous set
$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
track age, and time since observation. A shared causal track encoder updates
\begin{equation}
 \vect h_{t,i}=F_{\mathrm{trk}}\!\left(\vect h^-_{t,i},
 W_v\vect v_{t,i}+W_b\phi(\vect b_{t,i})+W_r\vect r_{t,i}\right).
 \label{eq:track}
\end{equation}
All tracks share $F_{\mathrm{trk}}$; the temporary ID only retrieves
$\vect h^-_{t,i}$ and is never embedded. The global-audio query then reads the
current person set by masked cross-attention,
\begin{align}
 \vect u_t &= \mathrm{MHA}\!\left(
   W_q[\vect g^{(m)}_t;\vect a_t],
   [\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
 \right), \\
 \vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t,
 \qquad W_o\vert_{\mathrm{init}}=0,
 \label{eq:adapter}
\end{align}
where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
masked, so the same network handles $N_t=0$ and changing group size. For a
bounded implementation, $N_t>N_{\max}$ is resolved by a deterministic ranking
using detector confidence and track age available at $t$, and truncation is
reported. The operator is permutation equivariant
\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
identity semantics.

Equation~\eqref{eq:track} is a candidate organization of local evidence, not a
claim that recurrent identity is necessary. Simpler alternatives replace each
$\vect h_{t,i}$ with a current-frame observation or pool the set into one
aggregate vector. This distinction is important because the current oracle
screen favors aggregate activity and does not separate correct from shuffled
identity histories.

This decomposition is informed by MuVAP: its global audio representation
queries same-frame causal ASD track representations, followed by temporal
modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
useful separation between global floor dynamics and person-organized evidence,
but not its role-relative output: our fixed target is the invisible wearer's
entry/yield action, not the next visible speaker. The zero-initialized residual
output follows the conservative adapter principle used to inject visual
evidence without perturbing the base prediction at initialization
\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits
the residual output layer to receive a gradient immediately. Local evidence
is derived from the same causal raw media, not close microphones, transcripts,
speaker labels, global identity anchors, or future frames.

We evaluate candidates in increasing structural complexity: pooled recent
audio/vision; a capacity-matched temporal branch; a per-frame person set with
no cross-time identity; a shared track-wise temporal encoder; and person--audio
cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
candidate that improves the pre-registered event metric over simpler,
aggregate, identity-shuffled, and capacity-matched controls becomes the final
method. If person continuity does not separate from those controls, the final
model remains a global-plus-local aggregate floor-evidence model. A limited latent
bottleneck is tested only if large visible sets make full person attention a
runtime bottleneck; it is not assumed to improve accuracy.

\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
parameters and effective local history. Compare a shared local encoder across
backbones (cross-model control) with each backbone's native local encoder
(within-model control); do not mix these protocols in one claimed delta.
For track candidates, specify visibility/age masks, gap/source resets,
OFFSCREEN handling, and global-rename invariance. Describe only the selected
architecture as the final method; move rejected candidates to ablations.}

\subsection{Training objectives}

The common main loss is class-balanced binary cross entropy over all horizons
and activity channels. This dense EgoSpeak target is the only supervision
required by the task. If a person-aware candidate survives the activity-only
comparison, an optional auxiliary head predicts current active-speaker state:
\begin{equation}
 \mathcal{L}=\mathcal{L}_{\mathrm{act}}+
 \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}.
 \label{eq:loss}
\end{equation}
Participant ID, target-of-speech, transcript, close-microphone audio, and
future frames are never model inputs. Target-of-speech may stratify qualitative
analysis but is not a predicted variable or a requirement for the primary
multi-other event definition.

\todo{First compare all representation/adaptation protocols with activity loss
only. Then ablate +ASD for the selected person-aware candidate. Add an
event-specific wearer onset/yield loss only as a separately reported extension.
Select all coefficients and early stopping on a training-side calibration
fold; do not inspect Sessions 1--3 during selection.}

\subsection{Track source and deployment contract}

The intended inference pipeline runs a face/head detector on each incoming
frame and associates detections with an online multi-object tracker using only
observations at or before $t$. Its temporary IDs route the shared recurrent
state in Eq.~\eqref{eq:track}; they do not denote names, fixed seats, or
cross-session identities. New tracks initialize from a learned state, short
misses retain a causally aged state, and expired tracks are removed. With no
visible face, the global/audio path and NULL/OFFSCREEN token remain valid. No
future-frame smoothing, full-video identity anchor, or cross-session
recognition is permitted. EasyCom state is continued only across genuinely
adjacent source segments and reset across redacted gaps; Ego4D state is local
to a clip.

During method isolation, dataset-provided boxes and anonymous IDs form a
separate \emph{provided-track} condition. This answers whether person
structure can help under accurate localization, but is not reported as online
detector performance. The deployable \emph{online-track} condition runs the
detector/tracker above and includes its errors and latency. This separation is
important because MuVAP used random full-video identity anchors to build its
dataset tracks; that preprocessing is useful for annotation but is not a
strictly online association algorithm \cite{qi2026muvap}.

\todo{Select and freeze the face/head detector, causal tracker, thresholds,
maximum age, and $N_{\max}$ on training-side data. Report detector recall,
provided/online-track coverage, mean/maximum people, truncations, ID switches,
fragments, occlusion gaps, source resets, and P50/P95 detector--tracker latency
for both datasets. Compare provided tracks, online tracks, and controlled box
jitter/dropout/fragmentation. Until the online row is run, limit deployment
claims to the intended pipeline and label all numbers ``provided-track.''}

\section{Experiments}
\label{sec:setup}

\textbf{Data and splits.}
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
and hyperparameter choices use training-side folds only. Inputs contain the
camera stream and shared array audio available to the wearer; close-talking
microphones, per-person voice labels, and target-of-speech labels are not model
inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D
split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies
clip-local anonymous participant tracks; its per-person voice coverage on the
433-clip EgoSpeak split must be audited before reporting the same multi-other
event analysis.

\textbf{Oracle information audit.}
Before selecting a learned local module, we measure what information could in
principle complement the global state. Analysis-only sidecars expose
transcript-derived speech activity no later than the current cutoff under six
conditions: local audio, visibility only, one-second-delayed activity, timely
aggregate visible activity, per-person histories, and histories with their
cross-time identity assignments shuffled. These labels never enter a
deployable model. The audit separates three hypotheses that ordinary feature
ablation conflates: the value of seeing people, the value of knowing current
visible-other activity, and the incremental value of preserving who produced
that activity.

\textbf{Backbone comparison.}
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence
augmentation $G_m+L$. The primary diagnostic is the within-backbone change;
cross-backbone absolute scores are secondary because native tokenization,
pretraining, streaming interfaces, and hidden-state semantics differ. Common
local encoders test whether adaptation transfers across global backbones,
whereas native local encoders are a separate, model-specific protocol. All
rows share physical input cutoffs, targets, data splits, and evaluation.
Unsupported causal readouts are disclosed rather than replaced by offline or
future-conditioned features.

\todo{Implement the three backbone adapters and validate state timestamps with
an impulse/cutoff test. Pre-register one matched PEFT budget and one common
local branch. Report native-vs-common local encoders in separate table panels.
Keep future model additions outside the confirmatory comparison once this
matrix is locked.}

\textbf{Event construction.}
We derive \emph{wearer onset} events directly from the wearer activity target.
A \emph{multi-other exchange} is a wearer-silent opportunity interval in which
at least two distinct non-wearer participant IDs contribute temporally ordered
speech activity. It is an observable interaction proxy, not a claim about the
speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the
development split to obtain wearer-onset recall
$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer
entries per minute of multi-other opportunity at matched onset recall; the
headline operating point is $r=0.8$. We also report signed onset latency,
wearer-yield recall/latency, and the original ten-horizon per-frame mAP.

\todo{Freeze the event pairing and competing-risk rules, including the
multi-other alternation window, minimum speech/silence, overlap, maximum onset
delay, duplicate detections, one detection per event, track visibility, source
boundaries, and group/offscreen cases. Publish train/test onset counts,
multi-other events/minutes, number of distinct visible speakers, and exclusion
counts. Manually audit at least 100 events. Treat all previously viewed
Sessions 1--3 scores as exploratory; lock the protocol and model before the
confirmatory three-seed run.}

\textbf{Baselines and controls.}
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
This compact matrix tests the representation hypothesis before investing in a
complex fusion module. The first learned target after the oracle audit is a
causal estimator of timely aggregate visible-person floor evidence. Only if
person-organized features survive three seeds, visual time/person corruptions,
and the event gate do we compare a parameter-matched temporal branch without
person continuity, a separately trained per-frame/no-ID set branch, and a
track-consistent masked cross-attention branch. Provided-track and online-track
inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
global renaming must leave outputs unchanged, whereas \emph{broken
continuity} periodically remaps IDs while preserving every crop, timestamp,
audio sample, and target. Time-shifted and removed vision provide additional
alignment controls.

\todo{Define the broken-continuity corruption before evaluation: remap only at
pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
frame counts, audio, labels, and visibility masks. Train the no-ID/reset
baseline from scratch with matched parameters; evaluate all test-time
counterfactuals using one frozen checkpoint and one frozen threshold.}

\textbf{Implementation and statistics.}
All prediction heads run at 5~Hz and use an 8-s causal observation window;
model-specific deviations required by a native streaming interface are stated
explicitly. In the frozen protocol the omni backbone does not update. In PEFT
and local protocols, trainable parameters and optimizer exposure are reported
separately. Every temporal state resets at non-contiguous source boundaries.
We report three seeds and mean$\pm$standard deviation. Confidence intervals for
event metrics use paired cluster bootstrap over sessions/conversation blocks
rather than treating adjacent frames as independent. Runtime includes P50/P95
decision latency, throughput, peak memory, and trainable parameter count.

\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
count, tracker timeout/ranking, hardware, and end-to-end real-time latency
after the architecture is frozen.}

\section{Results and Analysis}
\label{sec:results}

\begin{table}[t]
\caption{Causal oracle-information screen on EasyCom development Session~12
(seed 0, four epochs). Oracle speech activity is analysis-only. Values are
percentages under the causal-valid protocol.}
\label{tab:oracle}
\centering
\small
\begin{tabular}{lcc}
\hline
Input condition & 0.2-s mAP & Mean mAP\\
\hline
Global + local audio              & 72.58 & 58.77\\
Visibility only                   & 71.95 & 58.86\\
Activity delayed by 1 s           & 72.02 & 58.90\\
Aggregate visible activity        & \textbf{79.55} & \textbf{61.68}\\
Per-person activity histories     & 78.80 & 61.11\\
Identity-shuffled histories       & 78.86 & 61.19\\
\hline
\end{tabular}
\end{table}

Table~\ref{tab:oracle} establishes an information opportunity, not a deployable
result. Timely aggregate activity among visible non-wearers provides a large
gain, while visibility alone and stale activity do not. However, preserving
person-resolved history gives no advantage over destroying its cross-time
identity assignment, and both trail the aggregate representation. The learned
method must therefore first recover useful local floor evidence; person-track
continuity is retained only if subsequent no-ID and broken-track controls
separate.

\begin{table}[t]
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
confirmatory result.}
\label{tab:preliminary}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
\hline
Qwen-3B $G$+local A       & 81.72 & 85.07 & 65.11\\
Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
\hline
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
\hline
\end{tabular}}
\end{table}

Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
Pooled vision gives a small gain, while immediate wearer AP is nearly
unchanged. This is compatible with either useful generic local context or
regularization; it is not evidence of person grounding. Only matched-recall
event evaluation and visual-alignment controls can distinguish these cases.
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
feature protocol, it is not combined numerically with
Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
study remains necessary.

\begin{table}[t]
\caption{Planned cross-backbone EasyCom event evaluation. Lower
$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
development-selected operating protocol. ``Local'' denotes the train-side
selected common local-evidence candidate.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R &
Onset latency & Yield R & Mean mAP\\
\hline
Qwen-3B frozen       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B PEFT         & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B + local      & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B frozen       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B PEFT/local   & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o frozen     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}

\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs
PEFT and frozen vs local within each backbone. A general omni-representation
claim requires a consistent pattern across at least two backbone families; an
effect confined to Qwen must be stated as Qwen-specific. The behavior claim is
supported only if multi-other FP/min decreases at matched wearer-onset recall
without worse onset latency or wearer-yield recall. If the selected local
candidate uses tracks, also report its paired intervals against no-ID and
temporal-capacity controls.}

\begin{table}[t]
\caption{Conditional track-continuity counterfactual on the best track-aware
model. This table is retained only if train-side selection chooses such a
candidate; all rows reuse one frozen checkpoint.}
\label{tab:counterfactual}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\
\hline
Correct tracks    & \todo{--} & \todo{--} & \todo{--}\\
Global ID rename  & \todo{--} & \todo{--} & \todo{--}\\
Broken continuity & \todo{--} & \todo{--} & \todo{--}\\
Time shifted      & \todo{--} & \todo{--} & \todo{--}\\
Removed           & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}

If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the
continuity test. Global renaming must be
numerically invariant because IDs carry no semantics. If the model exploits
person history, correct tracks should outperform broken continuity even though
both contain exactly the same per-frame people. If broken, time-shifted, and
removed variants perform similarly to correct tracks, the defensible
conclusion is additional capacity or generic visual context, not useful
person-continuous reasoning.

\todo{Add one compact error analysis split by model family, track visibility,
number of visible people, overlap, track age/fragmentation, and
backchannel/interruption proxy. Include event counts with every rate. Move
rejected PEFT recipes and fusion candidates to supplementary material unless
they directly explain the selected method.}

\textbf{Qualitative evidence.}
For a multi-other exchange, visualize annotation-derived per-person activity
as an analysis reference, provided track IDs when applicable, wearer ground
truth, the frozen-global and selected-local wearer-entry scores, and the single
frozen threshold on one physical time axis. If the selected candidate is
track-aware, also show its broken-continuity score. A useful
example shows the frozen baseline rising at an other-speaker boundary while
the selected adapter remains below threshold; a companion positive example
must show that a real wearer onset is not suppressed.

\todo{Do not hand-pick only successful clips. Before viewing the final test
plots, freeze a deterministic selection rule and show: largest paired
improvement, near-median improvement, one failure, and one correctly retained
wearer onset. Include full local timelines and source frame times, not cropped
screenshots without context. State that examples illustrate the aggregate
event result and cannot establish it alone.}

\section{Conclusion}
\label{sec:conclusion}

We study when an egocentric agent should enter or yield a multiparty floor, not
merely whether someone will speak. The working decomposition is global Omni
context plus causal local audiovisual floor evidence. The current
Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate
visible-other activity is informative beyond the global state and recent
mixed audio. It does not support the stronger claim that participant identity
or track continuity adds value, because per-person histories do not beat an
identity-shuffled control. Dense activity forecasting supplies a comparable
target, while matched-recall event evaluation must determine whether a learned
local estimator actually prevents intrusions during multi-other exchanges
without suppressing valid wearer entry or yield. \todo{After the controlled
study, state which learned local variant was selected, how much of the oracle
gap it recovered, whether the pattern generalized beyond Qwen, the
quantitative false-entry and onset/yield results, and the main deployment
limitation. If no learned variant passes these tests, conclude with the
information audit rather than claiming a track-based solution.}

\bibliographystyle{IEEEbib}
\bibliography{references}

\end{document}