icassp2027-experiment/manuscript/main/main.tex

% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
% in this repository. Replace the class/style files when the 2027 author kit
% is released, and remove every visible \todo{} marker before submission.
% Compile from this directory with: make
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}

\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
\newcommand{\vect}[1]{\boldsymbol{#1}}
\setlength{\emergencystretch}{2em}

\title{WHEN SHOULD AN EGOCENTRIC AGENT SPEAK? DIAGNOSING AND ADAPTING
OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION}

% TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized.
\name{Internal Manuscript Draft}
\address{}

\begin{document}
\ninept
\maketitle

\begin{abstract}
Before generating a response, an egocentric conversational agent must decide
whether to speak or keep listening in a scene that may contain several other
people. General-purpose omni models offer broad streaming audio--visual
context, but it is unclear whether their global representations expose the
fine-grained local cues required for a decision every 200~ms. We study this
question across representative omni backbones and adaptation regimes: frozen
global readout, parameter-efficient tuning, and causal augmentation with
recent audio and visible-person histories. The human camera wearer serves as a
behavioral proxy for the agent, while the dense EgoSpeak target remains the
common output. An EasyCom event protocol reports false wearer entries during
multi-other exchanges at matched wearer-onset recall, and track/alignment
controls test whether visual gains follow person continuity rather than added
capacity. Exploratory Qwen2.5-Omni-3B results motivate the study but do not yet
establish a limitation shared by omni models or determine the final adapter.
\todo{After the train-side study is locked, name the selected backbones and
adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction
at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.}
\end{abstract}

\begin{keywords}
turn-taking, speech initiation, egocentric video, omni models,
audio-visual adaptation
\end{keywords}

\section{Introduction}
\label{sec:intro}

Consider an embodied conversational agent that observes the scene through an
egocentric camera and hears mixed environmental audio. Before response
generation and speech synthesis, a causal floor controller must decide every
200~ms whether the agent should continue listening, begin speaking, or yield.
In a dyadic exchange, a pause or an apparent end of turn may be an invitation
to respond. This assumption fails in multiparty interaction: one participant
may hand the floor to another visible participant, and an agent that reacts
only to acoustic availability may interrupt an exchange that does not involve
it. We study this floor-action controller; response content generation and
closed-loop conversational effects are outside our scope. The observed human
camera wearer is a behavioral proxy rather than an equivalent autonomous
agent: its activity supplies human demonstrations of floor timing under the
same first-person observation geometry.

Voice Activity Projection (VAP) models future joint speech activity and
derives turn-taking events without manually labeling each event
\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
untrimmed egocentric video, forecasting wearer and aggregate non-wearer
activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
output is attractive for a wearable agent, but an ``others'' channel discards
whether the same or different visible people produced the recent activity and
how their behavior evolved across time. Conventional per-frame average
precision can consequently be dominated by ordinary activity while hiding
precisely the false-entry behavior that person-consistent visual cues may
prevent. Event-oriented evaluation likewise emphasizes recall, false alarms,
and latency rather than a single frame-level score \cite{jiang2026turnbench}.

Related work supplies complementary parts of the problem. Egocentric
talking-to-me detection predicts whether a completed utterance addresses the
camera wearer and benefits from long-term social context
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
The former is generally utterance-level rather than continuous anticipation;
the latter predicts visible participants rather than the floor action of an
invisible camera wearer. Our gap is therefore narrower than ``audio--visual
multiparty turn taking'': we study whether short-term anonymous participant
continuity helps direct, continuous wearer-action projection when the candidate
responder is behind the camera and the visible participant set changes.

We call this task \emph{egocentric floor-action projection}. The study asks
four questions: (RQ1) how much wearer activity is linearly or shallowly
readable from a frozen global omni state; (RQ2) whether standard
parameter-efficient tuning recovers missing performance; (RQ3) whether causal
local audio and person-level visual evidence provide complementary information;
and (RQ4) whether any aggregate gain reduces false agent entries without
making the controller globally passive. EasyCom is suitable because it
provides synchronized egocentric audio--video, per-person tracks, and voice
activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
analysis strata only and are never model input or required output.

The intended contributions are therefore a controlled cross-backbone
representation diagnosis, a lightweight local-evidence adaptation selected
from pre-registered candidates, and a behavior-aligned event evaluation. The
current Qwen2.5-Omni observations are motivation, not a claim that all omni
models intrinsically fail. Likewise, track-consistent cross-attention becomes
the final method only if it beats pooled, no-ID, and capacity-matched controls.

\begin{figure}[t]
\centering
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
\todo{After model selection, draw the final system figure. For now it must show
the shared experiment interface: raw causal AV, backbone-specific global
readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous
person histories, and the common $10\times3$ head. Mark causal cutoffs,
trainable modules, parameter budgets, and source-boundary resets.}}}
\caption{Planned model-agnostic floor-action study. The local adapter remains a
candidate family until the controlled train-side comparison is complete.}
\label{fig:method}
\end{figure}

\section{Methods}
\label{sec:method}

\subsection{Wearer-relative activity projection}

At decision time $t$, the model observes only the causal stream
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
For bin $k$, the target
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed; hence this is not a three-way softmax. The
output is
\begin{equation}
  \hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
  \qquad q_t=\hat y^{\mathrm{w}}_{t,1},
  \label{eq:task}
\end{equation}
where $q_t$ is the immediate wearer-entry score used by the event protocol.
It is interpreted as an \emph{entry} score only in wearer-silent opportunity
windows; elsewhere it remains a future-activity probability. This retains the
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
signal.

\subsection{Omni backbones and representation protocols}

Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a
backbone-specific causal global readout at the same physical cutoff $t$.
Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family;
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5
\cite{cui2026minicpmo45} form the planned initial comparison. These models have
different native streaming APIs, temporal tokenization, hidden dimensions, and
adaptation interfaces. We therefore emphasize within-backbone deltas rather
than treating raw cross-model scores as a pure architecture ranking.

Every supported backbone is evaluated under three representation protocols:
(i) a frozen global readout with a dimension-matched prediction head;
(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the
same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak
targets, context duration, training examples, head capacity, and evaluation
are held fixed whenever the backbone permits it. A failure of one LoRA recipe
is reported as that recipe's result, not as proof that the backbone cannot
adapt.

\todo{Implement and audit one adapter for each planned backbone. For every
model, document model revision, processor, native audio/video rates, exact
hidden-state readout, causal/streaming semantics, context limit, dtype,
trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
expose a comparable causal state, report it as unsupported rather than inventing
a non-equivalent readout. Add future backbones only before the experiment
protocol is locked.}

\subsection{Candidate local-evidence adaptation}

The current hypothesis is that global omni states preserve broad semantic
context but may make recent prosody, activity boundaries, and individual
visual motion difficult for a shallow floor head to recover. From the most
recent $L_a$ seconds, a local encoder produces audio evidence $\vect A_t$ and,
optionally, an anonymous visible-person set $\mathcal P_t$. A generic candidate
adapter is
\begin{equation}
 \vect z^{(m)}_t=\vect g^{(m)}_t+
 \gamma_t H_\phi(\vect g^{(m)}_t,\vect A_t,\mathcal P_t),
 \qquad \gamma_t\vert_{\mathrm{init}}=0,
 \label{eq:adapter}
\end{equation}
followed by the common prediction head. Local evidence is derived from the
same causal raw media, not close microphones, transcripts, speaker labels, or
future frames.

We evaluate candidates in increasing structural complexity: pooled recent
audio/vision; a capacity-matched temporal branch; a per-frame person set with
no cross-time identity; a shared track-wise temporal encoder; and person--audio
cross-attention. Only a candidate that improves the pre-registered event metric
over simpler and capacity-matched controls becomes the final method. If
person continuity is selected, temporary IDs route hidden state only---there
is no ID embedding, name recognition, or cross-recording identity---and the set
operator is permutation equivariant \cite{lee2019settransformer}.

\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
parameters and effective local history. Compare a shared local encoder across
backbones (cross-model control) with each backbone's native local encoder
(within-model control); do not mix these protocols in one claimed delta.
For track candidates, specify visibility/age masks, gap/source resets,
OFFSCREEN handling, and global-rename invariance. Describe only the selected
architecture as the final method; move rejected candidates to ablations.}

\subsection{Training objectives}

The common main loss is class-balanced binary cross entropy over all horizons
and activity channels. This dense EgoSpeak target is the only supervision
required by the task. If a person-aware candidate survives the activity-only
comparison, an optional auxiliary head predicts current active-speaker state:
\begin{equation}
 \mathcal{L}=\mathcal{L}_{\mathrm{act}}+
 \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}.
 \label{eq:loss}
\end{equation}
Participant ID, target-of-speech, transcript, close-microphone audio, and
future frames are never model inputs. Target-of-speech may stratify qualitative
analysis but is not a predicted variable or a requirement for the primary
multi-other event definition.

\todo{First compare all representation/adaptation protocols with activity loss
only. Then ablate +ASD for the selected person-aware candidate. Add an
event-specific wearer onset/yield loss only as a separately reported extension.
Select all coefficients and early stopping on a training-side calibration
fold; do not inspect Sessions 1--3 during selection.}

\subsection{Track source and deployment contract}

Person-aware candidates first use dataset-provided boxes and anonymous IDs to
isolate the value of local visual structure. A deployable version instead
obtains causal tracklets from a face/head detector followed by an online
multi-object tracker; no future-frame smoothing or cross-session recognition
is permitted. EasyCom IDs are continued only across genuinely adjacent source
segments and reset across redacted gaps. Ego4D IDs are local to a clip.

\todo{If a person-aware candidate is selected, state precisely whether face or
head boxes are used. Report provided-track
coverage, mean/maximum people, occlusion gaps, track fragments, and source-gap
resets for both datasets. Add either one online detector/tracker experiment or
controlled box jitter, dropout, fragmentation, and ID-switch curves; otherwise
limit every deployment claim to ``given causal participant tracklets.''}

\section{Experiments}
\label{sec:setup}

\textbf{Data and splits.}
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
and hyperparameter choices use training-side folds only. Inputs contain the
camera stream and shared array audio available to the wearer; close-talking
microphones, per-person voice labels, and target-of-speech labels are not model
inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D
split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies
clip-local anonymous participant tracks; its per-person voice coverage on the
433-clip EgoSpeak split must be audited before reporting the same multi-other
event analysis.

\textbf{Backbone comparison.}
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence
augmentation $G_m+L$. The primary diagnostic is the within-backbone change;
cross-backbone absolute scores are secondary because native tokenization,
pretraining, streaming interfaces, and hidden-state semantics differ. Common
local encoders test whether adaptation transfers across global backbones,
whereas native local encoders are a separate, model-specific protocol. All
rows share physical input cutoffs, targets, data splits, and evaluation.
Unsupported causal readouts are disclosed rather than replaced by offline or
future-conditioned features.

\todo{Implement the three backbone adapters and validate state timestamps with
an impulse/cutoff test. Pre-register one matched PEFT budget and one common
local branch. Report native-vs-common local encoders in separate table panels.
Keep future model additions outside the confirmatory comparison once this
matrix is locked.}

\textbf{Event construction.}
We derive \emph{wearer onset} events directly from the wearer activity target.
A \emph{multi-other exchange} is a wearer-silent opportunity interval in which
at least two distinct non-wearer participant IDs contribute temporally ordered
speech activity. It is an observable interaction proxy, not a claim about the
speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the
development split to obtain wearer-onset recall
$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer
entries per minute of multi-other opportunity at matched onset recall; the
headline operating point is $r=0.8$. We also report signed onset latency,
wearer-yield recall/latency, and the original ten-horizon per-frame mAP.

\todo{Freeze the event pairing and competing-risk rules, including the
multi-other alternation window, minimum speech/silence, overlap, maximum onset
delay, duplicate detections, one detection per event, track visibility, source
boundaries, and group/offscreen cases. Publish train/test onset counts,
multi-other events/minutes, number of distinct visible speakers, and exclusion
counts. Manually audit at least 100 events. Treat all previously viewed
Sessions 1--3 scores as exploratory; lock the protocol and model before the
confirmatory three-seed run.}

\textbf{Baselines and controls.}
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
This compact matrix tests the representation hypothesis before investing in a
complex fusion module. If local vision survives three seeds and the event
gate, we then compare a parameter-matched temporal branch without person
continuity, a separately trained per-frame/no-ID set branch, and a
track-consistent branch. On a frozen track-aware checkpoint, a clip-consistent
global renaming must leave outputs unchanged, whereas \emph{broken
continuity} periodically remaps IDs while preserving every crop, timestamp,
audio sample, and target. Time-shifted and removed vision provide additional
alignment controls.

\todo{Define the broken-continuity corruption before evaluation: remap only at
pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
frame counts, audio, labels, and visibility masks. Train the no-ID/reset
baseline from scratch with matched parameters; evaluate all test-time
counterfactuals using one frozen checkpoint and one frozen threshold.}

\textbf{Implementation and statistics.}
All prediction heads run at 5~Hz and use an 8-s causal observation window;
model-specific deviations required by a native streaming interface are stated
explicitly. In the frozen protocol the omni backbone does not update. In PEFT
and local protocols, trainable parameters and optimizer exposure are reported
separately. Every temporal state resets at non-contiguous source boundaries.
We report three seeds and mean$\pm$standard deviation. Confidence intervals for
event metrics use paired cluster bootstrap over sessions/conversation blocks
rather than treating adjacent frames as independent. Runtime includes P50/P95
decision latency, throughput, peak memory, and trainable parameter count.

\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
count, hardware, and real-time latency after the architecture is frozen.}

\section{Results and Analysis}
\label{sec:results}

\begin{table}[t]
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
confirmatory result.}
\label{tab:preliminary}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
\hline
Qwen-3B $G$+local A       & 81.72 & 85.07 & 65.11\\
Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
\hline
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
\hline
\end{tabular}}
\end{table}

Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
Pooled vision gives a small gain, while immediate wearer AP is nearly
unchanged. This is compatible with either weak but useful social grounding or
generic visual regularization; only matched-recall event evaluation and
track-continuity controls can distinguish them.
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
feature protocol, it is not combined numerically with
Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
study remains necessary.

\begin{table}[t]
\caption{Planned cross-backbone EasyCom event evaluation. Lower
$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
development-selected operating protocol. ``Local'' denotes the train-side
selected common local-evidence candidate.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R &
Onset latency & Yield R & Mean mAP\\
\hline
Qwen-3B frozen       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B PEFT         & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B + local      & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B frozen       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B PEFT/local   & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o frozen     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}

\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs
PEFT and frozen vs local within each backbone. A general omni-representation
claim requires a consistent pattern across at least two backbone families; an
effect confined to Qwen must be stated as Qwen-specific. The behavior claim is
supported only if multi-other FP/min decreases at matched wearer-onset recall
without worse onset latency or wearer-yield recall. If the selected local
candidate uses tracks, also report its paired intervals against no-ID and
temporal-capacity controls.}

\begin{table}[t]
\caption{Conditional track-continuity counterfactual on the best track-aware
model. This table is retained only if train-side selection chooses such a
candidate; all rows reuse one frozen checkpoint.}
\label{tab:counterfactual}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\
\hline
Correct tracks    & \todo{--} & \todo{--} & \todo{--}\\
Global ID rename  & \todo{--} & \todo{--} & \todo{--}\\
Broken continuity & \todo{--} & \todo{--} & \todo{--}\\
Time shifted      & \todo{--} & \todo{--} & \todo{--}\\
Removed           & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}

If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the
continuity test. Global renaming must be
numerically invariant because IDs carry no semantics. If the model exploits
person history, correct tracks should outperform broken continuity even though
both contain exactly the same per-frame people. If broken, time-shifted, and
removed variants perform similarly to correct tracks, the defensible
conclusion is additional capacity or generic visual context, not useful
person-continuous reasoning.

\todo{Add one compact error analysis split by model family, track visibility,
number of visible people, overlap, track age/fragmentation, and
backchannel/interruption proxy. Include event counts with every rate. Move
rejected PEFT recipes and fusion candidates to supplementary material unless
they directly explain the selected method.}

\textbf{Qualitative evidence.}
For a multi-other exchange, visualize per-person speech activity and track IDs,
wearer ground truth, the frozen-global and selected-local wearer-entry scores,
and the single frozen threshold on one physical time axis. If the selected
candidate is track-aware, also show its broken-continuity score. A useful
example shows the frozen baseline rising at an other-speaker boundary while
the selected adapter remains below threshold; a companion positive example
must show that a real wearer onset is not suppressed.

\todo{Do not hand-pick only successful clips. Before viewing the final test
plots, freeze a deterministic selection rule and show: largest paired
improvement, near-median improvement, one failure, and one correctly retained
wearer onset. Include full local timelines and source frame times, not cropped
screenshots without context. State that examples illustrate the aggregate
event result and cannot establish it alone.}

\section{Conclusion}
\label{sec:conclusion}

We study when an egocentric agent should enter or yield a multiparty floor, not
merely whether someone will speak. The central question is model-agnostic:
whether frozen or parameter-efficiently adapted omni representations expose
the fine-grained floor cues required by a 5-Hz controller, and whether causal
local evidence closes any gap. Dense activity forecasting supplies a common
target, while matched-recall event evaluation tests whether an apparent gain
actually prevents intrusions during multi-other exchanges. Current Qwen-3B
results are exploratory motivation only. \todo{After the controlled study,
state which backbone/adapter was selected, whether the representation pattern
generalized beyond Qwen, the quantitative false-entry and onset/yield results,
and the main deployment limitation. Do not infer success from aggregate mAP or
selected examples alone.}

\bibliographystyle{IEEEbib}
\bibliography{references}

\end{document}