|
|
@@ -10,8 +10,8 @@
|
|
|
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
|
|
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
|
|
|
\setlength{\emergencystretch}{2em}
|
|
|
|
\setlength{\emergencystretch}{2em}
|
|
|
|
|
|
|
|
|
|
|
|
\title{WHEN SHOULD AN EGOCENTRIC CONVERSATIONAL AGENT SPEAK? PROBING AND
|
|
|
|
\title{GLOBAL OMNI CONTEXT AND CAUSAL LOCAL AUDIOVISUAL EVIDENCE FOR
|
|
|
|
ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION}
|
|
|
|
EGOCENTRIC MULTIPARTY FLOOR-ACTION FORECASTING}
|
|
|
|
|
|
|
|
|
|
|
|
% TODO: Insert the final author list and affiliations after the ICASSP 2027
|
|
|
|
% TODO: Insert the final author list and affiliations after the ICASSP 2027
|
|
|
|
% review policy and author order are finalized.
|
|
|
|
% review policy and author order are finalized.
|
|
|
@@ -24,26 +24,28 @@ ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION}
|
|
|
|
|
|
|
|
|
|
|
|
\begin{abstract}
|
|
|
|
\begin{abstract}
|
|
|
|
An egocentric conversational agent operating among several people must decide
|
|
|
|
An egocentric conversational agent operating among several people must decide
|
|
|
|
whether to speak or keep listening before it generates a response. EgoSpeak
|
|
|
|
whether to speak or keep listening before generating a response. Following
|
|
|
|
operationalizes this problem using recordings from human camera wearers:
|
|
|
|
EgoSpeak, we use causal first-person audio--video as observation and the future
|
|
|
|
causal first-person audio--video forms the observation, and the wearer's future
|
|
|
|
speaking behavior of the human camera wearer as an imitation target for this
|
|
|
|
speaking behavior forms the action target. The wearer is thus a data-collection
|
|
|
|
agent decision. We ask whether a jointly pretrained Omni model exposes the
|
|
|
|
surrogate rather than the deployed system. EgoSpeak evaluates
|
|
|
|
fine-grained floor cues required every 200~ms. Our global--local formulation
|
|
|
|
separately extracted audio--visual features with Transformer, GRU, and Mamba
|
|
|
|
uses a causal global Omni readout for broad scene and conversational context,
|
|
|
|
temporal models; whether end-to-end pretrained omni models that jointly process
|
|
|
|
while a causal local branch preserves recent mixed audio and a variable set of
|
|
|
|
raw audio and video expose the
|
|
|
|
track-organized person observations. Anonymous tracks are an organizational
|
|
|
|
fine-grained cues required for a decision every 200~ms remains unclear. We
|
|
|
|
interface whose incremental value must be established, not an assumption that
|
|
|
|
probe representative omni backbones using frozen readout,
|
|
|
|
the model knows speaker identity. On EasyCom development data, a single-seed
|
|
|
|
parameter-efficient tuning, and causal side branches that preserve recent
|
|
|
|
oracle analysis finds that perfectly aligned aggregate activity among visible
|
|
|
|
audio and visible-person evidence. The study tests whether a global omni state
|
|
|
|
non-wearers improves 0.2-s mAP by 6.97 points and ten-horizon mean mAP by 2.91
|
|
|
|
underrepresents other-person activity and whether local evidence compensates
|
|
|
|
points over global Omni plus recent audio. Person-resolved histories do not
|
|
|
|
without making the agent passive. An EasyCom event protocol measures false
|
|
|
|
outperform an identity-shuffled control. This result motivates timely local
|
|
|
|
agent entries during multi-other exchanges at matched recall of wearer-derived
|
|
|
|
floor evidence, while explicitly withholding a person-continuity claim. The
|
|
|
|
onset targets. Exploratory Qwen2.5-Omni-3B results motivate this hypothesis but
|
|
|
|
full study compares frozen, parameter-efficient, and learned local variants
|
|
|
|
do not yet establish a cross-model limitation or determine the final adapter.
|
|
|
|
and evaluates false agent entry at matched wearer-onset recall.
|
|
|
|
\todo{After the train-side study is locked, name the selected backbones and
|
|
|
|
\todo{After the train-side study is locked, replace the final two sentences
|
|
|
|
adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction
|
|
|
|
with the selected learned local variant and insert: its oracle-gap recovery,
|
|
|
|
at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.}
|
|
|
|
multi-other false-entry reduction at 80\% onset recall, onset/yield
|
|
|
|
|
|
|
|
non-degradation, and 95\% intervals. Mention track continuity only if its
|
|
|
|
|
|
|
|
controls separate.}
|
|
|
|
\end{abstract}
|
|
|
|
\end{abstract}
|
|
|
|
|
|
|
|
|
|
|
|
\begin{keywords}
|
|
|
|
\begin{keywords}
|
|
|
@@ -79,26 +81,39 @@ wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba
|
|
|
|
temporal models, forecasting the camera wearer and aggregate other-person
|
|
|
|
temporal models, forecasting the camera wearer and aggregate other-person
|
|
|
|
activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is
|
|
|
|
activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is
|
|
|
|
not the task definition or the use of first-person data. The unresolved model
|
|
|
|
not the task definition or the use of first-person data. The unresolved model
|
|
|
|
question is whether an end-to-end pretrained audio--visual omni backbone,
|
|
|
|
question is whether a jointly pretrained audio--visual Omni backbone,
|
|
|
|
operating on causal raw media, exposes an adequate state for this dense
|
|
|
|
operating on causal raw media, exposes an adequate state for this dense
|
|
|
|
floor-action controller. To our knowledge, such omni representations have not been
|
|
|
|
floor-action controller. To our knowledge, such omni representations have not been
|
|
|
|
systematically evaluated or adapted for the EgoSpeak task.
|
|
|
|
systematically evaluated or adapted for the EgoSpeak task.
|
|
|
|
|
|
|
|
|
|
|
|
Our working hypothesis is specific. An omni model may retain broad scene and
|
|
|
|
Our working hypothesis is a representational decomposition. A global causal
|
|
|
|
speech semantics while its global causal state makes recent prosody, activity
|
|
|
|
Omni state may retain broad scene, speech, and semantic context while making
|
|
|
|
boundaries, and individual visible-speaker motion difficult for a shallow
|
|
|
|
recent prosody, activity boundaries, and localized person motion difficult for
|
|
|
|
5-Hz head to recover. This would appear as inaccurate prediction of
|
|
|
|
a shallow 5-Hz controller to recover. A causal local branch therefore
|
|
|
|
other-person activity and poorly timed agent entries, especially in
|
|
|
|
preserves recent mixed audio and a variable-size set of person observations
|
|
|
|
multiparty exchanges. A causal local side branch may compensate by preserving
|
|
|
|
before fusing them back into the global state. The local set can be pooled
|
|
|
|
recent audio and anonymous person-level evidence before fusion with the global
|
|
|
|
within a frame or organized into anonymous tracks across time. Whether that
|
|
|
|
state. This is a hypothesis to test, not an assumed omni-model deficiency:
|
|
|
|
continuity adds useful information beyond aggregate activity is itself a
|
|
|
|
frozen, parameter-efficiently tuned, and locally augmented variants must be
|
|
|
|
falsifiable question. Frozen, parameter-efficiently tuned, and locally
|
|
|
|
compared within each backbone.
|
|
|
|
augmented variants must consequently be compared within each backbone.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
An initial causal oracle screen sharpens this hypothesis. On held-out EasyCom
|
|
|
|
|
|
|
|
Session~12, perfectly timed aggregate activity from currently visible
|
|
|
|
|
|
|
|
non-wearers improves a matched global-plus-audio model by 6.97 points at 0.2~s
|
|
|
|
|
|
|
|
and 2.91 points in mean mAP. Visibility alone and activity delayed by one
|
|
|
|
|
|
|
|
second remain near the audio baseline. However, person-resolved histories are
|
|
|
|
|
|
|
|
weaker than aggregate histories and indistinguishable from histories whose
|
|
|
|
|
|
|
|
identity continuity is shuffled. Thus the current evidence supports the value
|
|
|
|
|
|
|
|
of timely local floor information, not a claim that person identity or track
|
|
|
|
|
|
|
|
continuity has already been exploited.
|
|
|
|
|
|
|
|
|
|
|
|
Voice Activity Projection (VAP) provides the broader future-activity view of
|
|
|
|
Voice Activity Projection (VAP) provides the broader future-activity view of
|
|
|
|
turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves
|
|
|
|
turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves
|
|
|
|
compatibility with that view, but its aggregate ``others'' channel discards
|
|
|
|
compatibility with that view, but its aggregate ``others'' channel discards
|
|
|
|
whether the same or different visible people produced recent activity.
|
|
|
|
whether the same or different visible people produced recent activity.
|
|
|
|
|
|
|
|
Our oracle result shows that this discarded distinction is not automatically
|
|
|
|
|
|
|
|
useful under the dense target: identity continuity must beat aggregate and
|
|
|
|
|
|
|
|
shuffled controls before it enters the method claim.
|
|
|
|
Moreover, conventional per-frame average precision can hide the behavioral
|
|
|
|
Moreover, conventional per-frame average precision can hide the behavioral
|
|
|
|
failure of entering during an exchange between others. We therefore retain the
|
|
|
|
failure of entering during an exchange between others. We therefore retain the
|
|
|
|
dense EgoSpeak target while adding event evaluation of agent entry and yield;
|
|
|
|
dense EgoSpeak target while adding event evaluation of agent entry and yield;
|
|
|
@@ -121,37 +136,40 @@ direct, continuous agent floor-action prediction and whether causal local
|
|
|
|
evidence helps when the visible participant set changes.
|
|
|
|
evidence helps when the visible participant set changes.
|
|
|
|
|
|
|
|
|
|
|
|
We call this task \emph{egocentric floor-action projection}. The study asks
|
|
|
|
We call this task \emph{egocentric floor-action projection}. The study asks
|
|
|
|
four questions: (RQ1) how accurately a frozen global omni state predicts the
|
|
|
|
four questions: (RQ1) how accurately a frozen global Omni state predicts the
|
|
|
|
wearer-derived agent target and other-person activity; (RQ2) whether standard
|
|
|
|
wearer-derived agent target and other-person activity; (RQ2) whether timely
|
|
|
|
parameter-efficient tuning recovers missing performance; (RQ3) whether causal
|
|
|
|
causal local audio and visible-person evidence complement that state; (RQ3)
|
|
|
|
local audio and person-level visual evidence provide complementary
|
|
|
|
whether aggregate person-set evidence is sufficient or anonymous track
|
|
|
|
information; and (RQ4) whether any aggregate gain reduces false agent entries
|
|
|
|
continuity adds measurable value; and (RQ4) whether any dense gain reduces
|
|
|
|
without making the controller globally passive. EasyCom is suitable because
|
|
|
|
false agent entries without making the controller globally passive.
|
|
|
|
|
|
|
|
Parameter-efficient tuning is a matched adaptation control rather than an
|
|
|
|
|
|
|
|
assumed remedy. EasyCom is suitable because
|
|
|
|
it provides synchronized egocentric audio--video, per-person tracks, and voice
|
|
|
|
it provides synchronized egocentric audio--video, per-person tracks, and voice
|
|
|
|
activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
|
|
|
|
activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
|
|
|
|
analysis strata only and are never model input or required output.
|
|
|
|
analysis strata only and are never model input or required output.
|
|
|
|
|
|
|
|
|
|
|
|
The intended contributions are therefore: a controlled cross-backbone study
|
|
|
|
The intended contributions are therefore: a controlled study of global Omni
|
|
|
|
of pretrained omni representations for EgoSpeak-style floor action; a
|
|
|
|
representations for EgoSpeak-style floor action; an oracle information audit
|
|
|
|
lightweight local-evidence adaptation selected from
|
|
|
|
that separates timely aggregate floor evidence from person continuity; a
|
|
|
|
pre-registered candidates; and a behavior-aligned event evaluation that maps
|
|
|
|
global--local adaptation selected from progressively structured candidates;
|
|
|
|
wearer demonstrations to agent entry and yield decisions. Current
|
|
|
|
and a behavior-aligned event evaluation that maps wearer demonstrations to
|
|
|
|
Qwen2.5-Omni observations are motivation, not a claim that all omni models
|
|
|
|
agent entry and yield decisions. Current Qwen2.5-Omni observations are
|
|
|
|
intrinsically fail. Likewise, track-consistent cross-attention becomes the
|
|
|
|
motivation, not a claim that all Omni models intrinsically fail. Likewise,
|
|
|
|
final method only if it beats pooled, no-ID, and capacity-matched controls.
|
|
|
|
track-consistent cross-attention becomes the final method only if it beats
|
|
|
|
|
|
|
|
aggregate, shuffled/no-ID, and capacity-matched controls.
|
|
|
|
|
|
|
|
|
|
|
|
\begin{figure}[t]
|
|
|
|
\begin{figure}[t]
|
|
|
|
\centering
|
|
|
|
\centering
|
|
|
|
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
|
|
|
|
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
|
|
|
|
\todo{After model selection, draw the final system figure. For now it must show
|
|
|
|
\todo{After model selection, draw the final system figure. Show the shared
|
|
|
|
the shared experiment interface: raw causal AV; online face/head detection and
|
|
|
|
interface: raw causal AV to a global Omni readout; a parallel recent-audio and
|
|
|
|
causal tracking; backbone-specific global readout; recent audio and a masked
|
|
|
|
visible-person path; aggregate/no-ID/track-organized variants; residual fusion;
|
|
|
|
set of anonymous track states; the gated local adapter; and the common
|
|
|
|
and the common $10\times3$ head. Draw the transcript-derived oracle only as a
|
|
|
|
$10\times3$ head. Mark causal cutoffs, trainable modules, parameter budgets,
|
|
|
|
dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus
|
|
|
|
NULL/OFFSCREEN handling, and source-boundary resets.}}}
|
|
|
|
online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}}
|
|
|
|
\caption{Planned omni-model study for egocentric conversational floor action.
|
|
|
|
\caption{Global context and causal local evidence for egocentric floor action.
|
|
|
|
The local adapter remains a candidate family until the controlled train-side
|
|
|
|
Track organization is a controlled candidate, not a presupposed source of
|
|
|
|
comparison is complete.}
|
|
|
|
speaker identity.}
|
|
|
|
\label{fig:method}
|
|
|
|
\label{fig:method}
|
|
|
|
\end{figure}
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
|
@@ -207,13 +225,14 @@ expose a comparable causal state, report it as unsupported rather than inventing
|
|
|
|
a non-equivalent readout. Add future backbones only before the experiment
|
|
|
|
a non-equivalent readout. Add future backbones only before the experiment
|
|
|
|
protocol is locked.}
|
|
|
|
protocol is locked.}
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Candidate local-evidence adaptation}
|
|
|
|
\subsection{Global context and causal local evidence}
|
|
|
|
|
|
|
|
|
|
|
|
The current hypothesis is that global omni states preserve broad semantic
|
|
|
|
The global state $\vect g^{(m)}_t$ is responsible for broad scene and
|
|
|
|
context but may make recent prosody, activity boundaries, and individual
|
|
|
|
conversational context, but is not assumed to retain every localized cue needed
|
|
|
|
visual motion difficult for a shallow floor head to recover. From the most
|
|
|
|
for floor control. From the most
|
|
|
|
recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
|
|
|
|
recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
|
|
|
|
time $t$, an online detector and tracker produce a variable-size anonymous set
|
|
|
|
time $t$, provided boxes during method isolation---and an online detector and
|
|
|
|
|
|
|
|
tracker in the deployable condition---produce a variable-size anonymous set
|
|
|
|
$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
|
|
|
|
$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
|
|
|
|
where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
|
|
|
|
where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
|
|
|
|
$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
|
|
|
|
$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
|
|
|
@@ -231,8 +250,8 @@ current person set by masked cross-attention,
|
|
|
|
W_q[\vect g^{(m)}_t;\vect a_t],
|
|
|
|
W_q[\vect g^{(m)}_t;\vect a_t],
|
|
|
|
[\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
|
|
|
|
[\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}]
|
|
|
|
\right), \\
|
|
|
|
\right), \\
|
|
|
|
\vect z^{(m)}_t &= \vect g^{(m)}_t+\tanh(\alpha)W_o\vect u_t,
|
|
|
|
\vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t,
|
|
|
|
\qquad \alpha\vert_{\mathrm{init}}=0,
|
|
|
|
\qquad W_o\vert_{\mathrm{init}}=0,
|
|
|
|
\label{eq:adapter}
|
|
|
|
\label{eq:adapter}
|
|
|
|
\end{align}
|
|
|
|
\end{align}
|
|
|
|
where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
|
|
|
|
where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is
|
|
|
@@ -243,14 +262,23 @@ reported. The operator is permutation equivariant
|
|
|
|
\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
|
|
|
|
\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
|
|
|
|
identity semantics.
|
|
|
|
identity semantics.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Equation~\eqref{eq:track} is a candidate organization of local evidence, not a
|
|
|
|
|
|
|
|
claim that recurrent identity is necessary. Simpler alternatives replace each
|
|
|
|
|
|
|
|
$\vect h_{t,i}$ with a current-frame observation or pool the set into one
|
|
|
|
|
|
|
|
aggregate vector. This distinction is important because the current oracle
|
|
|
|
|
|
|
|
screen favors aggregate activity and does not separate correct from shuffled
|
|
|
|
|
|
|
|
identity histories.
|
|
|
|
|
|
|
|
|
|
|
|
This decomposition is informed by MuVAP: its global audio representation
|
|
|
|
This decomposition is informed by MuVAP: its global audio representation
|
|
|
|
queries same-frame causal ASD track representations, followed by temporal
|
|
|
|
queries same-frame causal ASD track representations, followed by temporal
|
|
|
|
modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
|
|
|
|
modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
|
|
|
|
useful separation between global floor dynamics and person-grounded evidence,
|
|
|
|
useful separation between global floor dynamics and person-organized evidence,
|
|
|
|
but not its role-relative output: our fixed target is the invisible wearer's
|
|
|
|
but not its role-relative output: our fixed target is the invisible wearer's
|
|
|
|
entry/yield action, not the next visible speaker. The zero-initialized gate
|
|
|
|
entry/yield action, not the next visible speaker. The zero-initialized residual
|
|
|
|
also follows the conservative adapter principle used to inject visual evidence
|
|
|
|
output follows the conservative adapter principle used to inject visual
|
|
|
|
into strong audio models \cite{rouditchenko2024whisperflamingo}. Local evidence
|
|
|
|
evidence without perturbing the base prediction at initialization
|
|
|
|
|
|
|
|
\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits
|
|
|
|
|
|
|
|
the residual output layer to receive a gradient immediately. Local evidence
|
|
|
|
is derived from the same causal raw media, not close microphones, transcripts,
|
|
|
|
is derived from the same causal raw media, not close microphones, transcripts,
|
|
|
|
speaker labels, global identity anchors, or future frames.
|
|
|
|
speaker labels, global identity anchors, or future frames.
|
|
|
|
|
|
|
|
|
|
|
@@ -258,8 +286,10 @@ We evaluate candidates in increasing structural complexity: pooled recent
|
|
|
|
audio/vision; a capacity-matched temporal branch; a per-frame person set with
|
|
|
|
audio/vision; a capacity-matched temporal branch; a per-frame person set with
|
|
|
|
no cross-time identity; a shared track-wise temporal encoder; and person--audio
|
|
|
|
no cross-time identity; a shared track-wise temporal encoder; and person--audio
|
|
|
|
cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
|
|
|
|
cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
|
|
|
|
candidate that improves the pre-registered event metric over simpler and
|
|
|
|
candidate that improves the pre-registered event metric over simpler,
|
|
|
|
capacity-matched controls becomes the final method. A limited latent
|
|
|
|
aggregate, identity-shuffled, and capacity-matched controls becomes the final
|
|
|
|
|
|
|
|
method. If person continuity does not separate from those controls, the final
|
|
|
|
|
|
|
|
model remains a global-plus-local aggregate floor-evidence model. A limited latent
|
|
|
|
bottleneck is tested only if large visible sets make full person attention a
|
|
|
|
bottleneck is tested only if large visible sets make full person attention a
|
|
|
|
runtime bottleneck; it is not assumed to improve accuracy.
|
|
|
|
runtime bottleneck; it is not assumed to improve accuracy.
|
|
|
|
|
|
|
|
|
|
|
@@ -339,6 +369,18 @@ clip-local anonymous participant tracks; its per-person voice coverage on the
|
|
|
|
433-clip EgoSpeak split must be audited before reporting the same multi-other
|
|
|
|
433-clip EgoSpeak split must be audited before reporting the same multi-other
|
|
|
|
event analysis.
|
|
|
|
event analysis.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\textbf{Oracle information audit.}
|
|
|
|
|
|
|
|
Before selecting a learned local module, we measure what information could in
|
|
|
|
|
|
|
|
principle complement the global state. Analysis-only sidecars expose
|
|
|
|
|
|
|
|
transcript-derived speech activity no later than the current cutoff under six
|
|
|
|
|
|
|
|
conditions: local audio, visibility only, one-second-delayed activity, timely
|
|
|
|
|
|
|
|
aggregate visible activity, per-person histories, and histories with their
|
|
|
|
|
|
|
|
cross-time identity assignments shuffled. These labels never enter a
|
|
|
|
|
|
|
|
deployable model. The audit separates three hypotheses that ordinary feature
|
|
|
|
|
|
|
|
ablation conflates: the value of seeing people, the value of knowing current
|
|
|
|
|
|
|
|
visible-other activity, and the incremental value of preserving who produced
|
|
|
|
|
|
|
|
that activity.
|
|
|
|
|
|
|
|
|
|
|
|
\textbf{Backbone comparison.}
|
|
|
|
\textbf{Backbone comparison.}
|
|
|
|
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
|
|
|
|
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
|
|
|
|
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
|
|
|
|
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
|
|
|
@@ -383,9 +425,11 @@ confirmatory three-seed run.}
|
|
|
|
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
|
|
|
|
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
|
|
|
|
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
|
|
|
|
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
|
|
|
|
This compact matrix tests the representation hypothesis before investing in a
|
|
|
|
This compact matrix tests the representation hypothesis before investing in a
|
|
|
|
complex fusion module. If local vision survives three seeds and the event
|
|
|
|
complex fusion module. The first learned target after the oracle audit is a
|
|
|
|
gate, we then compare a parameter-matched temporal branch without person
|
|
|
|
causal estimator of timely aggregate visible-person floor evidence. Only if
|
|
|
|
continuity, a separately trained per-frame/no-ID set branch, and a
|
|
|
|
person-organized features survive three seeds, visual time/person corruptions,
|
|
|
|
|
|
|
|
and the event gate do we compare a parameter-matched temporal branch without
|
|
|
|
|
|
|
|
person continuity, a separately trained per-frame/no-ID set branch, and a
|
|
|
|
track-consistent masked cross-attention branch. Provided-track and online-track
|
|
|
|
track-consistent masked cross-attention branch. Provided-track and online-track
|
|
|
|
inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
|
|
|
|
inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
|
|
|
|
global renaming must leave outputs unchanged, whereas \emph{broken
|
|
|
|
global renaming must leave outputs unchanged, whereas \emph{broken
|
|
|
@@ -419,6 +463,36 @@ after the architecture is frozen.}
|
|
|
|
\section{Results and Analysis}
|
|
|
|
\section{Results and Analysis}
|
|
|
|
\label{sec:results}
|
|
|
|
\label{sec:results}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{table}[t]
|
|
|
|
|
|
|
|
\caption{Causal oracle-information screen on EasyCom development Session~12
|
|
|
|
|
|
|
|
(seed 0, four epochs). Oracle speech activity is analysis-only. Values are
|
|
|
|
|
|
|
|
percentages under the causal-valid protocol.}
|
|
|
|
|
|
|
|
\label{tab:oracle}
|
|
|
|
|
|
|
|
\centering
|
|
|
|
|
|
|
|
\small
|
|
|
|
|
|
|
|
\begin{tabular}{lcc}
|
|
|
|
|
|
|
|
\hline
|
|
|
|
|
|
|
|
Input condition & 0.2-s mAP & Mean mAP\\
|
|
|
|
|
|
|
|
\hline
|
|
|
|
|
|
|
|
Global + local audio & 72.58 & 58.77\\
|
|
|
|
|
|
|
|
Visibility only & 71.95 & 58.86\\
|
|
|
|
|
|
|
|
Activity delayed by 1 s & 72.02 & 58.90\\
|
|
|
|
|
|
|
|
Aggregate visible activity & \textbf{79.55} & \textbf{61.68}\\
|
|
|
|
|
|
|
|
Per-person activity histories & 78.80 & 61.11\\
|
|
|
|
|
|
|
|
Identity-shuffled histories & 78.86 & 61.19\\
|
|
|
|
|
|
|
|
\hline
|
|
|
|
|
|
|
|
\end{tabular}
|
|
|
|
|
|
|
|
\end{table}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table~\ref{tab:oracle} establishes an information opportunity, not a deployable
|
|
|
|
|
|
|
|
result. Timely aggregate activity among visible non-wearers provides a large
|
|
|
|
|
|
|
|
gain, while visibility alone and stale activity do not. However, preserving
|
|
|
|
|
|
|
|
person-resolved history gives no advantage over destroying its cross-time
|
|
|
|
|
|
|
|
identity assignment, and both trail the aggregate representation. The learned
|
|
|
|
|
|
|
|
method must therefore first recover useful local floor evidence; person-track
|
|
|
|
|
|
|
|
continuity is retained only if subsequent no-ID and broken-track controls
|
|
|
|
|
|
|
|
separate.
|
|
|
|
|
|
|
|
|
|
|
|
\begin{table}[t]
|
|
|
|
\begin{table}[t]
|
|
|
|
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
|
|
|
|
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
|
|
|
|
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
|
|
|
|
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
|
|
|
@@ -441,9 +515,9 @@ $\Delta$ visual & +0.30 & +0.02 & +0.29\\
|
|
|
|
|
|
|
|
|
|
|
|
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
|
|
|
|
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
|
|
|
|
Pooled vision gives a small gain, while immediate wearer AP is nearly
|
|
|
|
Pooled vision gives a small gain, while immediate wearer AP is nearly
|
|
|
|
unchanged. This is compatible with either weak but useful social grounding or
|
|
|
|
unchanged. This is compatible with either useful generic local context or
|
|
|
|
generic visual regularization; only matched-recall event evaluation and
|
|
|
|
regularization; it is not evidence of person grounding. Only matched-recall
|
|
|
|
track-continuity controls can distinguish them.
|
|
|
|
event evaluation and visual-alignment controls can distinguish these cases.
|
|
|
|
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
|
|
|
|
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
|
|
|
|
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
|
|
|
|
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
|
|
|
|
feature protocol, it is not combined numerically with
|
|
|
|
feature protocol, it is not combined numerically with
|
|
|
@@ -522,10 +596,11 @@ rejected PEFT recipes and fusion candidates to supplementary material unless
|
|
|
|
they directly explain the selected method.}
|
|
|
|
they directly explain the selected method.}
|
|
|
|
|
|
|
|
|
|
|
|
\textbf{Qualitative evidence.}
|
|
|
|
\textbf{Qualitative evidence.}
|
|
|
|
For a multi-other exchange, visualize per-person speech activity and track IDs,
|
|
|
|
For a multi-other exchange, visualize annotation-derived per-person activity
|
|
|
|
wearer ground truth, the frozen-global and selected-local wearer-entry scores,
|
|
|
|
as an analysis reference, provided track IDs when applicable, wearer ground
|
|
|
|
and the single frozen threshold on one physical time axis. If the selected
|
|
|
|
truth, the frozen-global and selected-local wearer-entry scores, and the single
|
|
|
|
candidate is track-aware, also show its broken-continuity score. A useful
|
|
|
|
frozen threshold on one physical time axis. If the selected candidate is
|
|
|
|
|
|
|
|
track-aware, also show its broken-continuity score. A useful
|
|
|
|
example shows the frozen baseline rising at an other-speaker boundary while
|
|
|
|
example shows the frozen baseline rising at an other-speaker boundary while
|
|
|
|
the selected adapter remains below threshold; a companion positive example
|
|
|
|
the selected adapter remains below threshold; a companion positive example
|
|
|
|
must show that a real wearer onset is not suppressed.
|
|
|
|
must show that a real wearer onset is not suppressed.
|
|
|
@@ -541,17 +616,21 @@ event result and cannot establish it alone.}
|
|
|
|
\label{sec:conclusion}
|
|
|
|
\label{sec:conclusion}
|
|
|
|
|
|
|
|
|
|
|
|
We study when an egocentric agent should enter or yield a multiparty floor, not
|
|
|
|
We study when an egocentric agent should enter or yield a multiparty floor, not
|
|
|
|
merely whether someone will speak. The central question is model-agnostic:
|
|
|
|
merely whether someone will speak. The working decomposition is global Omni
|
|
|
|
whether frozen or parameter-efficiently adapted omni representations expose
|
|
|
|
context plus causal local audiovisual floor evidence. The current
|
|
|
|
the fine-grained floor cues required by a 5-Hz controller, and whether causal
|
|
|
|
Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate
|
|
|
|
local evidence closes any gap. Dense activity forecasting supplies a common
|
|
|
|
visible-other activity is informative beyond the global state and recent
|
|
|
|
target, while matched-recall event evaluation tests whether an apparent gain
|
|
|
|
mixed audio. It does not support the stronger claim that participant identity
|
|
|
|
actually prevents intrusions during multi-other exchanges. Current Qwen-3B
|
|
|
|
or track continuity adds value, because per-person histories do not beat an
|
|
|
|
results are exploratory motivation only. \todo{After the controlled study,
|
|
|
|
identity-shuffled control. Dense activity forecasting supplies a comparable
|
|
|
|
state which backbone/adapter was selected, whether the representation pattern
|
|
|
|
target, while matched-recall event evaluation must determine whether a learned
|
|
|
|
generalized beyond Qwen, the quantitative false-entry and onset/yield results,
|
|
|
|
local estimator actually prevents intrusions during multi-other exchanges
|
|
|
|
and the main deployment limitation. Do not infer success from aggregate mAP or
|
|
|
|
without suppressing valid wearer entry or yield. \todo{After the controlled
|
|
|
|
selected examples alone.}
|
|
|
|
study, state which learned local variant was selected, how much of the oracle
|
|
|
|
|
|
|
|
gap it recovered, whether the pattern generalized beyond Qwen, the
|
|
|
|
|
|
|
|
quantitative false-entry and onset/yield results, and the main deployment
|
|
|
|
|
|
|
|
limitation. If no learned variant passes these tests, conclude with the
|
|
|
|
|
|
|
|
information audit rather than claiming a track-based solution.}
|
|
|
|
|
|
|
|
|
|
|
|
\bibliographystyle{IEEEbib}
|
|
|
|
\bibliographystyle{IEEEbib}
|
|
|
|
\bibliography{references}
|
|
|
|
\bibliography{references}
|
|
|
|
Xin Wang