docs: clarify agent framing in manuscript

manuscript/main/main.tex

@@ -10,8 +10,8 @@
 \newcommand{\vect}[1]{\boldsymbol{#1}}
 \setlength{\emergencystretch}{2em}
 
-\title{WHEN SHOULD AN EGOCENTRIC AGENT SPEAK? DIAGNOSING AND ADAPTING
-OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION}
+\title{WHEN SHOULD AN EGOCENTRIC CONVERSATIONAL AGENT SPEAK? PROBING AND
+ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION}
 
 % TODO: Insert the final author list and affiliations after the ICASSP 2027
 % review policy and author order are finalized.
@@ -23,59 +23,87 @@ OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION}
 \maketitle
 
 \begin{abstract}
-Before generating a response, an egocentric conversational agent must decide
-whether to speak or keep listening in a scene that may contain several other
-people. General-purpose omni models offer broad streaming audio--visual
-context, but it is unclear whether their global representations expose the
-fine-grained local cues required for a decision every 200~ms. We study this
-question across representative omni backbones and adaptation regimes: frozen
-global readout, parameter-efficient tuning, and causal augmentation with
-recent audio and visible-person histories. The human camera wearer serves as a
-behavioral proxy for the agent, while the dense EgoSpeak target remains the
-common output. An EasyCom event protocol reports false wearer entries during
-multi-other exchanges at matched wearer-onset recall, and track/alignment
-controls test whether visual gains follow person continuity rather than added
-capacity. Exploratory Qwen2.5-Omni-3B results motivate the study but do not yet
-establish a limitation shared by omni models or determine the final adapter.
+An egocentric conversational agent operating among several people must decide
+whether to speak or keep listening before it generates a response. EgoSpeak
+operationalizes this problem using recordings from human camera wearers:
+causal first-person audio--video forms the observation, and the wearer's future
+speaking behavior forms the action target. The wearer is thus a data-collection
+surrogate rather than the deployed system. EgoSpeak evaluates
+separately extracted audio--visual features with Transformer, GRU, and Mamba
+temporal models; whether end-to-end pretrained omni models that jointly process
+raw audio and video expose the
+fine-grained cues required for a decision every 200~ms remains unclear. We
+probe representative omni backbones using frozen readout,
+parameter-efficient tuning, and causal side branches that preserve recent
+audio and visible-person evidence. The study tests whether a global omni state
+underrepresents other-person activity and whether local evidence compensates
+without making the agent passive. An EasyCom event protocol measures false
+agent entries during multi-other exchanges at matched recall of wearer-derived
+onset targets. Exploratory Qwen2.5-Omni-3B results motivate this hypothesis but
+do not yet establish a cross-model limitation or determine the final adapter.
 \todo{After the train-side study is locked, name the selected backbones and
 adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction
 at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.}
 \end{abstract}
 
 \begin{keywords}
-turn-taking, speech initiation, egocentric video, omni models,
-audio-visual adaptation
+conversational agents, turn-taking, speech initiation, egocentric video,
+omni models
 \end{keywords}
 
 \section{Introduction}
 \label{sec:intro}
 
-Consider an embodied conversational agent that observes the scene through an
-egocentric camera and hears mixed environmental audio. Before response
-generation and speech synthesis, a causal floor controller must decide every
-200~ms whether the agent should continue listening, begin speaking, or yield.
-In a dyadic exchange, a pause or an apparent end of turn may be an invitation
-to respond. This assumption fails in multiparty interaction: one participant
-may hand the floor to another visible participant, and an agent that reacts
-only to acoustic availability may interrupt an exchange that does not involve
-it. We study this floor-action controller; response content generation and
-closed-loop conversational effects are outside our scope. The observed human
-camera wearer is a behavioral proxy rather than an equivalent autonomous
-agent: its activity supplies human demonstrations of floor timing under the
-same first-person observation geometry.
+We consider a conversational agent equipped with an egocentric camera and
+microphone and deployed in a scene where several people may speak. Before
+response generation and speech synthesis, its causal floor controller must
+decide every 200~ms whether to continue listening, begin speaking, or yield. A
+pause in dyadic dialogue may invite a response; in multiparty interaction it
+may instead mark a handover between two other people. An agent that reacts only
+to acoustic availability can therefore interrupt a conversation that it
+should continue observing.
 
-Voice Activity Projection (VAP) models future joint speech activity and
-derives turn-taking events without manually labeling each event
-\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
-untrimmed egocentric video, forecasting wearer and aggregate non-wearer
-activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
-output is attractive for a wearable agent, but an ``others'' channel discards
-whether the same or different visible people produced the recent activity and
-how their behavior evolved across time. Conventional per-frame average
-precision can consequently be dominated by ordinary activity while hiding
-precisely the false-entry behavior that person-consistent visual cues may
-prevent. Event-oriented evaluation likewise emphasizes recall, false alarms,
-and latency rather than a single frame-level score \cite{jiang2026turnbench}.
+Existing datasets do not contain autonomous agents making these decisions at
+scale. Following EgoSpeak \cite{kim2025egospeak}, we use human egocentric
+recordings to construct supervision: the camera wearer defines the first-person
+sensor origin, and the wearer's observed speaking behavior defines the desired
+floor-action target. This is an imitation-learning proxy for a future agent
+under comparable observations. Throughout the paper, \emph{wearer} names this
+dataset role and target channel; it is neither the deployed agent nor a
+semantic model input. Response content and closed-loop effects of deploying an
+agent remain outside our scope.
+
+EgoSpeak already establishes real-time speech-initiation prediction in causal,
+untrimmed egocentric video. It combines separately pretrained ResNet-50 and
+wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba
+temporal models, forecasting the camera wearer and aggregate other-person
+activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is
+not the task definition or the use of first-person data. The unresolved model
+question is whether an end-to-end pretrained audio--visual omni backbone,
+operating on causal raw media, exposes an adequate state for this dense
+floor-action controller. To our knowledge, such omni representations have not been
+systematically evaluated or adapted for the EgoSpeak task.
+
+Our working hypothesis is specific. An omni model may retain broad scene and
+speech semantics while its global causal state makes recent prosody, activity
+boundaries, and individual visible-speaker motion difficult for a shallow
+5-Hz head to recover. This would appear as inaccurate prediction of
+other-person activity and poorly timed agent entries, especially in
+multiparty exchanges. A causal local side branch may compensate by preserving
+recent audio and anonymous person-level evidence before fusion with the global
+state. This is a hypothesis to test, not an assumed omni-model deficiency:
+frozen, parameter-efficiently tuned, and locally augmented variants must be
+compared within each backbone.
+
+Voice Activity Projection (VAP) provides the broader future-activity view of
+turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves
+compatibility with that view, but its aggregate ``others'' channel discards
+whether the same or different visible people produced recent activity.
+Moreover, conventional per-frame average precision can hide the behavioral
+failure of entering during an exchange between others. We therefore retain the
+dense EgoSpeak target while adding event evaluation of agent entry and yield;
+event-oriented work likewise emphasizes recall, false alarms, and latency
+rather than a single frame-level score \cite{jiang2026turnbench}.
 
 Related work supplies complementary parts of the problem. Egocentric
 talking-to-me detection predicts whether a completed utterance addresses the
@@ -83,29 +111,32 @@ camera wearer and benefits from long-term social context
 \cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
 tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
 The former is generally utterance-level rather than continuous anticipation;
-the latter predicts visible participants rather than the floor action of an
-invisible camera wearer. Our gap is therefore narrower than ``audio--visual
-multiparty turn taking'': we study whether short-term anonymous participant
-continuity helps direct, continuous wearer-action projection when the candidate
-responder is behind the camera and the visible participant set changes.
+the latter predicts visible participants rather than the action of an
+egocentric conversational agent represented by a wearer-derived target. Our
+gap is therefore narrower than ``audio--visual multiparty turn taking'': we
+study whether omni representations support direct, continuous agent
+floor-action prediction and whether causal local evidence helps when the
+visible participant set changes.
 
 We call this task \emph{egocentric floor-action projection}. The study asks
-four questions: (RQ1) how much wearer activity is linearly or shallowly
-readable from a frozen global omni state; (RQ2) whether standard
+four questions: (RQ1) how accurately a frozen global omni state predicts the
+wearer-derived agent target and other-person activity; (RQ2) whether standard
 parameter-efficient tuning recovers missing performance; (RQ3) whether causal
-local audio and person-level visual evidence provide complementary information;
-and (RQ4) whether any aggregate gain reduces false agent entries without
-making the controller globally passive. EasyCom is suitable because it
-provides synchronized egocentric audio--video, per-person tracks, and voice
+local audio and person-level visual evidence provide complementary
+information; and (RQ4) whether any aggregate gain reduces false agent entries
+without making the controller globally passive. EasyCom is suitable because
+it provides synchronized egocentric audio--video, per-person tracks, and voice
 activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
 analysis strata only and are never model input or required output.
 
-The intended contributions are therefore a controlled cross-backbone
-representation diagnosis, a lightweight local-evidence adaptation selected
-from pre-registered candidates, and a behavior-aligned event evaluation. The
-current Qwen2.5-Omni observations are motivation, not a claim that all omni
-models intrinsically fail. Likewise, track-consistent cross-attention becomes
-the final method only if it beats pooled, no-ID, and capacity-matched controls.
+The intended contributions are therefore: a controlled cross-backbone study
+of pretrained omni representations for EgoSpeak-style floor action; a
+lightweight local-evidence adaptation selected from
+pre-registered candidates; and a behavior-aligned event evaluation that maps
+wearer demonstrations to agent entry and yield decisions. Current
+Qwen2.5-Omni observations are motivation, not a claim that all omni models
+intrinsically fail. Likewise, track-consistent cross-attention becomes the
+final method only if it beats pooled, no-ID, and capacity-matched controls.
 
 \begin{figure}[t]
 \centering
@@ -115,8 +146,9 @@ the shared experiment interface: raw causal AV, backbone-specific global
 readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous
 person histories, and the common $10\times3$ head. Mark causal cutoffs,
 trainable modules, parameter budgets, and source-boundary resets.}}}
-\caption{Planned model-agnostic floor-action study. The local adapter remains a
-candidate family until the controlled train-side comparison is complete.}
+\caption{Planned omni-model study for egocentric conversational floor action.
+The local adapter remains a candidate family until the controlled train-side
+comparison is complete.}
 \label{fig:method}
 \end{figure}