Freeze targeted introduction storyline

manuscript/targeted/main.tex

@@ -54,82 +54,80 @@ streaming, voice activity projection, conversational agents
 \section{Introduction}
 \label{sec:intro}
 
-An egocentric conversational agent must decide \emph{when} to speak even
-though its camera wearer is usually invisible.  It hears a mixed conversation
-and sees other participants, but must anticipate its own speech from the
-observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
-This decision differs from recognizing who is speaking now: the relevant
-evidence may be a brief pause, overlap, or response by someone in view, while
-the target is the wearer's \emph{future} activity.  It also differs from
-predicting a semantic end of turn or selecting an addressee.  We therefore
-study a deliberately narrower and measurable question: at each 200-ms
-cutoff, how well can a causal observer forecast background, wearer, and
-aggregate-other activity over the next 0.2--2.0~s?
-Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
-extensions, backchannel prediction, online end-of-turn detection, and event
-benchmarks address increasingly realistic timing decisions
-\cite{ekstedt2022vap,russell2025mmvap,elmers2025triadic,
-lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
-Their input viewpoints and target decisions, however, are not interchangeable
-with forecasting the future activity of an unseen wearer.
-The distinction matters when comparing systems: a correct current-speaker
-label or an utterance-final response can conceal errors in the first few
-hundred milliseconds, precisely where a wearable agent must decide whether
-to wait or begin speaking.
+An egocentric conversational agent could answer situated questions, supply
+context, and assist its wearer during face-to-face conversation. To
+participate naturally, it must decide not only \emph{what} to say but when to
+begin, wait, or stop speaking. This requires interpreting mixed audio,
+visible people's behavior, and linguistic content as the number and roles of
+interlocutors change. A pause is not necessarily an invitation: two people
+visible to the wearer may be talking to each other, and their turn exchange
+should not trigger the agent's response. Since the wearer is normally
+off-camera, the relevant target is not simply which visible face speaks now,
+but whether the wearer is about to speak or cease speaking.
+Both boundaries matter: the agent should withhold an entry during a
+third-party exchange and yield when another participant takes the floor.
+From a moving first-person view, face visibility and overlapping voices can
+change independently; neither a silence threshold nor one detected face
+provides a fixed map of who is interacting with the wearer. These timing
+errors arise before response content is considered.
 
-\textbf{Egocentric audiovisual conversation.}
-EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
-RGB features with specialized encoders, fuses their temporal evidence, and
-forecasts wearer speech on EasyCom and Ego4D \cite{kim2025egospeak,
-donley2021easycom,grauman2022ego4d}.  Other first-person work addresses
-multi-view next-speaker prediction, speaking/listening graphs, or addressee
-detection \cite{serj2024threem,jia2024avconv,kong2024sicnet}.
-Egocentric active-speaker methods \cite{jiang2022asl,huh2025egoasd,
-clarke2025asd} and long--short person modeling \cite{wang2024loconet}
-recover present speaker evidence; audiovisual cues can also help turn
-prediction in noise \cite{oconnorrussell2025noise}.  These studies make a
-broad ``first audiovisual turn-taking'' claim untenable.  What remains
-less examined is how a single causal first-person stream should expose
-recent audio--visual evidence to forecast the invisible wearer's activity.
-EgoSpeak establishes that compact audio and video representations can be
-fused for this task, but it does not answer whether a large integrated Omni
-representation makes those cutoff-local cues readily available to a small
-forecasting head.  Conversely, work on active speakers or addressees need
-not solve wearer anticipation: the visible interlocutor and the predicted
-speaker occupy different roles.  This leaves an architectural question,
-rather than a claim that audiovisual evidence itself is new.
+Prior work addresses complementary parts of this decision. Voice activity
+detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice
+activity projection (VAP) predicts future speaker activity
+\cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and
+linguistic evidence for spoken-agent turn detection
+\cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel
+prediction exploits visible cues but is commonly evaluated with two
+interlocutors \cite{russell2025mmvap,lin2025backchannel,
+oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed
+three-person conversations
+\cite{elmers2025triadic}; online three-party EOT has also used controlled
+multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of
+speakers using single-camera video and face tracks, but studies visible
+speakers from a
+static third-party view \cite{qi2026muvap}. Device-directed speech detection
+can combine acoustics, ASR output, and text to ask whether an utterance
+addresses an assistant, yet the cited model does not observe visual
+relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly
+forecasts the unseen wearer's future speech from causal
+first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak,
+donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction,
+first-person speaking/listening graphs, and addressee detection provide
+related but distinct targets \cite{serj2024threem,jia2024avconv,
+kong2024sicnet}. Egocentric active-speaker localization likewise concerns
+current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}.
+Together, these studies leave open how an integrated streaming model should
+retain cutoff-local evidence for forecasting the unseen wearer's activity in
+a natural multiparty, first-person stream. This is a representation-and-timing
+question, not a claim to first egocentric turn prediction. Our activity
+labels do not establish addressee or semantic floor ownership.
 
-\textbf{Streaming Omni and temporal access.}
-Omni systems integrate modalities and support streaming interaction
-\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
-yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}.  Encoder-side
-audio--visual adaptation offers another route to accessible modality evidence
-\cite{lin2023lavish,rouditchenko2024whisperflamingo}.  Qwen2.5-Omni uses
-block-wise audio processing \cite{xu2025qwenomni}; in our extraction, a 2-s
-block is long relative to a 200-ms decision interval.  A streaming interface
-alone therefore does not specify how fresh the integrated representation is
-at each forecast cutoff.  We study two complementary, cutoff-aligned
-interventions: \emph{short-window encoder evidence} (SWE) directly exposes
-recent modality features outside the Thinker state, while \emph{partial-tail
-refresh} (PTR) queries the observed prefix of the incomplete current block.
-SWE denotes temporal locality, not shallow encoder layers; the argument
-concerns this pipeline, not an intrinsic deficit of every Omni model.
-These interventions target different bottlenecks.  SWE asks whether recent
-encoder evidence improves prediction when the global state is held fixed;
-PTR asks whether the same integrated backbone benefits from a fresher readout
-without admitting future media.  A gain from one does not establish a gain
-from the other.  In particular, the effect of PTR requires a matched
-completed-only comparison at the same context cap, rather than comparison
-with a longer or shorter history.
+Omni models couple modality encoders to a shared language-model state, so
+audio, video, and text can inform streaming interaction and longer-range
+semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
+Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
+time-aligned audio--video interleaving feed a Thinker for integrated
+understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
+We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a
+measured result. The native audio attention block spans 2~s, whereas our
+wearer forecast updates every 200~ms. Holding only completed blocks leaves
+stale context between boundaries; forcing 200-ms commits changes the native
+block construction and may alter its representation. Neither implies that
+all Omni models lack local evidence. An integrated, multi-layer state can
+carry broader audiovisual and linguistic context, yet transient cues may be
+less accessible to a small head than encoder-side features themselves
+\cite{lin2023lavish,rouditchenko2024whisperflamingo}.
 
-This framing yields three concrete contributions.  First, we formulate a
-causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
-ways to expose local evidence to a frozen Omni model.  Second, on EasyCom we
-compare global, recent-audio, and pooled-visual features, then use privileged
-past visible activity only as an information audit, not a deployable input.
-Third, we specify cross-dataset, endpoint, and hard-slice tests.  EasyCom
-screening is the only measured evidence; matched PTR, Ego4D transfer, and
-event behavior remain open.
+We therefore retain the global Thinker state while exposing recent modality
+evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
+suggest that global context and local audio are complementary; gains from
+pooled vision and isolated PTR effects remain unproven. This study makes
+three contributions: (1) short-window encoder evidence (SWE) supplies recent
+audio and optional person-crop vision to a frozen-Omni forecasting head;
+(2) partial-tail refresh (PTR) queries the observed current-block prefix
+without committing it to persistent history; and (3) detailed EasyCom
+analyses accompany a specified Ego4D transfer protocol. Matched PTR,
+Ego4D scores, and event behavior remain pending.
 
 \begin{figure}[t]
 \centering