Expand targeted introduction and move method figure to page two

This commit is contained in:
Xin Wang
2026-09-24 16:28:20 +08:00
parent 8ccfc364d0
commit 98b3ad2d69
5 changed files with 105 additions and 65 deletions
+39 -21
View File
@@ -57,7 +57,14 @@ streaming, voice activity projection, conversational agents
An egocentric conversational agent must decide \emph{when} to speak even
though its camera wearer is usually invisible. It hears a mixed conversation
and sees other participants, but must anticipate its own speech from the
observed stream before a turn boundary is known (Fig.~\ref{fig:task}).
observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
This decision differs from recognizing who is speaking now: the relevant
evidence may be a brief pause, overlap, or response by someone in view, while
the target is the wearer's \emph{future} activity. It also differs from
predicting a semantic end of turn or selecting an addressee. We therefore
study a deliberately narrower and measurable question: at each 200-ms
cutoff, how well can a causal observer forecast background, wearer, and
aggregate-other activity over the next 0.2--2.0~s?
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
extensions, backchannel prediction, online end-of-turn detection, and event
benchmarks address increasingly realistic timing decisions
@@ -65,16 +72,10 @@ benchmarks address increasingly realistic timing decisions
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
Their input viewpoints and target decisions, however, are not interchangeable
with forecasting the future activity of an unseen wearer.
\begin{figure}[H]
\centering
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
only the past audio--video prefix is input; ten future activity states are
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
not semantic floor ownership.}
\label{fig:task}
\end{figure}
The distinction matters when comparing systems: a correct current-speaker
label or an utterance-final response can conceal errors in the first few
hundred milliseconds, precisely where a wearable agent must decide whether
to wait or begin speaking.
\textbf{Egocentric audiovisual conversation.}
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
@@ -90,6 +91,13 @@ prediction in noise \cite{oconnorrussell2025noise}. These studies make a
broad ``first audiovisual turn-taking'' claim untenable. What remains
less examined is how a single causal first-person stream should expose
recent audio--visual evidence to forecast the invisible wearer's activity.
EgoSpeak establishes that compact audio and video representations can be
fused for this task, but it does not answer whether a large integrated Omni
representation makes those cutoff-local cues readily available to a small
forecasting head. Conversely, work on active speakers or addressees need
not solve wearer anticipation: the visible interlocutor and the predicted
speaker occupy different roles. This leaves an architectural question,
rather than a claim that audiovisual evidence itself is new.
\textbf{Streaming Omni and temporal access.}
Omni systems integrate modalities and support streaming interaction
@@ -106,21 +114,31 @@ recent modality features outside the Thinker state, while \emph{partial-tail
refresh} (PTR) queries the observed prefix of the incomplete current block.
SWE denotes temporal locality, not shallow encoder layers; the argument
concerns this pipeline, not an intrinsic deficit of every Omni model.
These interventions target different bottlenecks. SWE asks whether recent
encoder evidence improves prediction when the global state is held fixed;
PTR asks whether the same integrated backbone benefits from a fresher readout
without admitting future media. A gain from one does not establish a gain
from the other. In particular, the effect of PTR requires a matched
completed-only comparison at the same context cap, rather than comparison
with a longer or shorter history.
Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity
anticipation; an EasyCom analysis separating global, recent-audio,
pooled-visual, and privileged visible-activity information; and a specified
EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics.
Only EasyCom screening scores are currently available: matched PTR effects,
Ego4D transfer, and event-level behavior remain open tests.
This framing yields three concrete contributions. First, we formulate a
causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
ways to expose local evidence to a frozen Omni model. Second, on EasyCom we
compare global, recent-audio, and pooled-visual features, then use privileged
past visible activity only as an information audit, not a deployable input.
Third, we specify cross-dataset, endpoint, and hard-slice tests. EasyCom
screening is the only measured evidence; matched PTR, Ego4D transfer, and
event behavior remain open.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
audio and optional pooled person-crop vision to a frozen-Omni head. PTR
temporarily queries the observed current-block prefix while reusing completed
2-s payloads. Privileged past-activity labels are not model inputs.}
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
with recent audio and optional pooled person-crop evidence in the forecasting
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
the observed current-block tail at each 200-ms cutoff. The tail is not
committed to persistent history.}
\label{fig:overview}
\end{figure}