Expand targeted introduction and move method figure to page two
This commit is contained in:
@@ -57,7 +57,14 @@ streaming, voice activity projection, conversational agents
|
||||
An egocentric conversational agent must decide \emph{when} to speak even
|
||||
though its camera wearer is usually invisible. It hears a mixed conversation
|
||||
and sees other participants, but must anticipate its own speech from the
|
||||
observed stream before a turn boundary is known (Fig.~\ref{fig:task}).
|
||||
observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
|
||||
This decision differs from recognizing who is speaking now: the relevant
|
||||
evidence may be a brief pause, overlap, or response by someone in view, while
|
||||
the target is the wearer's \emph{future} activity. It also differs from
|
||||
predicting a semantic end of turn or selecting an addressee. We therefore
|
||||
study a deliberately narrower and measurable question: at each 200-ms
|
||||
cutoff, how well can a causal observer forecast background, wearer, and
|
||||
aggregate-other activity over the next 0.2--2.0~s?
|
||||
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
|
||||
extensions, backchannel prediction, online end-of-turn detection, and event
|
||||
benchmarks address increasingly realistic timing decisions
|
||||
@@ -65,16 +72,10 @@ benchmarks address increasingly realistic timing decisions
|
||||
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
|
||||
Their input viewpoints and target decisions, however, are not interchangeable
|
||||
with forecasting the future activity of an unseen wearer.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
|
||||
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
|
||||
only the past audio--video prefix is input; ten future activity states are
|
||||
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
|
||||
not semantic floor ownership.}
|
||||
\label{fig:task}
|
||||
\end{figure}
|
||||
The distinction matters when comparing systems: a correct current-speaker
|
||||
label or an utterance-final response can conceal errors in the first few
|
||||
hundred milliseconds, precisely where a wearable agent must decide whether
|
||||
to wait or begin speaking.
|
||||
|
||||
\textbf{Egocentric audiovisual conversation.}
|
||||
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
|
||||
@@ -90,6 +91,13 @@ prediction in noise \cite{oconnorrussell2025noise}. These studies make a
|
||||
broad ``first audiovisual turn-taking'' claim untenable. What remains
|
||||
less examined is how a single causal first-person stream should expose
|
||||
recent audio--visual evidence to forecast the invisible wearer's activity.
|
||||
EgoSpeak establishes that compact audio and video representations can be
|
||||
fused for this task, but it does not answer whether a large integrated Omni
|
||||
representation makes those cutoff-local cues readily available to a small
|
||||
forecasting head. Conversely, work on active speakers or addressees need
|
||||
not solve wearer anticipation: the visible interlocutor and the predicted
|
||||
speaker occupy different roles. This leaves an architectural question,
|
||||
rather than a claim that audiovisual evidence itself is new.
|
||||
|
||||
\textbf{Streaming Omni and temporal access.}
|
||||
Omni systems integrate modalities and support streaming interaction
|
||||
@@ -106,21 +114,31 @@ recent modality features outside the Thinker state, while \emph{partial-tail
|
||||
refresh} (PTR) queries the observed prefix of the incomplete current block.
|
||||
SWE denotes temporal locality, not shallow encoder layers; the argument
|
||||
concerns this pipeline, not an intrinsic deficit of every Omni model.
|
||||
These interventions target different bottlenecks. SWE asks whether recent
|
||||
encoder evidence improves prediction when the global state is held fixed;
|
||||
PTR asks whether the same integrated backbone benefits from a fresher readout
|
||||
without admitting future media. A gain from one does not establish a gain
|
||||
from the other. In particular, the effect of PTR requires a matched
|
||||
completed-only comparison at the same context cap, rather than comparison
|
||||
with a longer or shorter history.
|
||||
|
||||
Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity
|
||||
anticipation; an EasyCom analysis separating global, recent-audio,
|
||||
pooled-visual, and privileged visible-activity information; and a specified
|
||||
EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics.
|
||||
Only EasyCom screening scores are currently available: matched PTR effects,
|
||||
Ego4D transfer, and event-level behavior remain open tests.
|
||||
This framing yields three concrete contributions. First, we formulate a
|
||||
causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
|
||||
ways to expose local evidence to a frozen Omni model. Second, on EasyCom we
|
||||
compare global, recent-audio, and pooled-visual features, then use privileged
|
||||
past visible activity only as an information audit, not a deployable input.
|
||||
Third, we specify cross-dataset, endpoint, and hard-slice tests. EasyCom
|
||||
screening is the only measured evidence; matched PTR, Ego4D transfer, and
|
||||
event behavior remain open.
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
||||
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
|
||||
audio and optional pooled person-crop vision to a frozen-Omni head. PTR
|
||||
temporarily queries the observed current-block prefix while reusing completed
|
||||
2-s payloads. Privileged past-activity labels are not model inputs.}
|
||||
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
|
||||
with recent audio and optional pooled person-crop evidence in the forecasting
|
||||
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
|
||||
the observed current-block tail at each 200-ms cutoff. The tail is not
|
||||
committed to persistent history.}
|
||||
\label{fig:overview}
|
||||
\end{figure}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user