2026-09-23 07:54:36 +08:00
|
|
|
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
|
2026-09-24 11:07:48 +08:00
|
|
|
% Uses the repository's ICASSP 2027 author kit.
|
|
|
|
|
% Four technical pages plus an optional reference-only page.
|
2026-09-23 07:54:36 +08:00
|
|
|
\documentclass{article}
|
|
|
|
|
\usepackage{spconf}
|
2026-09-24 11:07:48 +08:00
|
|
|
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
|
2026-09-23 07:54:36 +08:00
|
|
|
\hypersetup{hidelinks}
|
|
|
|
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
|
|
|
|
\newcommand{\G}{\mathrm{G}}
|
|
|
|
|
\newcommand{\A}{\mathrm{A}}
|
|
|
|
|
\newcommand{\V}{\mathrm{V}}
|
|
|
|
|
\setlength{\emergencystretch}{2em}
|
2026-09-24 11:07:48 +08:00
|
|
|
\setcounter{topnumber}{5}
|
|
|
|
|
\setcounter{dbltopnumber}{5}
|
|
|
|
|
\renewcommand{\topfraction}{0.95}
|
|
|
|
|
\renewcommand{\dbltopfraction}{0.95}
|
|
|
|
|
\renewcommand{\textfraction}{0.05}
|
|
|
|
|
\renewcommand{\floatpagefraction}{0.75}
|
|
|
|
|
\renewcommand{\dblfloatpagefraction}{0.75}
|
2026-09-24 14:34:35 +08:00
|
|
|
\title{Egocentric Speech Anticipation with Omni Models:\\
|
|
|
|
|
Short-Window Evidence and Partial-Tail Refresh}
|
2026-09-24 15:44:25 +08:00
|
|
|
\name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\
|
2026-09-24 14:34:35 +08:00
|
|
|
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
|
2026-09-24 15:44:25 +08:00
|
|
|
\address{China Telecom Corporation Limited Shanghai Branch}
|
2026-09-23 07:54:36 +08:00
|
|
|
|
|
|
|
|
\begin{document}
|
|
|
|
|
\ninept
|
|
|
|
|
\maketitle
|
|
|
|
|
|
|
|
|
|
\begin{abstract}
|
2026-10-25 23:02:03 +08:00
|
|
|
An egocentric conversational agent must anticipate when its unseen wearer
|
|
|
|
|
will speak while other participants exchange turns in a causal multiparty
|
|
|
|
|
audio--video stream. Integrated Omni states provide broad audiovisual
|
|
|
|
|
context, yet recent evidence near each prediction cutoff may be difficult
|
|
|
|
|
for a small forecasting head to access. We study short-window encoder
|
|
|
|
|
evidence (SWE), which exposes the preceding second of audio and optional
|
|
|
|
|
causal visual features alongside frozen Qwen2.5-Omni context. We also
|
|
|
|
|
specify partial-tail refresh (PTR), which queries only the observed prefix
|
|
|
|
|
of the current audiovisual block at each 200-ms cutoff without committing
|
|
|
|
|
that prefix to history. We predict background, wearer, and aggregate-other
|
|
|
|
|
activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory
|
|
|
|
|
global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean
|
|
|
|
|
mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision
|
|
|
|
|
adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches
|
|
|
|
|
65.37\%. On Ego4D, paired three-seed global-plus-audio models improve
|
|
|
|
|
ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%,
|
|
|
|
|
while a single visual-attention run reaches 70.93\%. These results highlight
|
|
|
|
|
the value of recent encoder evidence for dense wearer-activity forecasting
|
|
|
|
|
across both datasets, with strictly causal inputs and a frozen backbone
|
|
|
|
|
throughout.
|
2026-09-23 07:54:36 +08:00
|
|
|
\end{abstract}
|
|
|
|
|
|
|
|
|
|
\begin{keywords}
|
|
|
|
|
egocentric speech anticipation, multimodal foundation models, causal
|
|
|
|
|
streaming, voice activity projection, conversational agents
|
|
|
|
|
\end{keywords}
|
|
|
|
|
|
|
|
|
|
\section{Introduction}
|
|
|
|
|
\label{sec:intro}
|
|
|
|
|
|
2026-09-24 17:15:16 +08:00
|
|
|
An egocentric conversational agent could answer situated questions, supply
|
|
|
|
|
context, and assist its wearer during face-to-face conversation. To
|
|
|
|
|
participate naturally, it must decide not only \emph{what} to say but when to
|
|
|
|
|
begin, wait, or stop speaking. This requires interpreting mixed audio,
|
|
|
|
|
visible people's behavior, and linguistic content as the number and roles of
|
|
|
|
|
interlocutors change. A pause is not necessarily an invitation: two people
|
|
|
|
|
visible to the wearer may be talking to each other, and their turn exchange
|
|
|
|
|
should not trigger the agent's response. Since the wearer is normally
|
|
|
|
|
off-camera, the relevant target is not simply which visible face speaks now,
|
|
|
|
|
but whether the wearer is about to speak or cease speaking.
|
|
|
|
|
Both boundaries matter: the agent should withhold an entry during a
|
|
|
|
|
third-party exchange and yield when another participant takes the floor.
|
|
|
|
|
From a moving first-person view, face visibility and overlapping voices can
|
|
|
|
|
change independently; neither a silence threshold nor one detected face
|
|
|
|
|
provides a fixed map of who is interacting with the wearer. These timing
|
|
|
|
|
errors arise before response content is considered.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 17:15:16 +08:00
|
|
|
Prior work addresses complementary parts of this decision. Voice activity
|
|
|
|
|
detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice
|
|
|
|
|
activity projection (VAP) predicts future speaker activity
|
|
|
|
|
\cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and
|
|
|
|
|
linguistic evidence for spoken-agent turn detection
|
|
|
|
|
\cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel
|
|
|
|
|
prediction exploits visible cues but is commonly evaluated with two
|
|
|
|
|
interlocutors \cite{russell2025mmvap,lin2025backchannel,
|
|
|
|
|
oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed
|
|
|
|
|
three-person conversations
|
|
|
|
|
\cite{elmers2025triadic}; online three-party EOT has also used controlled
|
|
|
|
|
multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of
|
|
|
|
|
speakers using single-camera video and face tracks, but studies visible
|
|
|
|
|
speakers from a
|
|
|
|
|
static third-party view \cite{qi2026muvap}. Device-directed speech detection
|
|
|
|
|
can combine acoustics, ASR output, and text to ask whether an utterance
|
|
|
|
|
addresses an assistant, yet the cited model does not observe visual
|
|
|
|
|
relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly
|
|
|
|
|
forecasts the unseen wearer's future speech from causal
|
|
|
|
|
first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak,
|
|
|
|
|
donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction,
|
|
|
|
|
first-person speaking/listening graphs, and addressee detection provide
|
|
|
|
|
related but distinct targets \cite{serj2024threem,jia2024avconv,
|
|
|
|
|
kong2024sicnet}. Egocentric active-speaker localization likewise concerns
|
|
|
|
|
current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}.
|
|
|
|
|
Together, these studies leave open how an integrated streaming model should
|
|
|
|
|
retain cutoff-local evidence for forecasting the unseen wearer's activity in
|
|
|
|
|
a natural multiparty, first-person stream. This is a representation-and-timing
|
|
|
|
|
question, not a claim to first egocentric turn prediction. Our activity
|
|
|
|
|
labels do not establish addressee or semantic floor ownership.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 17:15:16 +08:00
|
|
|
Omni models couple modality encoders to a shared language-model state, so
|
|
|
|
|
audio, video, and text can inform streaming interaction and longer-range
|
|
|
|
|
semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
|
|
|
|
|
Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
|
|
|
|
|
time-aligned audio--video interleaving feed a Thinker for integrated
|
|
|
|
|
understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
|
2026-10-25 23:02:03 +08:00
|
|
|
The main EasyCom comparison uses frozen 3B Thinker readouts; a separate
|
|
|
|
|
seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our
|
2026-09-24 17:15:16 +08:00
|
|
|
wearer forecast updates every 200~ms. Holding only completed blocks leaves
|
|
|
|
|
stale context between boundaries; forcing 200-ms commits changes the native
|
|
|
|
|
block construction and may alter its representation. Neither implies that
|
|
|
|
|
all Omni models lack local evidence. An integrated, multi-layer state can
|
|
|
|
|
carry broader audiovisual and linguistic context, yet transient cues may be
|
|
|
|
|
less accessible to a small head than encoder-side features themselves
|
|
|
|
|
\cite{lin2023lavish,rouditchenko2024whisperflamingo}.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 17:15:16 +08:00
|
|
|
We therefore retain the global Thinker state while exposing recent modality
|
|
|
|
|
evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
|
|
|
|
|
suggest that global context and local audio are complementary; gains from
|
|
|
|
|
pooled vision and isolated PTR effects remain unproven. This study makes
|
|
|
|
|
three contributions: (1) short-window encoder evidence (SWE) supplies recent
|
2026-09-24 18:38:51 +08:00
|
|
|
audio and optional visual encoder evidence to a frozen-Omni forecasting head;
|
2026-09-24 17:15:16 +08:00
|
|
|
(2) partial-tail refresh (PTR) queries the observed current-block prefix
|
|
|
|
|
without committing it to persistent history; and (3) detailed EasyCom
|
2026-10-25 23:02:03 +08:00
|
|
|
analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run.
|
|
|
|
|
Matched PTR and event behavior remain pending.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 11:07:48 +08:00
|
|
|
\begin{figure}[t]
|
2026-09-23 07:54:36 +08:00
|
|
|
\centering
|
2026-09-24 11:07:48 +08:00
|
|
|
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
2026-09-24 16:28:20 +08:00
|
|
|
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
|
2026-09-24 18:38:51 +08:00
|
|
|
with recent audio and optional local visual evidence in the forecasting
|
2026-09-24 16:28:20 +08:00
|
|
|
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
|
|
|
|
|
the observed current-block tail at each 200-ms cutoff. The tail is not
|
|
|
|
|
committed to persistent history.}
|
2026-09-23 07:54:36 +08:00
|
|
|
\label{fig:overview}
|
2026-09-24 11:07:48 +08:00
|
|
|
\end{figure}
|
2026-09-23 07:54:36 +08:00
|
|
|
|
|
|
|
|
\section{Method}
|
|
|
|
|
\label{sec:method}
|
|
|
|
|
|
2026-09-24 18:00:38 +08:00
|
|
|
\subsection{Task and Global--Local Fusion}
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 15:44:25 +08:00
|
|
|
At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by
|
|
|
|
|
$t_n$ and predicts activity at $t_n+h$, where
|
|
|
|
|
$h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
|
2026-09-23 07:54:36 +08:00
|
|
|
horizon has three channels in fixed order: background, camera wearer, and
|
|
|
|
|
aggregate others. Wearer and others may overlap; no output identifies an
|
|
|
|
|
individual speaker, addressee, or semantic end of turn.
|
|
|
|
|
|
2026-09-24 18:00:38 +08:00
|
|
|
The global representation $\vect g_n$ concatenates the final four frozen
|
|
|
|
|
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
|
|
|
|
|
already integrates audio and video over the available context; $\G$ is not
|
|
|
|
|
an audio-free baseline. Short-window encoder evidence (SWE) adds a direct,
|
|
|
|
|
cutoff-aligned path for recent audio $\vect a_n$ and, when enabled, visual
|
|
|
|
|
evidence $\vect v_n$. This separates the question of what the global state
|
|
|
|
|
encodes from whether a small prediction head can access transient cues.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 18:00:38 +08:00
|
|
|
In the pooled fusion, each available local vector is layer-normalized,
|
|
|
|
|
projected to 256 dimensions, and accompanied by a validity bit. The
|
|
|
|
|
concatenated global and local features are passed to a layer-normalized MLP:
|
2026-09-23 07:54:36 +08:00
|
|
|
\begin{equation}
|
2026-09-24 18:00:38 +08:00
|
|
|
\widehat{\vect Y}_{n,h,c}=\operatorname{softmax}_{c}
|
|
|
|
|
F_{\theta}\!\left([\vect g_n;\widetilde{\vect a}_n;m_n^a;
|
|
|
|
|
\widetilde{\vect v}_n;m_n^v]\right)_{h,c},
|
2026-09-23 07:54:36 +08:00
|
|
|
\label{eq:fusion}
|
|
|
|
|
\end{equation}
|
2026-09-24 18:00:38 +08:00
|
|
|
where a disabled branch is omitted and $F_\theta$ produces ten sets of
|
|
|
|
|
three logits. For token-preserving visual variants, $\vect v_n$ denotes
|
|
|
|
|
an attention-pooled context: a query formed from $\vect g_n$ and valid
|
|
|
|
|
$\vect a_n$ attends to the causal visual token set (and a learned null
|
|
|
|
|
token), rather than averaging it beforehand. The resulting context and
|
|
|
|
|
validity bit enter the same prediction head. We train the projections,
|
|
|
|
|
visual aggregator when present, and head with the EgoSpeak multi-positive
|
|
|
|
|
cross-entropy over causal-valid source--horizon pairs. The principal SWE
|
|
|
|
|
configuration freezes all feature encoders and the Thinker: this is
|
|
|
|
|
frozen-backbone adaptation, not task-level end-to-end training.
|
|
|
|
|
|
|
|
|
|
We separate \emph{where} parameters are updated from \emph{which} Thinker
|
|
|
|
|
layers are read. The primary comparison trains only the layer-combination
|
|
|
|
|
projection, local-fusion modules, and 30-logit prediction head on cached
|
|
|
|
|
frozen features. Parameter-adaptation controls instead attach low-rank
|
|
|
|
|
adapters to Thinker attention projections, either across the configured
|
|
|
|
|
Thinker or restricted to its upper layers; the prediction head may be
|
|
|
|
|
updated jointly or held fixed to isolate representation changes. These
|
|
|
|
|
controls require the same causal media, source cutoffs, supervision,
|
|
|
|
|
initialization, and model-selection protocol as the frozen-head condition.
|
|
|
|
|
Archived LoRA runs, including an upper-eight-layer setting, are
|
|
|
|
|
exploratory and not a matched layer-scope ablation. Last-four-layer
|
|
|
|
|
\emph{readout} fusion changes the head input, not the LoRA update scope;
|
|
|
|
|
neither its gain nor an unmatched LoRA score establishes an adaptation
|
|
|
|
|
advantage.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 14:34:35 +08:00
|
|
|
\subsection{Causal Partial-Tail Refresh}
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 14:34:35 +08:00
|
|
|
PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
|
2026-09-24 15:44:25 +08:00
|
|
|
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
|
|
|
|
|
observed current tail are available; span varies with block phase. Reusing
|
2026-09-24 14:34:35 +08:00
|
|
|
completed modality payloads, we crop the tail at $t_n$ and query it in a
|
2026-09-24 15:44:25 +08:00
|
|
|
temporary cache. The tail and query are not committed. Only completed
|
2026-09-24 14:34:35 +08:00
|
|
|
blocks enter persistent history; session and source discontinuities reset
|
2026-09-24 15:44:25 +08:00
|
|
|
caches. Thus each 5-Hz readout sees the observed prefix without future
|
|
|
|
|
backfill. To test whether access policy matters, the matched design crosses
|
|
|
|
|
completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap;
|
|
|
|
|
the separate 2--12-s cap sweep tests context length. Those comparisons are
|
|
|
|
|
pending, so PTR is a causal access rule rather than a demonstrated gain.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 18:00:38 +08:00
|
|
|
\subsection{Short-Window Feature Extraction}
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 18:52:10 +08:00
|
|
|
Each local branch observes only media preceding the prediction cutoff
|
|
|
|
|
$t_n$; unavailable observations retain explicit validity masks rather
|
|
|
|
|
than borrowing later samples. All encoders are frozen. The notation below
|
|
|
|
|
distinguishes encoder choice from fusion choice: $\G$ is the global Thinker
|
|
|
|
|
state, and local branches are appended to the same forecasting head.
|
2026-09-24 18:00:38 +08:00
|
|
|
|
2026-09-24 18:52:10 +08:00
|
|
|
The default audio feature $\A$ encodes the mixed waveform in
|
|
|
|
|
$[t_n-1\,\mathrm{s},t_n)$ with the Qwen2.5-Omni audio encoder and averages
|
|
|
|
|
valid tokens. $\A_{\rm E}$ denotes EgoSpeak's wav2vec~2.0 convolutional
|
|
|
|
|
representation \cite{kim2025egospeak}; comparing $\G+\A$ with
|
|
|
|
|
$\G+\A_{\rm E}$ requires the same window and head. Released EgoSpeak A+V
|
|
|
|
|
remains an external baseline, not this encoder substitution.
|
2026-09-24 18:00:38 +08:00
|
|
|
|
2026-09-24 18:52:10 +08:00
|
|
|
For vision, Pool averages Qwen tokens within provided person boxes and
|
|
|
|
|
across valid 2-FPS frames in the preceding second; the boxes are not
|
|
|
|
|
outputs of an evaluated online detector. Spatial visual attention (SVA)
|
|
|
|
|
instead lets $\G+\A$ query the unpooled spatial tokens of the latest
|
|
|
|
|
strictly pre-cutoff full frame. Temporal spatial visual attention (TSVA)
|
|
|
|
|
first applies a shared GRU to up to two causal tokens at each matching
|
|
|
|
|
grid position, then uses the same query. TSVA links image coordinates,
|
|
|
|
|
not person tracks. We denote these variants by $\G+\A+$Pool,
|
|
|
|
|
$\G+\A+$SVA, and $\G+\A+$TSVA.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 15:11:14 +08:00
|
|
|
\input{tables_compact}
|
2026-10-25 23:02:03 +08:00
|
|
|
|
|
|
|
|
\begin{figure*}[t]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf}
|
|
|
|
|
\caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending
|
|
|
|
|
at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap;
|
|
|
|
|
(B) other-to-other handover. Outcome-selected, unreviewed examples do not
|
|
|
|
|
establish an event-level gain.}
|
|
|
|
|
\label{fig:hq_cases}
|
|
|
|
|
\end{figure*}
|
2026-09-24 15:11:14 +08:00
|
|
|
% Historical full-precision table layouts retained as source audit only.
|
|
|
|
|
\iffalse
|
2026-09-24 11:07:48 +08:00
|
|
|
\begin{table*}[t]
|
|
|
|
|
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
|
|
|
|
|
three-seed privileged past-activity audit, which is not deployable. Mean is
|
|
|
|
|
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
|
2026-09-24 18:38:51 +08:00
|
|
|
TSVA is a no-box, two-frame causal full-frame screen; it and the historical
|
2026-09-24 11:07:48 +08:00
|
|
|
deployable rows are exploratory seed-0 runs with unmatched head/training
|
|
|
|
|
details. F1 has three seeds. Dashes are unavailable, not zero.}
|
|
|
|
|
\label{tab:main}
|
|
|
|
|
\centering
|
|
|
|
|
\normalsize
|
|
|
|
|
\renewcommand{\arraystretch}{0.87}
|
|
|
|
|
\setlength{\tabcolsep}{5.0pt}
|
|
|
|
|
\begin{tabular}{@{}lccccc@{}}
|
|
|
|
|
\toprule
|
|
|
|
|
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
|
|
|
|
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
|
|
|
|
\midrule
|
|
|
|
|
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
|
|
|
|
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
|
|
|
|
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
|
|
|
|
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
|
|
|
|
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
|
|
|
|
|
& 65.400 & 26.019 & 45.371 \\\\
|
2026-09-24 18:38:51 +08:00
|
|
|
No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\
|
2026-09-24 11:07:48 +08:00
|
|
|
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
|
|
|
|
& $64.951{\pm}0.095$ & -- & -- \\\\
|
|
|
|
|
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
|
|
|
|
|
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
|
|
|
|
|
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
|
|
|
|
|
\bottomrule
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\par\smallskip
|
|
|
|
|
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
|
|
|
|
|
\normalsize
|
|
|
|
|
\setlength{\tabcolsep}{3.2pt}
|
|
|
|
|
\begin{tabular}{@{}lccc@{}}
|
|
|
|
|
\toprule
|
|
|
|
|
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
|
|
|
|
\midrule
|
|
|
|
|
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
|
|
|
|
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
|
|
|
|
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
|
|
|
|
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
|
|
|
|
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
|
|
|
|
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
|
|
|
|
& $\mathbf{66.951{\pm}.022}$ \\\\
|
|
|
|
|
\bottomrule
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\end{table*}
|
|
|
|
|
|
|
|
|
|
\begin{table*}[t]
|
|
|
|
|
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
|
|
|
|
|
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
|
|
|
|
|
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
|
|
|
|
|
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
|
|
|
|
|
\label{tab:stream_context}
|
|
|
|
|
\centering\normalsize
|
|
|
|
|
\renewcommand{\arraystretch}{0.87}
|
|
|
|
|
\setlength{\tabcolsep}{4pt}
|
|
|
|
|
\begin{tabular}{@{}llccccc@{}}
|
|
|
|
|
\toprule
|
|
|
|
|
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
|
|
|
|
|
& $\Delta$ On & $\Delta$ Off \\
|
|
|
|
|
\midrule
|
|
|
|
|
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
\midrule
|
|
|
|
|
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
|
|
|
|
|
\bottomrule
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\par\smallskip
|
|
|
|
|
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
|
|
|
|
|
\begin{tabular}{@{}lccc@{}}
|
|
|
|
|
\toprule
|
|
|
|
|
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
|
|
|
|
|
\midrule
|
|
|
|
|
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
|
|
|
|
|
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
|
|
|
|
|
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
|
|
|
|
\bottomrule
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\end{table*}
|
|
|
|
|
|
2026-09-24 15:11:14 +08:00
|
|
|
\fi
|
2026-09-24 11:07:48 +08:00
|
|
|
\begin{table}[t]
|
2026-10-25 23:02:03 +08:00
|
|
|
\caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a
|
|
|
|
|
single-final-state readout and the same local-audio sidecar.
|
|
|
|
|
Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not
|
|
|
|
|
directly comparable across tables.}
|
2026-09-24 11:07:48 +08:00
|
|
|
\label{tab:backbone}
|
|
|
|
|
\centering\normalsize
|
|
|
|
|
\setlength{\tabcolsep}{3pt}
|
2026-10-25 23:02:03 +08:00
|
|
|
\begin{tabular}{@{}lcc@{}}
|
2026-09-24 11:07:48 +08:00
|
|
|
\toprule
|
2026-10-25 23:02:03 +08:00
|
|
|
Backbone & $\G$ & $\G+\A$ \\
|
2026-09-24 11:07:48 +08:00
|
|
|
\midrule
|
2026-10-25 23:02:03 +08:00
|
|
|
Omni 3B & 58.04 & 62.20 \\
|
|
|
|
|
Omni 7B & 57.70 & 62.18 \\
|
2026-09-24 11:07:48 +08:00
|
|
|
\bottomrule
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\end{table}
|
|
|
|
|
|
2026-09-23 07:54:36 +08:00
|
|
|
\section{Experiments}
|
|
|
|
|
\label{sec:experiments}
|
|
|
|
|
|
2026-09-24 18:52:10 +08:00
|
|
|
\textbf{Data Preparation.}
|
|
|
|
|
We use the EgoSpeak EasyCom split (Sessions~4--12 for training and
|
|
|
|
|
Sessions~1--3 for testing) and its Ego4D audiovisual manifest
|
|
|
|
|
\cite{donley2021easycom,grauman2022ego4d}, which contains 346 training and
|
|
|
|
|
87 test clips. Both datasets use 5-Hz causal cutoffs and targets, the same
|
|
|
|
|
one-second local-audio window, and visual samples strictly preceding each
|
|
|
|
|
cutoff; unavailable vision is masked rather than silently dropping rows.
|
|
|
|
|
EasyCom heads use four fixed training epochs without test-set checkpoint
|
2026-10-25 23:02:03 +08:00
|
|
|
selection. Its main deployable matrix is an archived single-seed screen;
|
|
|
|
|
the privileged information audit and context-cap screen use three seeds.
|
|
|
|
|
Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$
|
|
|
|
|
have paired seeds 0--2 under one four-epoch recipe on 87 test clips.
|
|
|
|
|
The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D
|
|
|
|
|
table \cite{kim2025egospeak}; it is not a paired rerun under our
|
|
|
|
|
causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D
|
|
|
|
|
$\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR
|
|
|
|
|
remains unmeasured.
|
2026-09-24 14:34:35 +08:00
|
|
|
|
2026-09-23 07:54:36 +08:00
|
|
|
\textbf{Metrics.}
|
2026-10-25 23:02:03 +08:00
|
|
|
Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP)
|
|
|
|
|
separately for background, wearer, and others at each horizon over common
|
|
|
|
|
causal-valid cutoff--target pairs. Their unweighted class average is the
|
|
|
|
|
horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s.
|
|
|
|
|
All reported AP and mAP values are percentages.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 14:34:35 +08:00
|
|
|
\section{Results and Analysis}
|
2026-09-23 07:54:36 +08:00
|
|
|
\label{sec:results}
|
|
|
|
|
|
2026-10-25 23:02:03 +08:00
|
|
|
\noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).}
|
|
|
|
|
All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores
|
|
|
|
|
81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus
|
|
|
|
|
73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released
|
|
|
|
|
EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84
|
|
|
|
|
points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores
|
|
|
|
|
79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion
|
|
|
|
|
gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$
|
|
|
|
|
80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published
|
|
|
|
|
EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired
|
|
|
|
|
$\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s.
|
|
|
|
|
The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets.
|
|
|
|
|
Because these tables report mAP rather than classwise AP, they do not
|
|
|
|
|
identify whether background, wearer, or others drives each gain.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-10-25 23:02:03 +08:00
|
|
|
\noindent\textbf{Visual and privileged evidence
|
|
|
|
|
(Tables~\ref{tab:main} and~\ref{tab:visual}).}
|
|
|
|
|
EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and
|
|
|
|
|
82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$.
|
|
|
|
|
Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s.
|
|
|
|
|
At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$,
|
|
|
|
|
so the visual difference is not uniform over horizons.
|
|
|
|
|
Ego4D's single SVA run scores 81.39/70.93\% against the three-seed
|
|
|
|
|
$\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at
|
|
|
|
|
2.0~s. Head/training differences prevent isolating a visual effect.
|
|
|
|
|
In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed
|
|
|
|
|
activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR
|
|
|
|
|
scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean
|
|
|
|
|
gaps over visibility-only suggest useful activity timing, but transcript
|
|
|
|
|
labels are unavailable at deployment and are not an upper bound. None of
|
|
|
|
|
these screens establishes speaker grounding or fewer false wearer entries.
|
|
|
|
|
Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a
|
|
|
|
|
population-level or modality-specific gain.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-10-25 23:02:03 +08:00
|
|
|
\noindent\textbf{Context and chunk construction
|
|
|
|
|
(Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).}
|
|
|
|
|
The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at
|
|
|
|
|
4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank
|
|
|
|
|
is 0.37/0.44 points above 4/12~s, but predates joint extraction and no
|
|
|
|
|
matched $\G$ cap arm exists. In the
|
|
|
|
|
separate global-only chunk screen, native hold scores 59.62/55.19\% at
|
|
|
|
|
0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score
|
|
|
|
|
67.05/57.84\% with 95,453 calls. The observed-tail row scores
|
|
|
|
|
70.85/59.17\%, with call count unavailable. Commits use roughly ten times
|
|
|
|
|
as many processor calls as native hold; this is not a latency ratio.
|
|
|
|
|
Changed chunk construction
|
|
|
|
|
and unmatched extraction preclude an isolated PTR or throughput claim.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-10-25 23:02:03 +08:00
|
|
|
\noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).}
|
|
|
|
|
With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\%
|
|
|
|
|
(+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface
|
|
|
|
|
differs from the four-layer main readout and does not rank backbone sizes.
|
|
|
|
|
Across all tables, AP/mAP assess future activity rather than semantic
|
|
|
|
|
floor-control decisions.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
|
|
|
|
\section{Conclusion}
|
|
|
|
|
|
2026-09-24 15:44:25 +08:00
|
|
|
For egocentric when-to-speak forecasting, cutoff-aligned access to recent
|
|
|
|
|
encoder evidence is a useful complement to frozen Omni context on EasyCom;
|
2026-10-25 23:02:03 +08:00
|
|
|
the measured mAP separation is largest near the prediction cutoff.
|
2026-09-24 15:44:25 +08:00
|
|
|
Pooled vision recovers little of the timely visible-activity information
|
|
|
|
|
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
|
|
|
|
|
but its independent predictive and efficiency effects still require matched
|
2026-10-25 23:02:03 +08:00
|
|
|
tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small
|
|
|
|
|
descriptive SVA gain in one run; censored event evaluation remains
|
|
|
|
|
necessary before claiming generalizable or improved semantic floor
|
|
|
|
|
behavior. These results therefore support dense future-activity prediction,
|
|
|
|
|
while the visual screens remain descriptive and the transcript-derived
|
|
|
|
|
activity audit remains an information probe. A decisive test must hold
|
|
|
|
|
cutoffs, context caps, training recipe, and valid pairs fixed while crossing
|
|
|
|
|
PTR with local audio. It should then report per-session variability and
|
|
|
|
|
event-level false entries and yields after censoring competing speaker
|
|
|
|
|
changes. Only then can forecast gains support claims about an agent's floor
|
|
|
|
|
decisions.
|
2026-09-23 07:54:36 +08:00
|
|
|
|
2026-09-24 11:07:48 +08:00
|
|
|
\clearpage
|
2026-09-23 07:54:36 +08:00
|
|
|
\bibliographystyle{IEEEbib}
|
|
|
|
|
\bibliography{references}
|
|
|
|
|
\end{document}
|