Files
icassp2027-experiment/manuscript/targeted/main.tex
T

471 lines
24 KiB
TeX
Raw Normal View History

2026-09-23 07:54:36 +08:00
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
% Uses the repository's ICASSP 2027 author kit.
% Four technical pages plus an optional reference-only page.
2026-09-23 07:54:36 +08:00
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
2026-09-23 07:54:36 +08:00
\hypersetup{hidelinks}
\newcommand{\vect}[1]{\boldsymbol{#1}}
\newcommand{\G}{\mathrm{G}}
\newcommand{\A}{\mathrm{A}}
\newcommand{\V}{\mathrm{V}}
\setlength{\emergencystretch}{2em}
\setcounter{topnumber}{5}
\setcounter{dbltopnumber}{5}
\renewcommand{\topfraction}{0.95}
\renewcommand{\dbltopfraction}{0.95}
\renewcommand{\textfraction}{0.05}
\renewcommand{\floatpagefraction}{0.75}
\renewcommand{\dblfloatpagefraction}{0.75}
\title{Egocentric Speech Anticipation with Omni Models:\\
Short-Window Evidence and Partial-Tail Refresh}
\name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\address{China Telecom Corporation Limited Shanghai Branch}
2026-09-23 07:54:36 +08:00
\begin{document}
\ninept
\maketitle
\begin{abstract}
An egocentric conversational agent must anticipate when its unseen wearer
will speak while other participants exchange turns in a causal multiparty
audio--video stream. Integrated Omni states provide broad audiovisual
context, yet recent evidence near each prediction cutoff may be difficult
for a small forecasting head to access. We study short-window encoder
evidence (SWE), which exposes the preceding second of audio and optional
causal visual features alongside frozen Qwen2.5-Omni context. We also
specify partial-tail refresh (PTR), which queries only the observed prefix
of the current audiovisual block at each 200-ms cutoff without committing
that prefix to history. We predict background, wearer, and aggregate-other
activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory
global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean
mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision
adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches
65.37\%. On Ego4D, paired three-seed global-plus-audio models improve
ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%,
while a single visual-attention run reaches 70.93\%. These results highlight
the value of recent encoder evidence for dense wearer-activity forecasting
across both datasets, with strictly causal inputs and a frozen backbone
throughout.
2026-09-23 07:54:36 +08:00
\end{abstract}
\begin{keywords}
egocentric speech anticipation, multimodal foundation models, causal
streaming, voice activity projection, conversational agents
\end{keywords}
\section{Introduction}
\label{sec:intro}
2026-09-24 17:15:16 +08:00
An egocentric conversational agent could answer situated questions, supply
context, and assist its wearer during face-to-face conversation. To
participate naturally, it must decide not only \emph{what} to say but when to
begin, wait, or stop speaking. This requires interpreting mixed audio,
visible people's behavior, and linguistic content as the number and roles of
interlocutors change. A pause is not necessarily an invitation: two people
visible to the wearer may be talking to each other, and their turn exchange
should not trigger the agent's response. Since the wearer is normally
off-camera, the relevant target is not simply which visible face speaks now,
but whether the wearer is about to speak or cease speaking.
Both boundaries matter: the agent should withhold an entry during a
third-party exchange and yield when another participant takes the floor.
From a moving first-person view, face visibility and overlapping voices can
change independently; neither a silence threshold nor one detected face
provides a fixed map of who is interacting with the wearer. These timing
errors arise before response content is considered.
2026-09-23 07:54:36 +08:00
2026-09-24 17:15:16 +08:00
Prior work addresses complementary parts of this decision. Voice activity
detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice
activity projection (VAP) predicts future speaker activity
\cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and
linguistic evidence for spoken-agent turn detection
\cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel
prediction exploits visible cues but is commonly evaluated with two
interlocutors \cite{russell2025mmvap,lin2025backchannel,
oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed
three-person conversations
\cite{elmers2025triadic}; online three-party EOT has also used controlled
multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of
speakers using single-camera video and face tracks, but studies visible
speakers from a
static third-party view \cite{qi2026muvap}. Device-directed speech detection
can combine acoustics, ASR output, and text to ask whether an utterance
addresses an assistant, yet the cited model does not observe visual
relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly
forecasts the unseen wearer's future speech from causal
first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak,
donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction,
first-person speaking/listening graphs, and addressee detection provide
related but distinct targets \cite{serj2024threem,jia2024avconv,
kong2024sicnet}. Egocentric active-speaker localization likewise concerns
current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}.
Together, these studies leave open how an integrated streaming model should
retain cutoff-local evidence for forecasting the unseen wearer's activity in
a natural multiparty, first-person stream. This is a representation-and-timing
question, not a claim to first egocentric turn prediction. Our activity
labels do not establish addressee or semantic floor ownership.
2026-09-23 07:54:36 +08:00
2026-09-24 17:15:16 +08:00
Omni models couple modality encoders to a shared language-model state, so
audio, video, and text can inform streaming interaction and longer-range
semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
time-aligned audio--video interleaving feed a Thinker for integrated
understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
The main EasyCom comparison uses frozen 3B Thinker readouts; a separate
seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our
2026-09-24 17:15:16 +08:00
wearer forecast updates every 200~ms. Holding only completed blocks leaves
stale context between boundaries; forcing 200-ms commits changes the native
block construction and may alter its representation. Neither implies that
all Omni models lack local evidence. An integrated, multi-layer state can
carry broader audiovisual and linguistic context, yet transient cues may be
less accessible to a small head than encoder-side features themselves
\cite{lin2023lavish,rouditchenko2024whisperflamingo}.
2026-09-23 07:54:36 +08:00
2026-09-24 17:15:16 +08:00
We therefore retain the global Thinker state while exposing recent modality
evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
suggest that global context and local audio are complementary; gains from
pooled vision and isolated PTR effects remain unproven. This study makes
three contributions: (1) short-window encoder evidence (SWE) supplies recent
audio and optional visual encoder evidence to a frozen-Omni forecasting head;
2026-09-24 17:15:16 +08:00
(2) partial-tail refresh (PTR) queries the observed current-block prefix
without committing it to persistent history; and (3) detailed EasyCom
analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run.
Matched PTR and event behavior remain pending.
2026-09-23 07:54:36 +08:00
\begin{figure}[t]
2026-09-23 07:54:36 +08:00
\centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
with recent audio and optional local visual evidence in the forecasting
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
the observed current-block tail at each 200-ms cutoff. The tail is not
committed to persistent history.}
2026-09-23 07:54:36 +08:00
\label{fig:overview}
\end{figure}
2026-09-23 07:54:36 +08:00
\section{Method}
\label{sec:method}
\subsection{Task and Global--Local Fusion}
2026-09-23 07:54:36 +08:00
At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by
$t_n$ and predicts activity at $t_n+h$, where
$h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
2026-09-23 07:54:36 +08:00
horizon has three channels in fixed order: background, camera wearer, and
aggregate others. Wearer and others may overlap; no output identifies an
individual speaker, addressee, or semantic end of turn.
The global representation $\vect g_n$ concatenates the final four frozen
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
already integrates audio and video over the available context; $\G$ is not
an audio-free baseline. Short-window encoder evidence (SWE) adds a direct,
cutoff-aligned path for recent audio $\vect a_n$ and, when enabled, visual
evidence $\vect v_n$. This separates the question of what the global state
encodes from whether a small prediction head can access transient cues.
2026-09-23 07:54:36 +08:00
In the pooled fusion, each available local vector is layer-normalized,
projected to 256 dimensions, and accompanied by a validity bit. The
concatenated global and local features are passed to a layer-normalized MLP:
2026-09-23 07:54:36 +08:00
\begin{equation}
\widehat{\vect Y}_{n,h,c}=\operatorname{softmax}_{c}
F_{\theta}\!\left([\vect g_n;\widetilde{\vect a}_n;m_n^a;
\widetilde{\vect v}_n;m_n^v]\right)_{h,c},
2026-09-23 07:54:36 +08:00
\label{eq:fusion}
\end{equation}
where a disabled branch is omitted and $F_\theta$ produces ten sets of
three logits. For token-preserving visual variants, $\vect v_n$ denotes
an attention-pooled context: a query formed from $\vect g_n$ and valid
$\vect a_n$ attends to the causal visual token set (and a learned null
token), rather than averaging it beforehand. The resulting context and
validity bit enter the same prediction head. We train the projections,
visual aggregator when present, and head with the EgoSpeak multi-positive
cross-entropy over causal-valid source--horizon pairs. The principal SWE
configuration freezes all feature encoders and the Thinker: this is
frozen-backbone adaptation, not task-level end-to-end training.
We separate \emph{where} parameters are updated from \emph{which} Thinker
layers are read. The primary comparison trains only the layer-combination
projection, local-fusion modules, and 30-logit prediction head on cached
frozen features. Parameter-adaptation controls instead attach low-rank
adapters to Thinker attention projections, either across the configured
Thinker or restricted to its upper layers; the prediction head may be
updated jointly or held fixed to isolate representation changes. These
controls require the same causal media, source cutoffs, supervision,
initialization, and model-selection protocol as the frozen-head condition.
Archived LoRA runs, including an upper-eight-layer setting, are
exploratory and not a matched layer-scope ablation. Last-four-layer
\emph{readout} fusion changes the head input, not the LoRA update scope;
neither its gain nor an unmatched LoRA score establishes an adaptation
advantage.
2026-09-23 07:54:36 +08:00
\subsection{Causal Partial-Tail Refresh}
2026-09-23 07:54:36 +08:00
PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
observed current tail are available; span varies with block phase. Reusing
completed modality payloads, we crop the tail at $t_n$ and query it in a
temporary cache. The tail and query are not committed. Only completed
blocks enter persistent history; session and source discontinuities reset
caches. Thus each 5-Hz readout sees the observed prefix without future
backfill. To test whether access policy matters, the matched design crosses
completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap;
the separate 2--12-s cap sweep tests context length. Those comparisons are
pending, so PTR is a causal access rule rather than a demonstrated gain.
2026-09-23 07:54:36 +08:00
\subsection{Short-Window Feature Extraction}
2026-09-23 07:54:36 +08:00
Each local branch observes only media preceding the prediction cutoff
$t_n$; unavailable observations retain explicit validity masks rather
than borrowing later samples. All encoders are frozen. The notation below
distinguishes encoder choice from fusion choice: $\G$ is the global Thinker
state, and local branches are appended to the same forecasting head.
The default audio feature $\A$ encodes the mixed waveform in
$[t_n-1\,\mathrm{s},t_n)$ with the Qwen2.5-Omni audio encoder and averages
valid tokens. $\A_{\rm E}$ denotes EgoSpeak's wav2vec~2.0 convolutional
representation \cite{kim2025egospeak}; comparing $\G+\A$ with
$\G+\A_{\rm E}$ requires the same window and head. Released EgoSpeak A+V
remains an external baseline, not this encoder substitution.
For vision, Pool averages Qwen tokens within provided person boxes and
across valid 2-FPS frames in the preceding second; the boxes are not
outputs of an evaluated online detector. Spatial visual attention (SVA)
instead lets $\G+\A$ query the unpooled spatial tokens of the latest
strictly pre-cutoff full frame. Temporal spatial visual attention (TSVA)
first applies a shared GRU to up to two causal tokens at each matching
grid position, then uses the same query. TSVA links image coordinates,
not person tracks. We denote these variants by $\G+\A+$Pool,
$\G+\A+$SVA, and $\G+\A+$TSVA.
2026-09-23 07:54:36 +08:00
2026-09-24 15:11:14 +08:00
\input{tables_compact}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf}
\caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending
at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap;
(B) other-to-other handover. Outcome-selected, unreviewed examples do not
establish an event-level gain.}
\label{fig:hq_cases}
\end{figure*}
2026-09-24 15:11:14 +08:00
% Historical full-precision table layouts retained as source audit only.
\iffalse
\begin{table*}[t]
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
three-seed privileged past-activity audit, which is not deployable. Mean is
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
TSVA is a no-box, two-frame causal full-frame screen; it and the historical
deployable rows are exploratory seed-0 runs with unmatched head/training
details. F1 has three seeds. Dashes are unavailable, not zero.}
\label{tab:main}
\centering
\normalsize
\renewcommand{\arraystretch}{0.87}
\setlength{\tabcolsep}{5.0pt}
\begin{tabular}{@{}lccccc@{}}
\toprule
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
\midrule
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
& 65.400 & 26.019 & 45.371 \\\\
No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\
Current-frame person set F1 & $81.777{\pm}0.156$ & --
& $64.951{\pm}0.095$ & -- & -- \\\\
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
\bottomrule
\end{tabular}
\par\smallskip
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
\normalsize
\setlength{\tabcolsep}{3.2pt}
\begin{tabular}{@{}lccc@{}}
\toprule
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
\midrule
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
& $\mathbf{66.951{\pm}.022}$ \\\\
\bottomrule
\end{tabular}
\end{table*}
\begin{table*}[t]
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
\label{tab:stream_context}
\centering\normalsize
\renewcommand{\arraystretch}{0.87}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{@{}llccccc@{}}
\toprule
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
& $\Delta$ On & $\Delta$ Off \\
\midrule
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
\midrule
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
\bottomrule
\end{tabular}
\par\smallskip
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
\begin{tabular}{@{}lccc@{}}
\toprule
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
\midrule
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
\bottomrule
\end{tabular}
\end{table*}
2026-09-24 15:11:14 +08:00
\fi
\begin{table}[t]
\caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a
single-final-state readout and the same local-audio sidecar.
Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not
directly comparable across tables.}
\label{tab:backbone}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lcc@{}}
\toprule
Backbone & $\G$ & $\G+\A$ \\
\midrule
Omni 3B & 58.04 & 62.20 \\
Omni 7B & 57.70 & 62.18 \\
\bottomrule
\end{tabular}
\end{table}
2026-09-23 07:54:36 +08:00
\section{Experiments}
\label{sec:experiments}
\textbf{Data Preparation.}
We use the EgoSpeak EasyCom split (Sessions~4--12 for training and
Sessions~1--3 for testing) and its Ego4D audiovisual manifest
\cite{donley2021easycom,grauman2022ego4d}, which contains 346 training and
87 test clips. Both datasets use 5-Hz causal cutoffs and targets, the same
one-second local-audio window, and visual samples strictly preceding each
cutoff; unavailable vision is masked rather than silently dropping rows.
EasyCom heads use four fixed training epochs without test-set checkpoint
selection. Its main deployable matrix is an archived single-seed screen;
the privileged information audit and context-cap screen use three seeds.
Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$
have paired seeds 0--2 under one four-epoch recipe on 87 test clips.
The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D
table \cite{kim2025egospeak}; it is not a paired rerun under our
causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D
$\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR
remains unmeasured.
2026-09-23 07:54:36 +08:00
\textbf{Metrics.}
Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP)
separately for background, wearer, and others at each horizon over common
causal-valid cutoff--target pairs. Their unweighted class average is the
horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s.
All reported AP and mAP values are percentages.
2026-09-23 07:54:36 +08:00
\section{Results and Analysis}
2026-09-23 07:54:36 +08:00
\label{sec:results}
\noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).}
All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores
81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus
73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released
EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84
points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores
79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion
gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$
80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published
EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired
$\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s.
The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets.
Because these tables report mAP rather than classwise AP, they do not
identify whether background, wearer, or others drives each gain.
2026-09-23 07:54:36 +08:00
\noindent\textbf{Visual and privileged evidence
(Tables~\ref{tab:main} and~\ref{tab:visual}).}
EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and
82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$.
Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s.
At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$,
so the visual difference is not uniform over horizons.
Ego4D's single SVA run scores 81.39/70.93\% against the three-seed
$\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at
2.0~s. Head/training differences prevent isolating a visual effect.
In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed
activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR
scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean
gaps over visibility-only suggest useful activity timing, but transcript
labels are unavailable at deployment and are not an upper bound. None of
these screens establishes speaker grounding or fewer false wearer entries.
Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a
population-level or modality-specific gain.
2026-09-23 07:54:36 +08:00
\noindent\textbf{Context and chunk construction
(Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).}
The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at
4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank
is 0.37/0.44 points above 4/12~s, but predates joint extraction and no
matched $\G$ cap arm exists. In the
separate global-only chunk screen, native hold scores 59.62/55.19\% at
0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score
67.05/57.84\% with 95,453 calls. The observed-tail row scores
70.85/59.17\%, with call count unavailable. Commits use roughly ten times
as many processor calls as native hold; this is not a latency ratio.
Changed chunk construction
and unmatched extraction preclude an isolated PTR or throughput claim.
2026-09-23 07:54:36 +08:00
\noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).}
With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\%
(+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface
differs from the four-layer main readout and does not rank backbone sizes.
Across all tables, AP/mAP assess future activity rather than semantic
floor-control decisions.
2026-09-23 07:54:36 +08:00
\section{Conclusion}
For egocentric when-to-speak forecasting, cutoff-aligned access to recent
encoder evidence is a useful complement to frozen Omni context on EasyCom;
the measured mAP separation is largest near the prediction cutoff.
Pooled vision recovers little of the timely visible-activity information
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
but its independent predictive and efficiency effects still require matched
tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small
descriptive SVA gain in one run; censored event evaluation remains
necessary before claiming generalizable or improved semantic floor
behavior. These results therefore support dense future-activity prediction,
while the visual screens remain descriptive and the transcript-derived
activity audit remains an information probe. A decisive test must hold
cutoffs, context caps, training recipe, and valid pairs fixed while crossing
PTR with local audio. It should then report per-session variability and
event-level false entries and yields after censoring competing speaker
changes. Only then can forecast gains support claims about an agent's floor
decisions.
2026-09-23 07:54:36 +08:00
\clearpage
2026-09-23 07:54:36 +08:00
\bibliographystyle{IEEEbib}
\bibliography{references}
\end{document}