603 lines
29 KiB
TeX
603 lines
29 KiB
TeX
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
|
|
% Uses the repository's ICASSP 2027 author kit.
|
|
% Four technical pages plus an optional reference-only page.
|
|
\documentclass{article}
|
|
\usepackage{spconf}
|
|
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
|
|
\hypersetup{hidelinks}
|
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
|
\newcommand{\G}{\mathrm{G}}
|
|
\newcommand{\A}{\mathrm{A}}
|
|
\newcommand{\V}{\mathrm{V}}
|
|
\setlength{\emergencystretch}{2em}
|
|
\setcounter{topnumber}{5}
|
|
\setcounter{dbltopnumber}{5}
|
|
\renewcommand{\topfraction}{0.95}
|
|
\renewcommand{\dbltopfraction}{0.95}
|
|
\renewcommand{\textfraction}{0.05}
|
|
\renewcommand{\floatpagefraction}{0.75}
|
|
\renewcommand{\dblfloatpagefraction}{0.75}
|
|
\title{Egocentric Speech Anticipation with Omni Models:\\
|
|
Short-Window Evidence and Partial-Tail Refresh}
|
|
\name{\shortstack{Xin Wang, Dingtian Yan, Changdong Liu, Shengxiang Hu, \\
|
|
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
|
|
\address{China Telecom Corporation Limited Shanghai Branch}
|
|
|
|
\begin{document}
|
|
\ninept
|
|
\maketitle
|
|
|
|
\begin{abstract}
|
|
An egocentric conversational agent must anticipate when its unseen camera
|
|
wearer will speak from a causal multiparty audio--video stream. Turn-taking
|
|
models and streaming Omni assistants address related problems, but the
|
|
cutoff-level availability of local evidence in an integrated Omni state is
|
|
underexplored for this wearer-centered task. We examine two interfaces:
|
|
short-window encoder evidence (SWE) supplies recent audio and optional
|
|
causal visual evidence directly to a frozen-Omni forecasting head; partial-tail
|
|
refresh (PTR) queries only the observed prefix of the current audiovisual
|
|
block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio
|
|
screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
|
|
74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP
|
|
points, while an analysis-only visible-activity audit exposes further local
|
|
information. A separate chunk screen suggests sensitivity to state
|
|
freshness but does not isolate PTR. We specify an Ego4D transfer protocol;
|
|
its scores remain pending. The evidence supports causal activity
|
|
anticipation, not semantic floor control or established cross-dataset gains.
|
|
\end{abstract}
|
|
|
|
\begin{keywords}
|
|
egocentric speech anticipation, multimodal foundation models, causal
|
|
streaming, voice activity projection, conversational agents
|
|
\end{keywords}
|
|
|
|
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
An egocentric conversational agent could answer situated questions, supply
|
|
context, and assist its wearer during face-to-face conversation. To
|
|
participate naturally, it must decide not only \emph{what} to say but when to
|
|
begin, wait, or stop speaking. This requires interpreting mixed audio,
|
|
visible people's behavior, and linguistic content as the number and roles of
|
|
interlocutors change. A pause is not necessarily an invitation: two people
|
|
visible to the wearer may be talking to each other, and their turn exchange
|
|
should not trigger the agent's response. Since the wearer is normally
|
|
off-camera, the relevant target is not simply which visible face speaks now,
|
|
but whether the wearer is about to speak or cease speaking.
|
|
Both boundaries matter: the agent should withhold an entry during a
|
|
third-party exchange and yield when another participant takes the floor.
|
|
From a moving first-person view, face visibility and overlapping voices can
|
|
change independently; neither a silence threshold nor one detected face
|
|
provides a fixed map of who is interacting with the wearer. These timing
|
|
errors arise before response content is considered.
|
|
|
|
Prior work addresses complementary parts of this decision. Voice activity
|
|
detection (VAD) detects speech presence \cite{sohn1999vad}; dyadic voice
|
|
activity projection (VAP) predicts future speaker activity
|
|
\cite{ekstedt2022vap}, while Easy Turn and FastTurn combine acoustic and
|
|
linguistic evidence for spoken-agent turn detection
|
|
\cite{li2026easyturn,wang2026fastturn}. Audiovisual turn and backchannel
|
|
prediction exploits visible cues but is commonly evaluated with two
|
|
interlocutors \cite{russell2025mmvap,lin2025backchannel,
|
|
oconnorrussell2025noise}. Triadic VAP extends audio forecasting to fixed
|
|
three-person conversations
|
|
\cite{elmers2025triadic}; online three-party EOT has also used controlled
|
|
multimodal sensor data \cite{lee2024eot}. MuVAP handles a variable number of
|
|
speakers using single-camera video and face tracks, but studies visible
|
|
speakers from a
|
|
static third-party view \cite{qi2026muvap}. Device-directed speech detection
|
|
can combine acoustics, ASR output, and text to ask whether an utterance
|
|
addresses an assistant, yet the cited model does not observe visual
|
|
relations among interlocutors \cite{wagner2024ddsd}. EgoSpeak directly
|
|
forecasts the unseen wearer's future speech from causal
|
|
first-person audio--video on EasyCom and Ego4D \cite{kim2025egospeak,
|
|
donley2021easycom,grauman2022ego4d}. Multi-view next-speaker prediction,
|
|
first-person speaking/listening graphs, and addressee detection provide
|
|
related but distinct targets \cite{serj2024threem,jia2024avconv,
|
|
kong2024sicnet}. Egocentric active-speaker localization likewise concerns
|
|
current speaker evidence \cite{jiang2022asl,huh2025egoasd,clarke2025asd}.
|
|
Together, these studies leave open how an integrated streaming model should
|
|
retain cutoff-local evidence for forecasting the unseen wearer's activity in
|
|
a natural multiparty, first-person stream. This is a representation-and-timing
|
|
question, not a claim to first egocentric turn prediction. Our activity
|
|
labels do not establish addressee or semantic floor ownership.
|
|
|
|
Omni models couple modality encoders to a shared language-model state, so
|
|
audio, video, and text can inform streaming interaction and longer-range
|
|
semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}.
|
|
Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and
|
|
time-aligned audio--video interleaving feed a Thinker for integrated
|
|
understanding; a separate Talker generates speech \cite{xu2025qwenomni}.
|
|
We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a
|
|
measured result. The native audio attention block spans 2~s, whereas our
|
|
wearer forecast updates every 200~ms. Holding only completed blocks leaves
|
|
stale context between boundaries; forcing 200-ms commits changes the native
|
|
block construction and may alter its representation. Neither implies that
|
|
all Omni models lack local evidence. An integrated, multi-layer state can
|
|
carry broader audiovisual and linguistic context, yet transient cues may be
|
|
less accessible to a small head than encoder-side features themselves
|
|
\cite{lin2023lavish,rouditchenko2024whisperflamingo}.
|
|
|
|
We therefore retain the global Thinker state while exposing recent modality
|
|
evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
|
|
suggest that global context and local audio are complementary; gains from
|
|
pooled vision and isolated PTR effects remain unproven. This study makes
|
|
three contributions: (1) short-window encoder evidence (SWE) supplies recent
|
|
audio and optional visual encoder evidence to a frozen-Omni forecasting head;
|
|
(2) partial-tail refresh (PTR) queries the observed current-block prefix
|
|
without committing it to persistent history; and (3) detailed EasyCom
|
|
analyses accompany a specified Ego4D transfer protocol. Matched PTR,
|
|
Ego4D scores, and event behavior remain pending.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
|
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
|
|
with recent audio and optional local visual evidence in the forecasting
|
|
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
|
|
the observed current-block tail at each 200-ms cutoff. The tail is not
|
|
committed to persistent history.}
|
|
\label{fig:overview}
|
|
\end{figure}
|
|
|
|
\section{Method}
|
|
\label{sec:method}
|
|
|
|
\subsection{Task and Global--Local Fusion}
|
|
|
|
At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by
|
|
$t_n$ and predicts activity at $t_n+h$, where
|
|
$h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
|
|
horizon has three channels in fixed order: background, camera wearer, and
|
|
aggregate others. Wearer and others may overlap; no output identifies an
|
|
individual speaker, addressee, or semantic end of turn.
|
|
|
|
The global representation $\vect g_n$ concatenates the final four frozen
|
|
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
|
|
already integrates audio and video over the available context; $\G$ is not
|
|
an audio-free baseline. Short-window encoder evidence (SWE) adds a direct,
|
|
cutoff-aligned path for recent audio $\vect a_n$ and, when enabled, visual
|
|
evidence $\vect v_n$. This separates the question of what the global state
|
|
encodes from whether a small prediction head can access transient cues.
|
|
|
|
In the pooled fusion, each available local vector is layer-normalized,
|
|
projected to 256 dimensions, and accompanied by a validity bit. The
|
|
concatenated global and local features are passed to a layer-normalized MLP:
|
|
\begin{equation}
|
|
\widehat{\vect Y}_{n,h,c}=\operatorname{softmax}_{c}
|
|
F_{\theta}\!\left([\vect g_n;\widetilde{\vect a}_n;m_n^a;
|
|
\widetilde{\vect v}_n;m_n^v]\right)_{h,c},
|
|
\label{eq:fusion}
|
|
\end{equation}
|
|
where a disabled branch is omitted and $F_\theta$ produces ten sets of
|
|
three logits. For token-preserving visual variants, $\vect v_n$ denotes
|
|
an attention-pooled context: a query formed from $\vect g_n$ and valid
|
|
$\vect a_n$ attends to the causal visual token set (and a learned null
|
|
token), rather than averaging it beforehand. The resulting context and
|
|
validity bit enter the same prediction head. We train the projections,
|
|
visual aggregator when present, and head with the EgoSpeak multi-positive
|
|
cross-entropy over causal-valid source--horizon pairs. The principal SWE
|
|
configuration freezes all feature encoders and the Thinker: this is
|
|
frozen-backbone adaptation, not task-level end-to-end training.
|
|
|
|
We separate \emph{where} parameters are updated from \emph{which} Thinker
|
|
layers are read. The primary comparison trains only the layer-combination
|
|
projection, local-fusion modules, and 30-logit prediction head on cached
|
|
frozen features. Parameter-adaptation controls instead attach low-rank
|
|
adapters to Thinker attention projections, either across the configured
|
|
Thinker or restricted to its upper layers; the prediction head may be
|
|
updated jointly or held fixed to isolate representation changes. These
|
|
controls require the same causal media, source cutoffs, supervision,
|
|
initialization, and model-selection protocol as the frozen-head condition.
|
|
Archived LoRA runs, including an upper-eight-layer setting, are
|
|
exploratory and not a matched layer-scope ablation. Last-four-layer
|
|
\emph{readout} fusion changes the head input, not the LoRA update scope;
|
|
neither its gain nor an unmatched LoRA score establishes an adaptation
|
|
advantage.
|
|
|
|
\subsection{Causal Partial-Tail Refresh}
|
|
|
|
PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
|
|
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
|
|
observed current tail are available; span varies with block phase. Reusing
|
|
completed modality payloads, we crop the tail at $t_n$ and query it in a
|
|
temporary cache. The tail and query are not committed. Only completed
|
|
blocks enter persistent history; session and source discontinuities reset
|
|
caches. Thus each 5-Hz readout sees the observed prefix without future
|
|
backfill. To test whether access policy matters, the matched design crosses
|
|
completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap;
|
|
the separate 2--12-s cap sweep tests context length. Those comparisons are
|
|
pending, so PTR is a causal access rule rather than a demonstrated gain.
|
|
|
|
\subsection{Short-Window Feature Extraction}
|
|
|
|
Each local branch observes only media preceding the prediction cutoff
|
|
$t_n$; unavailable observations retain explicit validity masks rather
|
|
than borrowing later samples. All encoders are frozen. The notation below
|
|
distinguishes encoder choice from fusion choice: $\G$ is the global Thinker
|
|
state, and local branches are appended to the same forecasting head.
|
|
|
|
The default audio feature $\A$ encodes the mixed waveform in
|
|
$[t_n-1\,\mathrm{s},t_n)$ with the Qwen2.5-Omni audio encoder and averages
|
|
valid tokens. $\A_{\rm E}$ denotes EgoSpeak's wav2vec~2.0 convolutional
|
|
representation \cite{kim2025egospeak}; comparing $\G+\A$ with
|
|
$\G+\A_{\rm E}$ requires the same window and head. Released EgoSpeak A+V
|
|
remains an external baseline, not this encoder substitution.
|
|
|
|
For vision, Pool averages Qwen tokens within provided person boxes and
|
|
across valid 2-FPS frames in the preceding second; the boxes are not
|
|
outputs of an evaluated online detector. Spatial visual attention (SVA)
|
|
instead lets $\G+\A$ query the unpooled spatial tokens of the latest
|
|
strictly pre-cutoff full frame. Temporal spatial visual attention (TSVA)
|
|
first applies a shared GRU to up to two causal tokens at each matching
|
|
grid position, then uses the same query. TSVA links image coordinates,
|
|
not person tracks. We denote these variants by $\G+\A+$Pool,
|
|
$\G+\A+$SVA, and $\G+\A+$TSVA.
|
|
|
|
\input{tables_compact}
|
|
% Historical full-precision table layouts retained as source audit only.
|
|
\iffalse
|
|
\begin{table*}[t]
|
|
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
|
|
three-seed privileged past-activity audit, which is not deployable. Mean is
|
|
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
|
|
TSVA is a no-box, two-frame causal full-frame screen; it and the historical
|
|
deployable rows are exploratory seed-0 runs with unmatched head/training
|
|
details. F1 has three seeds. Dashes are unavailable, not zero.}
|
|
\label{tab:main}
|
|
\centering
|
|
\normalsize
|
|
\renewcommand{\arraystretch}{0.87}
|
|
\setlength{\tabcolsep}{5.0pt}
|
|
\begin{tabular}{@{}lccccc@{}}
|
|
\toprule
|
|
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
|
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
|
\midrule
|
|
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
|
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
|
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
|
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
|
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
|
|
& 65.400 & 26.019 & 45.371 \\\\
|
|
No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\
|
|
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
|
& $64.951{\pm}0.095$ & -- & -- \\\\
|
|
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
|
|
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
|
|
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\par\smallskip
|
|
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
|
|
\normalsize
|
|
\setlength{\tabcolsep}{3.2pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
|
\midrule
|
|
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
|
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
|
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
|
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
|
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
|
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
|
& $\mathbf{66.951{\pm}.022}$ \\\\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\begin{table*}[t]
|
|
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
|
|
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
|
|
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
|
|
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
|
|
\label{tab:stream_context}
|
|
\centering\normalsize
|
|
\renewcommand{\arraystretch}{0.87}
|
|
\setlength{\tabcolsep}{4pt}
|
|
\begin{tabular}{@{}llccccc@{}}
|
|
\toprule
|
|
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
|
|
& $\Delta$ On & $\Delta$ Off \\
|
|
\midrule
|
|
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
|
|
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
|
|
\midrule
|
|
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\par\smallskip
|
|
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
|
|
\midrule
|
|
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
|
|
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
|
|
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\fi
|
|
\begin{table}[t]
|
|
\caption{Pending paired $\G\rightarrow\G+\A$ comparison on EasyCom
|
|
across Qwen2.5-Omni backbone sizes (unmeasured).}
|
|
\label{tab:backbone}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{3pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
|
|
\midrule
|
|
Omni 3B & -- & -- & -- \\
|
|
Omni 7B & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{H result slots: predeclared activity slices (unmeasured). Report
|
|
positive/negative counts and per-session scores; AP is undefined for a
|
|
pure-negative slice, which instead uses calibrated FPR.}
|
|
\label{tab:hard_slices}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{2.5pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Slice & $N_+/N_-$ & AP & FPR \\
|
|
\midrule
|
|
Wearer onset & -- & -- & -- \\
|
|
Wearer offset & -- & -- & -- \\
|
|
Other pause (negative) & -- & n/a & -- \\
|
|
Overlap (negative) & -- & n/a & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{S result slots: full-path cost on the same hardware (unmeasured).
|
|
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
|
|
\label{tab:efficiency}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{3pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Stage & P50 ms & P95 ms & Peak GB \\
|
|
\midrule
|
|
Completed block & -- & -- & -- \\
|
|
Observed tail & -- & -- & -- \\
|
|
Local path + head & -- & -- & -- \\
|
|
Full path & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\section{Experiments}
|
|
\label{sec:experiments}
|
|
|
|
\textbf{Data Preparation.}
|
|
We use the EgoSpeak EasyCom split (Sessions~4--12 for training and
|
|
Sessions~1--3 for testing) and its Ego4D audiovisual manifest
|
|
\cite{donley2021easycom,grauman2022ego4d}, which contains 346 training and
|
|
87 test clips. Both datasets use 5-Hz causal cutoffs and targets, the same
|
|
one-second local-audio window, and visual samples strictly preceding each
|
|
cutoff; unavailable vision is masked rather than silently dropping rows.
|
|
EasyCom heads use four fixed training epochs without test-set checkpoint
|
|
selection. Its deployable feature matrix is an archived single-seed screen,
|
|
whereas the privileged information audit uses three seeds under one recipe;
|
|
small single-seed differences are descriptive. Ego4D extraction and training
|
|
workflows exist, but test scores and a matched PTR comparison have not yet
|
|
been archived, so cross-dataset gains are not claimed.
|
|
|
|
\textbf{Metrics.}
|
|
At each horizon we compute AP for the three activity channels over common
|
|
causal-valid rows, then average classes to obtain mAP and average all ten
|
|
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint
|
|
diagnostics, onset AP conditions source rows on the wearer being silent and
|
|
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
|
|
speech and scores future silence with $1-p(\mathrm{wearer})$. These
|
|
threshold-free quantities average ten horizons. They are not one-event-one-
|
|
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
|
|
|
|
\section{Results and Analysis}
|
|
\label{sec:results}
|
|
|
|
\subsection{EasyCom: Local Encoder Evidence}
|
|
|
|
Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate
|
|
audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and
|
|
60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon
|
|
mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26
|
|
over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
|
|
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
|
|
points, onset AP by only 1.47: dense gains do not imply equal improvement
|
|
at both activity boundaries.
|
|
|
|
The $\G$-only head has more trainable parameters and longer history than the
|
|
$\A$-only head. Thus this screen is consistent with predictive
|
|
complementarity, not a capacity-matched proof that the Thinker lacks a
|
|
specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable
|
|
under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}.
|
|
|
|
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
|
|
% in the page-limited submission preview.
|
|
\iffalse
|
|
\begin{table*}[t]
|
|
\caption{Six-group \textbf{illustrative scenario, not measured results}.
|
|
These are conditional, pre-run interpretations for the pending matrix, not
|
|
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
|
|
cutoffs; signs and shape must be replaced by measured values even if contrary
|
|
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
|
|
\label{tab:scenario}
|
|
\centering
|
|
\normalsize
|
|
\renewcommand{\arraystretch}{1.12}
|
|
\setlength{\tabcolsep}{4.2pt}
|
|
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
|
|
\toprule
|
|
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
|
|
\midrule
|
|
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
|
|
past-delayed local cues; current-state persistence
|
|
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
|
|
near-horizon benefit; persistence does not recover the gain
|
|
& Width-only or persistence matches fusion, or delay does not hurt:
|
|
do not attribute gain to timely distinct evidence. \\
|
|
\addlinespace[2.5pt]
|
|
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
|
|
crossed with $\A$ on/off; causal replay and phase audit
|
|
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
|
|
future-only media never changes prior readouts
|
|
& No independent local gain under tail refresh implies redundancy;
|
|
replay/leakage failure invalidates timing claims. \\
|
|
\addlinespace[2.5pt]
|
|
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
|
|
add $\G$-only at each cap for a window-stability claim
|
|
& Fused performance plateaus around mid-range caps while
|
|
$\G$-only remains more cap-sensitive
|
|
& If long history substitutes for $\A$, local-evidence claim narrows;
|
|
without C2, no robustness comparison is justified. \\
|
|
\addlinespace[2.5pt]
|
|
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
|
|
interface audit, MiniCPM-o 4.5
|
|
& Positive within-backbone paired $\Delta$ on all admissible rows;
|
|
absolute backbone ranking is not inferred
|
|
& Failed causal/history match is unsupported, not zero gain;
|
|
one negative paired row defeats universal transfer. \\
|
|
\addlinespace[2.5pt]
|
|
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
|
|
same-horizon, cutoff-marked case selection
|
|
& Offset gains coexist with hard wearer-onset negatives; examples
|
|
explain, rather than replace, counted slice statistics
|
|
& Pure-negative slice AP is undefined; selected examples cannot
|
|
establish reduced false-entry rate. \\
|
|
\addlinespace[2.5pt]
|
|
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
|
|
encoding, tail recomputation, local path, and boundaries
|
|
& A feasible throughput--quality tradeoff can be plotted
|
|
against 200-ms output cadence
|
|
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
|
|
not real-time deployment. \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
\fi
|
|
|
|
\subsection{EasyCom: Visual Evidence and Ablations}
|
|
|
|
Table~\ref{tab:visual} separates visual screens from the cross-dataset main
|
|
matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and
|
|
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
|
|
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
|
|
current-frame person-set model is stable across seeds but reaches lower mean
|
|
mAP than the historical pooled row under a non-matched selection protocol.
|
|
These results do not support a claim of learned speaker grounding. A separate
|
|
seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\%
|
|
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
|
|
causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a
|
|
shared GRU, while head capacity and training batch differ; raw boundary
|
|
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
|
|
difference does not isolate visual motion or floor behavior.
|
|
|
|
The lower block of Table~\ref{tab:visual} is an analysis-only audit that
|
|
substitutes two seconds
|
|
of transcript-derived, strictly past visible-person activity on the 5-Hz
|
|
grid. The displayed conditions test visibility alone, a one-second delay,
|
|
and OR-pooled activity. These privileged labels are
|
|
neither deployable visual features nor a performance upper bound.
|
|
The three-seed audio baseline is 81.87\% at 0.2~s and 65.18\% mean mAP;
|
|
aggregate visible activity improves on it by 4.826 and 1.775 points,
|
|
respectively.
|
|
Visibility alone changes mean mAP by only +0.040, and
|
|
one-second-delayed activity by +0.164. Thus availability or extra parameters
|
|
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
|
|
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
|
|
condition. The current target therefore supports timely visible-floor
|
|
activity as useful information, but not the necessity of identity continuity.
|
|
The majority of the oracle's 0.2-s macro gain comes from background and
|
|
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
|
|
dense mAP gain establishes fewer false wearer entries.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
|
|
\centering
|
|
\textbf{Q1 case slots: examples pending}\\[2pt]
|
|
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
|
|
Show source cutoff, observed activity and same-horizon wearer scores.
|
|
\end{minipage}}
|
|
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
|
|
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
|
|
behavioral improvement is asserted by this placeholder.}
|
|
\label{fig:hq_cases}
|
|
\end{figure}
|
|
|
|
\subsection{Chunk Timing and Cross-Dataset Limits}
|
|
|
|
Table~\ref{tab:chunk} reports a separate global-only
|
|
chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
|
|
at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
|
|
processor calls over 12 sessions. Completed blocks plus observed tail reach
|
|
70.85\%; that older bank is not extraction-matched. Freshness and chunk
|
|
construction vary together, so this is motivation for, not evidence of, an
|
|
isolated PTR benefit or real-time throughput. Table~\ref{tab:stream_context}
|
|
reserves the matched tail $\times$ audio and context-cap tests.
|
|
Tables~\ref{tab:backbone}--
|
|
\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
|
|
cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
|
|
\textbf{Historical LoRA diagnostic.}
|
|
A single-run EasyCom 8-s/2-FPS screen used sparse training anchors (1~Hz,
|
|
plus 5~Hz near transitions) and dense 5-Hz evaluation. Within that legacy
|
|
pipeline, strict mean mAP was 58.17\% for a frozen dense-cache control,
|
|
57.21\% for frozen sparse training, and 56.21\% for a separate three-epoch
|
|
Thinker-LoRA run initialized from a trained frozen head. An epoch-1 LoRA
|
|
weights-only continuation fell to 54.21\% after optimizer/scheduler reset.
|
|
Sparse frozen training is 0.96 points below the dense control; the separate
|
|
LoRA run is another 1.01 below sparse frozen, but the epoch budgets differ.
|
|
Sparse-train/dense-test mismatch and the optimizer reset are possible, not
|
|
isolated, explanations.
|
|
Missing remote raw metrics, matched seeds, and feature equivalence make this
|
|
a negative recipe diagnostic, not a Table~\ref{tab:main} $\G+\A$ comparison
|
|
or proof that LoRA is inherently harmful.
|
|
Ego4D has no archived test metrics in this checkout, so transfer cannot yet
|
|
be claimed. The oracle uses privileged labels, provided boxes omit online
|
|
detection, and aggregate others cannot establish addressee or semantic floor
|
|
ownership.
|
|
|
|
\subsection{Decision Rules and Reproducibility}
|
|
|
|
Matched comparisons require identical cutoffs and targets; archive source
|
|
timestamps, valid-pair counts, seeds, per-session AP, trainable counts, and
|
|
checkpoint/processor revisions. The fixed recipe and selection rules use
|
|
training-side evidence only. Already inspected test sessions make repeat
|
|
seeds exploratory. Raw endpoint AP is not one-event-one-vote ENTRY/YIELD:
|
|
response windows must end at the next other onset, and yield windows at the
|
|
next other offset. Behavioral claims await a real-timeline evaluator audit.
|
|
|
|
\section{Conclusion}
|
|
|
|
For egocentric when-to-speak forecasting, cutoff-aligned access to recent
|
|
encoder evidence is a useful complement to frozen Omni context on EasyCom;
|
|
the strongest measured boundary difference is at wearer speech offset.
|
|
Pooled vision recovers little of the timely visible-activity information
|
|
revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule,
|
|
but its independent predictive and efficiency effects still require matched
|
|
tests. Ego4D transfer and censored event evaluation remain necessary before
|
|
claiming generalizable or improved semantic floor behavior.
|
|
|
|
\section{Acknowledgment}
|
|
OpenAI Codex assisted with drafting and figure layout; the authors remain
|
|
responsible for all experimental claims and final content.
|
|
|
|
\clearpage
|
|
\bibliographystyle{IEEEbib}
|
|
\bibliography{references}
|
|
\end{document}
|