560 lines
26 KiB
TeX
560 lines
26 KiB
TeX
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
|
|
% Uses the repository's ICASSP 2027 author kit.
|
|
% Four technical pages plus an optional reference-only page.
|
|
\documentclass{article}
|
|
\usepackage{spconf}
|
|
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
|
|
\hypersetup{hidelinks}
|
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
|
\newcommand{\G}{\mathrm{G}}
|
|
\newcommand{\A}{\mathrm{A}}
|
|
\newcommand{\V}{\mathrm{V}}
|
|
\setlength{\emergencystretch}{2em}
|
|
\setcounter{topnumber}{5}
|
|
\setcounter{dbltopnumber}{5}
|
|
\renewcommand{\topfraction}{0.95}
|
|
\renewcommand{\dbltopfraction}{0.95}
|
|
\renewcommand{\textfraction}{0.05}
|
|
\renewcommand{\floatpagefraction}{0.75}
|
|
\renewcommand{\dblfloatpagefraction}{0.75}
|
|
\title{Egocentric Speech Anticipation with Omni Models:\\
|
|
Short-Window Evidence and Partial-Tail Refresh}
|
|
\name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\
|
|
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
|
|
\address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom}
|
|
|
|
\begin{document}
|
|
\ninept
|
|
\maketitle
|
|
|
|
\begin{abstract}
|
|
An egocentric conversational agent must anticipate when its invisible wearer
|
|
will speak while observing a causal multiparty audio--video stream. Although
|
|
turn-taking prediction and omni-modal assistants have advanced, the temporal
|
|
accessibility of local evidence in an integrated Omni state remains unclear.
|
|
We study two interfaces: a short-window encoder-evidence path that exposes
|
|
recent audio and optional person-crop vision to a frozen-Omni forecasting
|
|
head, and partial-tail refresh that queries the observed prefix of an
|
|
incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the
|
|
exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\%
|
|
ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled
|
|
vision adds only 0.29 mean-mAP points, whereas an analysis-only causal
|
|
visible-activity oracle reveals additional information. A separate chunk
|
|
screen suggests sensitivity to state freshness, but does not isolate the
|
|
partial-tail effect. These results motivate finer local evidence access;
|
|
they do not establish semantic floor control or cross-dataset transfer.
|
|
\end{abstract}
|
|
|
|
\begin{keywords}
|
|
egocentric speech anticipation, multimodal foundation models, causal
|
|
streaming, voice activity projection, conversational agents
|
|
\end{keywords}
|
|
|
|
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
An egocentric conversational agent must decide \emph{when} to speak while its
|
|
camera wearer is usually invisible. It sees other participants and hears a
|
|
mixed conversation, but must predict the wearer's own future speech before a
|
|
turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models
|
|
and voice-activity projection (VAP) show the value of anticipatory rather than
|
|
silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap,
|
|
elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap,
|
|
jiang2026turnbench}. Yet these tasks differ in sensor placement, visible
|
|
speakers, dialogue roles, and whether the target is a future activity state
|
|
or a semantic intervention.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
|
|
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
|
|
only the past audio--video prefix is input; ten future activity states are
|
|
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
|
|
not semantic floor ownership.}
|
|
\label{fig:task}
|
|
\end{figure}
|
|
|
|
\textbf{Egocentric audiovisual conversation.}
|
|
EgoSpeak forecasts wearer speech from first-person RGB and audio using
|
|
specialized encoders and a temporal fusion model \cite{kim2025egospeak};
|
|
EasyCom and Ego4D supply complementary egocentric conversation settings
|
|
\cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker
|
|
anticipation, egocentric speaking/listening graphs, and addressee detection
|
|
study related but different targets \cite{serj2024threem,jia2024avconv,
|
|
kong2024sicnet}. Egocentric active-speaker localization and detection
|
|
\cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person
|
|
modeling \cite{wang2024loconet} estimate who is speaking now; they do not by
|
|
themselves forecast an unseen wearer's future activity. Visual cues can help
|
|
turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled
|
|
vision conveys the relevant local floor evidence here is empirical.
|
|
|
|
\textbf{Omni and local evidence.}
|
|
Omni systems integrate modalities and streaming interaction
|
|
\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
|
|
yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side
|
|
audio--visual adaptation offers another route to accessible modality evidence
|
|
\cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither
|
|
large-context integration nor a nominally streaming interface guarantees a
|
|
fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni
|
|
extraction, completed 2-s blocks and an incomplete current block have
|
|
different update rules. We therefore test two distinct interfaces:
|
|
\emph{short-window encoder evidence} (SWE) exposes recent modality features
|
|
outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR)
|
|
queries only the observed current-block prefix. Neither term implies that
|
|
all Omni models use the same chunk duration or lack local perception.
|
|
|
|
We contribute a causal implementation of SWE and PTR for wearer activity
|
|
anticipation; an EasyCom audit separating global, local-audio, pooled-visual,
|
|
and privileged information conditions; and a protocol for EasyCom and Ego4D
|
|
transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D
|
|
scores are still pending, so our measured claims remain confined to the
|
|
available EasyCom screens.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
|
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
|
|
audio and optional pooled person-crop vision to a frozen-Omni head. PTR
|
|
temporarily queries the observed current-block prefix while reusing completed
|
|
2-s payloads. Privileged past-activity labels are not model inputs.}
|
|
\label{fig:overview}
|
|
\end{figure}
|
|
|
|
\section{Method}
|
|
\label{sec:method}
|
|
|
|
\subsection{Task and Short-Window Encoder Evidence}
|
|
|
|
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
|
|
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
|
|
horizon has three channels in fixed order: background, camera wearer, and
|
|
aggregate others. Wearer and others may overlap; no output identifies an
|
|
individual speaker, addressee, or semantic end of turn.
|
|
|
|
The global feature $\vect g_n$ concatenates the final four frozen
|
|
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
|
|
already contains audio and video context; $\G$ is not audio-free. SWE
|
|
adds encoder-side features without requiring the Thinker state to preserve
|
|
their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed
|
|
waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and
|
|
averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens
|
|
inside provided person boxes, then across people and valid causal frames in
|
|
the latest second. This pooling has no identity or mouth-motion trajectory;
|
|
provided boxes do not imply an online detector.
|
|
|
|
Each enabled local feature is layer-normalized, projected to 256 dimensions,
|
|
and paired with a validity bit. The trainable head is
|
|
\begin{equation}
|
|
\widehat{\vect Y}_{n}=
|
|
\operatorname{softmax}_{c}F_{\theta}
|
|
\bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a;
|
|
\widetilde{\vect v}_n;m_n^v\bigr],
|
|
\label{eq:fusion}
|
|
\end{equation}
|
|
where omitted modalities remove their terms and $F_\theta$ is a
|
|
layer-normalized MLP with a 30-logit output. We train only the projections and
|
|
head with the EgoSpeak multi-positive cross-entropy over causal-valid
|
|
source--horizon pairs. The Omni and modality encoders remain frozen, so this
|
|
is frozen-backbone adaptation rather than end-to-end training.
|
|
|
|
\subsection{Causal Partial-Tail Refresh}
|
|
|
|
PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
|
|
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
|
|
observed current tail are available; span varies with block phase. Reusing
|
|
completed modality payloads, we crop the tail at $t_n$ and query it in a
|
|
temporary cache. The tail and query are not committed. Only completed
|
|
blocks enter persistent history; session and source discontinuities reset
|
|
caches. Thus each 5-Hz readout sees the observed prefix without future
|
|
backfill. PTR is a causal access rule, not by itself a proven accuracy or
|
|
real-time latency improvement.
|
|
|
|
\subsection{Visible-Activity Information Audit}
|
|
|
|
To distinguish missing information from ineffective learned vision, we build
|
|
analysis-only features from transcript-derived \emph{past} activity. At each
|
|
cutoff, visible non-wearer participants receive two seconds of binary history
|
|
sampled on the same 5-Hz grid. The preparation fails if its aggregate
|
|
activity disagrees with the released target. We compare: (i) visibility with
|
|
activity zeroed, (ii) per-person histories, (iii) identities shuffled while
|
|
preserving active-person counts at every lag, (iv) histories delayed by one
|
|
second, and (v) one OR-pooled aggregate-visible history. These conditions
|
|
never expose future activity, but their speech labels make them privileged
|
|
diagnostics, not deployable features or upper bounds.
|
|
|
|
\input{tables_compact}
|
|
% Historical full-precision table layouts retained as source audit only.
|
|
\iffalse
|
|
\begin{table*}[t]
|
|
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
|
|
three-seed privileged past-activity audit, which is not deployable. Mean is
|
|
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
|
|
F3 is a no-box, two-frame causal full-frame screen; it and the historical
|
|
deployable rows are exploratory seed-0 runs with unmatched head/training
|
|
details. F1 has three seeds. Dashes are unavailable, not zero.}
|
|
\label{tab:main}
|
|
\centering
|
|
\normalsize
|
|
\renewcommand{\arraystretch}{0.87}
|
|
\setlength{\tabcolsep}{5.0pt}
|
|
\begin{tabular}{@{}lccccc@{}}
|
|
\toprule
|
|
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
|
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
|
\midrule
|
|
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
|
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
|
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
|
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
|
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
|
|
& 65.400 & 26.019 & 45.371 \\\\
|
|
No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\
|
|
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
|
& $64.951{\pm}0.095$ & -- & -- \\\\
|
|
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
|
|
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
|
|
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\par\smallskip
|
|
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
|
|
\normalsize
|
|
\setlength{\tabcolsep}{3.2pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
|
\midrule
|
|
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
|
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
|
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
|
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
|
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
|
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
|
& $\mathbf{66.951{\pm}.022}$ \\\\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\begin{table*}[t]
|
|
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
|
|
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
|
|
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
|
|
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
|
|
\label{tab:stream_context}
|
|
\centering\normalsize
|
|
\renewcommand{\arraystretch}{0.87}
|
|
\setlength{\tabcolsep}{4pt}
|
|
\begin{tabular}{@{}llccccc@{}}
|
|
\toprule
|
|
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
|
|
& $\Delta$ On & $\Delta$ Off \\
|
|
\midrule
|
|
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
|
|
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
|
|
\midrule
|
|
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\par\smallskip
|
|
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
|
|
\midrule
|
|
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
|
|
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
|
|
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\fi
|
|
\begin{table}[t]
|
|
\caption{Pending paired $\G\rightarrow\G+\A$ transfer (unmeasured).
|
|
EC is EasyCom; MiniCPM first needs readout/refresh/history audits.}
|
|
\label{tab:backbone}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{3pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
|
|
\midrule
|
|
EC: Omni 3B & -- & -- & -- \\
|
|
EC: Omni 7B & -- & -- & -- \\
|
|
EC: MiniCPM-o 4.5 & -- & -- & -- \\
|
|
Ego4D: Omni 3B & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{H result slots: predeclared activity slices (unmeasured). Report
|
|
positive/negative counts and per-session scores; AP is undefined for a
|
|
pure-negative slice, which instead uses calibrated FPR.}
|
|
\label{tab:hard_slices}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{2.5pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Slice & $N_+/N_-$ & AP & FPR \\
|
|
\midrule
|
|
Wearer onset & -- & -- & -- \\
|
|
Wearer offset & -- & -- & -- \\
|
|
Other pause (negative) & -- & n/a & -- \\
|
|
Overlap (negative) & -- & n/a & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{S result slots: full-path cost on the same hardware (unmeasured).
|
|
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
|
|
\label{tab:efficiency}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{3pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Stage & P50 ms & P95 ms & Peak GB \\
|
|
\midrule
|
|
Completed block & -- & -- & -- \\
|
|
Observed tail & -- & -- & -- \\
|
|
Local path + head & -- & -- & -- \\
|
|
Full path & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\section{Experiments}
|
|
\label{sec:experiments}
|
|
|
|
\textbf{Data and protocol.}
|
|
We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training
|
|
and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads
|
|
are trained for four fixed epochs on all training sessions without test-time
|
|
checkpoint selection. The deployable feature matrix is an archived
|
|
single-seed screen; the information audit uses seeds 0, 1, and 2 with an
|
|
identical fixed recipe. We therefore report standard deviations only for the
|
|
latter and treat small differences in the former descriptively.
|
|
|
|
\textbf{Ego4D transfer protocol.}
|
|
Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak
|
|
manifest and targets, with 346 training and 87 test clips. The same 5-Hz
|
|
cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and
|
|
$\G+\A+\V$ comparisons; visual validity is retained rather than silently
|
|
dropping no-box rows. The repository contains the extraction and training
|
|
workflow but no archived Ego4D score or matched PTR control. Hence this
|
|
protocol is specified, not reported as an observed cross-dataset gain.
|
|
|
|
\textbf{Metrics.}
|
|
At each horizon we compute AP for the three activity channels over common
|
|
causal-valid rows, then average classes to obtain mAP and average all ten
|
|
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint
|
|
diagnostics, onset AP conditions source rows on the wearer being silent and
|
|
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
|
|
speech and scores future silence with $1-p(\mathrm{wearer})$. These
|
|
threshold-free quantities average ten horizons. They are not one-event-one-
|
|
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
|
|
|
|
\textbf{Comparisons.}
|
|
We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni
|
|
$\G$; $\G+\A$; and $\G+\A+$pooled-$\V$. A three-seed current-frame
|
|
frame-set attention screen tests whether a richer person aggregation is
|
|
already sufficient. Because that screen used Session~12 model selection
|
|
whereas the historical pooled row did not, their difference is diagnostic,
|
|
not a matched causal attribution.
|
|
|
|
\textbf{Six pre-specified control families.}
|
|
We freeze six pending control families: (L) capacity and recency, (T) tail
|
|
refresh crossed with local audio, (C) six context caps, (B) paired backbone
|
|
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
|
|
They share the official split and physical cutoffs; the 8-s T/C row reuses
|
|
one run. The source retains unmeasured hypotheses and falsifiers.
|
|
|
|
\section{Results and Analysis}
|
|
\label{sec:results}
|
|
|
|
\subsection{EasyCom: Local Encoder Evidence}
|
|
|
|
Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at
|
|
0.2~s while integrated $\G$ is slightly stronger across all horizons.
|
|
Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over
|
|
$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
|
|
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
|
|
points, onset AP by only 1.47: dense gains do not imply equal improvement
|
|
at both activity boundaries.
|
|
|
|
The $\G$-only head has more trainable parameters and longer history than the
|
|
$\A$-only head. Thus this screen shows predictive complementarity, not a
|
|
capacity-matched proof that the Thinker lacks a specific local cue.
|
|
|
|
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
|
|
% in the page-limited submission preview.
|
|
\iffalse
|
|
\begin{table*}[t]
|
|
\caption{Six-group \textbf{illustrative scenario, not measured results}.
|
|
These are conditional, pre-run interpretations for the pending matrix, not
|
|
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
|
|
cutoffs; signs and shape must be replaced by measured values even if contrary
|
|
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
|
|
\label{tab:scenario}
|
|
\centering
|
|
\normalsize
|
|
\renewcommand{\arraystretch}{1.12}
|
|
\setlength{\tabcolsep}{4.2pt}
|
|
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
|
|
\toprule
|
|
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
|
|
\midrule
|
|
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
|
|
past-delayed local cues; current-state persistence
|
|
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
|
|
near-horizon benefit; persistence does not recover the gain
|
|
& Width-only or persistence matches fusion, or delay does not hurt:
|
|
do not attribute gain to timely distinct evidence. \\
|
|
\addlinespace[2.5pt]
|
|
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
|
|
crossed with $\A$ on/off; causal replay and phase audit
|
|
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
|
|
future-only media never changes prior readouts
|
|
& No independent local gain under tail refresh implies redundancy;
|
|
replay/leakage failure invalidates timing claims. \\
|
|
\addlinespace[2.5pt]
|
|
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
|
|
add $\G$-only at each cap for a window-stability claim
|
|
& Fused performance plateaus around mid-range caps while
|
|
$\G$-only remains more cap-sensitive
|
|
& If long history substitutes for $\A$, local-evidence claim narrows;
|
|
without C2, no robustness comparison is justified. \\
|
|
\addlinespace[2.5pt]
|
|
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
|
|
interface audit, MiniCPM-o 4.5
|
|
& Positive within-backbone paired $\Delta$ on all admissible rows;
|
|
absolute backbone ranking is not inferred
|
|
& Failed causal/history match is unsupported, not zero gain;
|
|
one negative paired row defeats universal transfer. \\
|
|
\addlinespace[2.5pt]
|
|
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
|
|
same-horizon, cutoff-marked case selection
|
|
& Offset gains coexist with hard wearer-onset negatives; examples
|
|
explain, rather than replace, counted slice statistics
|
|
& Pure-negative slice AP is undefined; selected examples cannot
|
|
establish reduced false-entry rate. \\
|
|
\addlinespace[2.5pt]
|
|
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
|
|
encoding, tail recomputation, local path, and boundaries
|
|
& A feasible throughput--quality tradeoff can be plotted
|
|
against 200-ms output cadence
|
|
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
|
|
not real-time deployment. \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
\fi
|
|
|
|
\subsection{EasyCom: Visual Evidence and Ablations}
|
|
|
|
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
|
|
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
|
|
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
|
|
current-frame person-set model is stable across seeds but reaches lower mean
|
|
mAP than the historical pooled row under a non-matched selection protocol.
|
|
These results do not support a claim of learned speaker grounding. A separate
|
|
seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\%
|
|
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
|
|
causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a
|
|
shared GRU, while head capacity and training batch differ; raw boundary
|
|
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
|
|
difference does not isolate visual motion or floor behavior.
|
|
|
|
Table~\ref{tab:oracle} locates the missing information. Aggregate visible
|
|
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
|
|
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
|
|
one-second-delayed activity by +0.164. Thus availability or extra parameters
|
|
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
|
|
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
|
|
condition. The current target therefore supports timely visible-floor
|
|
activity as useful information, but not the necessity of identity continuity.
|
|
The majority of the oracle's 0.2-s macro gain comes from background and
|
|
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
|
|
dense mAP gain establishes fewer false wearer entries.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
|
|
\centering
|
|
\textbf{Q1 case slots: examples pending}\\[2pt]
|
|
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
|
|
Show source cutoff, observed activity and same-horizon wearer scores.
|
|
\end{minipage}}
|
|
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
|
|
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
|
|
behavioral improvement is asserted by this placeholder.}
|
|
\label{fig:hq_cases}
|
|
\end{figure}
|
|
|
|
\subsection{Chunk Timing and Cross-Dataset Limits}
|
|
|
|
Table~\ref{tab:chunk} reports a separate global-only
|
|
chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
|
|
at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
|
|
processor calls over 12 sessions. Completed blocks plus observed tail reach
|
|
70.85\%; that older bank is not extraction-matched. Freshness and chunk
|
|
construction vary together, so this is motivation for, not evidence of, an
|
|
isolated PTR benefit or real-time throughput. Table~\ref{tab:stream_context}
|
|
reserves the matched tail $\times$ audio and context-cap tests.
|
|
Tables~\ref{tab:backbone}--
|
|
\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
|
|
cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
|
|
\textbf{Historical LoRA diagnostic.}
|
|
A single-run EasyCom 8-s/2-FPS screen used sparse training anchors (1~Hz,
|
|
plus 5~Hz near transitions) and dense 5-Hz evaluation. Within that legacy
|
|
pipeline, strict mean mAP was 58.17\% for a frozen dense-cache control,
|
|
57.21\% for frozen sparse training, and 56.21\% for a separate three-epoch
|
|
Thinker-LoRA run initialized from a trained frozen head. An epoch-1 LoRA
|
|
weights-only continuation fell to 54.21\% after optimizer/scheduler reset.
|
|
Sparse frozen training is 0.96 points below the dense control; the separate
|
|
LoRA run is another 1.01 below sparse frozen, but the epoch budgets differ.
|
|
Sparse-train/dense-test mismatch and the optimizer reset are possible, not
|
|
isolated, explanations.
|
|
Missing remote raw metrics, matched seeds, and feature equivalence make this
|
|
a negative recipe diagnostic, not a Table~\ref{tab:main} $\G+\A$ comparison
|
|
or proof that LoRA is inherently harmful.
|
|
Ego4D has no archived test metrics in this checkout, so transfer cannot yet
|
|
be claimed. The oracle uses privileged labels, provided boxes omit online
|
|
detection, and aggregate others cannot establish addressee or semantic floor
|
|
ownership.
|
|
|
|
\subsection{Decision Rules and Reproducibility}
|
|
|
|
Matched comparisons require identical cutoffs and targets; archive source
|
|
timestamps, valid-pair counts, seeds, per-session AP, trainable counts, and
|
|
checkpoint/processor revisions. The fixed recipe and selection rules use
|
|
training-side evidence only. Already inspected test sessions make repeat
|
|
seeds exploratory. Raw endpoint AP is not one-event-one-vote ENTRY/YIELD:
|
|
response windows must end at the next other onset, and yield windows at the
|
|
next other offset. Behavioral claims await a real-timeline evaluator audit.
|
|
|
|
\section{Conclusion}
|
|
|
|
On EasyCom, a short-window audio-encoder path complements the frozen Omni
|
|
readout for future wearer activity, most clearly at speech offset. Pooled
|
|
vision recovers little of the timely visible-activity information exposed by
|
|
the analysis-only audit. Partial-tail refresh provides a causal 5-Hz
|
|
interface, but its independent predictive and efficiency effects await
|
|
matched tests. Ego4D transfer and censored event evaluation remain necessary
|
|
before claiming generalizable or improved semantic floor behavior.
|
|
|
|
\section{Acknowledgment}
|
|
OpenAI Codex assisted with drafting and figure layout; the authors remain
|
|
responsible for all experimental claims and final content.
|
|
|
|
\clearpage
|
|
\bibliographystyle{IEEEbib}
|
|
\bibliography{references}
|
|
\end{document}
|