529 lines
24 KiB
TeX
529 lines
24 KiB
TeX
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
|
|
% Uses the repository's ICASSP 2027 author kit.
|
|
% Four technical pages plus an optional reference-only page.
|
|
\documentclass{article}
|
|
\usepackage{spconf}
|
|
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
|
|
\hypersetup{hidelinks}
|
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
|
\newcommand{\G}{\mathrm{G}}
|
|
\newcommand{\A}{\mathrm{A}}
|
|
\newcommand{\V}{\mathrm{V}}
|
|
\setlength{\emergencystretch}{2em}
|
|
\setcounter{topnumber}{5}
|
|
\setcounter{dbltopnumber}{5}
|
|
\renewcommand{\topfraction}{0.95}
|
|
\renewcommand{\dbltopfraction}{0.95}
|
|
\renewcommand{\textfraction}{0.05}
|
|
\renewcommand{\floatpagefraction}{0.75}
|
|
\renewcommand{\dblfloatpagefraction}{0.75}
|
|
\title{Does an Omni Model Know When the Wearer Will Speak?\\
|
|
A Causal Global--Local Audit of Egocentric Speech Anticipation}
|
|
\name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,
|
|
Xiaoju Dai, Li Zhang, and Guorun Chen}
|
|
\address{School of Computer Science, Example University, Shanghai, China}
|
|
|
|
\begin{document}
|
|
\ninept
|
|
\maketitle
|
|
|
|
\begin{abstract}
|
|
We test whether a frozen Omni readout exposes sub-second evidence for
|
|
forecasting an invisible camera wearer's speech from causal multiparty
|
|
audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent
|
|
1-s audio to predict three activity channels at ten 0.2--2.0-s horizons.
|
|
On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\%
|
|
mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP
|
|
improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29
|
|
mean-mAP points. A three-seed causal information audit finds that privileged
|
|
past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP
|
|
points; visibility and delayed activity add little. A separate no-box
|
|
full-frame screen reaches 82.38/65.54\%, but unmatched training and head
|
|
capacity prevent visual attribution. Timely evidence access, not semantic
|
|
floor understanding, remains the supported conclusion.
|
|
\end{abstract}
|
|
|
|
\begin{keywords}
|
|
egocentric speech anticipation, multimodal foundation models, causal
|
|
streaming, voice activity projection, conversational agents
|
|
\end{keywords}
|
|
|
|
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
Conversational agents must decide not only \emph{what} to say, but also
|
|
\emph{when} to participate. This decision is particularly asymmetric in an
|
|
egocentric stream: the camera wearer is normally invisible, yet the model must
|
|
forecast that wearer's behavior from mixed audio and observations of other
|
|
people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known
|
|
turn boundary and excludes media after the decision cutoff.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
|
|
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
|
|
only the past audio--video prefix is input; ten future activity states are
|
|
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
|
|
not semantic floor ownership.}
|
|
\label{fig:task}
|
|
\end{figure}
|
|
|
|
Voice Activity Projection (VAP) casts turn taking as continuous prediction of
|
|
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to
|
|
untrimmed first-person video and defines the EasyCom protocol used here
|
|
\cite{kim2025egospeak,donley2021easycom}. Other work predicts future speakers
|
|
from synchronized egocentric sensors \cite{serj2024threem}, reconstructs
|
|
egocentric conversational graphs \cite{jia2024avconv}, or grounds causal
|
|
multiparty projections in visible face tracks \cite{qi2026muvap}. Those
|
|
systems establish that neither multiparty audiovisual modeling nor
|
|
first-person speech initiation is itself a novelty. Our question is narrower:
|
|
does a general-purpose Omni readout make timely evidence accessible for
|
|
forecasting the unseen wearer?
|
|
|
|
Qwen2.5-Omni integrates long-form audio and video in a Thinker representation
|
|
\cite{xu2025qwenomni}. Such integration is attractive for conversational
|
|
context, but a blockwise state can blur or delay the short acoustic and visual
|
|
cues needed at 200-ms intervals. We therefore separate an integrated global
|
|
state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
|
|
their value as an empirical question rather than assuming that a larger
|
|
backbone solves temporal access.
|
|
|
|
We contribute a cutoff-aligned global--local audit, a bounded Omni stream
|
|
with observed-tail refresh and recent audio, and a separation of deployable
|
|
features from an analysis-only visible-activity oracle. Generic pooled or
|
|
current-frame person features recover little of the available activity signal;
|
|
timely activity estimation is the next modeling target.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
|
\caption{Deployable global--local path and bounded stream. At each 200-ms
|
|
cutoff, the head fuses the frozen integrated Omni state with explicit recent
|
|
audio and optional pooled person-crop vision. Completed 2-s payloads are
|
|
reused; only the observed partial tail is temporarily recomputed. Privileged
|
|
past-activity audit labels are kept outside this graph and are not inputs.}
|
|
\label{fig:overview}
|
|
\end{figure}
|
|
|
|
\section{Method}
|
|
\label{sec:method}
|
|
|
|
\subsection{Task and Causal Inputs}
|
|
|
|
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
|
|
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
|
|
horizon has three channels in fixed order: background, camera wearer, and
|
|
aggregate others. Wearer and others may overlap; no output identifies an
|
|
individual speaker, addressee, or semantic end of turn.
|
|
|
|
The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B. We
|
|
concatenate the final four Thinker readout states and project them to 2048
|
|
dimensions. The readout contains both audio and video context; $\G$ is not an
|
|
audio-free condition. The local audio feature $\vect a_n$ is obtained by
|
|
cropping the most recent one-second mixed waveform \emph{before}
|
|
preprocessing, encoding it with the frozen Qwen audio encoder, and averaging
|
|
valid output tokens. An optional visual feature $\vect v_n$ averages spatial
|
|
tokens inside provided person boxes, people within each causal frame, and
|
|
valid frames from the most recent second. It preserves neither identity nor
|
|
mouth-motion trajectories, and provided boxes do not constitute an online
|
|
detector.
|
|
|
|
Each enabled local feature is layer-normalized, projected to 256 dimensions,
|
|
and paired with a validity bit. The trainable head is
|
|
\begin{equation}
|
|
\widehat{\vect Y}_{n}=
|
|
\operatorname{softmax}_{c}F_{\theta}
|
|
\bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a;
|
|
\widetilde{\vect v}_n;m_n^v\bigr],
|
|
\label{eq:fusion}
|
|
\end{equation}
|
|
where omitted modalities remove their terms and $F_\theta$ is a
|
|
layer-normalized MLP with a 30-logit output. We train only the projections and
|
|
head with the EgoSpeak multi-positive cross-entropy over causal-valid
|
|
source--horizon pairs. The Omni and modality encoders remain frozen, so this
|
|
is frozen-backbone adaptation rather than end-to-end training.
|
|
|
|
\subsection{Cutoff-Aligned Streaming}
|
|
|
|
Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s
|
|
presentation cap. After warm-up, three completed blocks and the observed
|
|
current tail are available; span varies with block phase. Completed
|
|
modality payloads are reused. At each cutoff, the tail is cropped at $t_n$
|
|
and queried in a temporary cache, without committing it. Only completed
|
|
blocks are committed; session or source discontinuities reset caches. This
|
|
refreshes the logical state at 5~Hz without backfilling earlier rows.
|
|
|
|
\subsection{Visible-Activity Information Audit}
|
|
|
|
To distinguish missing information from ineffective learned vision, we build
|
|
analysis-only features from transcript-derived \emph{past} activity. At each
|
|
cutoff, visible non-wearer participants receive two seconds of binary history
|
|
sampled on the same 5-Hz grid. The preparation fails if its aggregate
|
|
activity disagrees with the released target. We compare: (i) visibility with
|
|
activity zeroed, (ii) per-person histories, (iii) identities shuffled while
|
|
preserving active-person counts at every lag, (iv) histories delayed by one
|
|
second, and (v) one OR-pooled aggregate-visible history. These conditions
|
|
never expose future activity, but their speech labels make them privileged
|
|
diagnostics, not deployable features or upper bounds.
|
|
|
|
\begin{table*}[t]
|
|
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
|
|
three-seed privileged past-activity audit, which is not deployable. Mean is
|
|
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
|
|
F3 is a no-box, two-frame causal full-frame screen; it and the historical
|
|
deployable rows are exploratory seed-0 runs with unmatched head/training
|
|
details. F1 has three seeds. Dashes are unavailable, not zero.}
|
|
\label{tab:main}
|
|
\centering
|
|
\normalsize
|
|
\renewcommand{\arraystretch}{0.87}
|
|
\setlength{\tabcolsep}{5.0pt}
|
|
\begin{tabular}{@{}lccccc@{}}
|
|
\toprule
|
|
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
|
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
|
\midrule
|
|
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
|
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
|
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
|
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
|
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
|
|
& 65.400 & 26.019 & 45.371 \\\\
|
|
No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\
|
|
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
|
& $64.951{\pm}0.095$ & -- & -- \\\\
|
|
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
|
|
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
|
|
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\par\smallskip
|
|
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
|
|
\normalsize
|
|
\setlength{\tabcolsep}{3.2pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
|
\midrule
|
|
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
|
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
|
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
|
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
|
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
|
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
|
& $\mathbf{66.951{\pm}.022}$ \\\\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\begin{table*}[t]
|
|
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
|
|
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
|
|
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
|
|
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
|
|
\label{tab:stream_context}
|
|
\centering\normalsize
|
|
\renewcommand{\arraystretch}{0.87}
|
|
\setlength{\tabcolsep}{4pt}
|
|
\begin{tabular}{@{}llccccc@{}}
|
|
\toprule
|
|
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
|
|
& $\Delta$ On & $\Delta$ Off \\
|
|
\midrule
|
|
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
|
|
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
|
|
\midrule
|
|
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
|
|
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\par\smallskip
|
|
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
|
|
\midrule
|
|
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
|
|
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
|
|
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
|
|
\begin{table}[t]
|
|
\caption{B result slots: paired within-backbone local-audio gains (unmeasured).
|
|
MiniCPM requires a matched readout/refresh/history audit before comparison.}
|
|
\label{tab:backbone}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{3pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
|
|
\midrule
|
|
Qwen Omni 3B & -- & -- & -- \\
|
|
Qwen Omni 7B & -- & -- & -- \\
|
|
MiniCPM-o 4.5 & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{H result slots: predeclared activity slices (unmeasured). Report
|
|
positive/negative counts and per-session scores; AP is undefined for a
|
|
pure-negative slice, which instead uses calibrated FPR.}
|
|
\label{tab:hard_slices}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{2.5pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Slice & $N_+/N_-$ & AP & FPR \\
|
|
\midrule
|
|
Wearer onset & -- & -- & -- \\
|
|
Wearer offset & -- & -- & -- \\
|
|
Other pause (negative) & -- & n/a & -- \\
|
|
Overlap (negative) & -- & n/a & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{table}[t]
|
|
\caption{S result slots: full-path cost on the same hardware (unmeasured).
|
|
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
|
|
\label{tab:efficiency}
|
|
\centering\normalsize
|
|
\setlength{\tabcolsep}{3pt}
|
|
\begin{tabular}{@{}lccc@{}}
|
|
\toprule
|
|
Stage & P50 ms & P95 ms & Peak GB \\
|
|
\midrule
|
|
Completed block & -- & -- & -- \\
|
|
Observed tail & -- & -- & -- \\
|
|
Local path + head & -- & -- & -- \\
|
|
Full path & -- & -- & -- \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\section{Experiments}
|
|
\label{sec:experiments}
|
|
|
|
\textbf{Data and protocol.}
|
|
We use EasyCom with the official EgoSpeak split: Sessions~4--12 for training
|
|
and Sessions~1--3 for testing. Inputs and targets share a 5-Hz grid. Heads
|
|
are trained for four fixed epochs on all training sessions without test-time
|
|
checkpoint selection. The deployable feature matrix is an archived
|
|
single-seed screen; the information audit uses seeds 0, 1, and 2 with an
|
|
identical fixed recipe. We therefore report standard deviations only for the
|
|
latter and treat small differences in the former descriptively.
|
|
|
|
\textbf{Metrics.}
|
|
At each horizon we compute AP for the three activity channels over common
|
|
causal-valid rows, then average classes to obtain mAP and average all ten
|
|
horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint
|
|
diagnostics, onset AP conditions source rows on the wearer being silent and
|
|
scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer
|
|
speech and scores future silence with $1-p(\mathrm{wearer})$. These
|
|
threshold-free quantities average ten horizons. They are not one-event-one-
|
|
vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
|
|
|
|
\textbf{Comparisons.}
|
|
We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni
|
|
$\G$; $\G+\A$; and $\G+\A+$pooled-$\V$. A three-seed current-frame
|
|
frame-set attention screen tests whether a richer person aggregation is
|
|
already sufficient. Because that screen used Session~12 model selection
|
|
whereas the historical pooled row did not, their difference is diagnostic,
|
|
not a matched causal attribution.
|
|
|
|
\textbf{Six pre-specified control families.}
|
|
We freeze six pending control families: (L) capacity and recency, (T) tail
|
|
refresh crossed with local audio, (C) six context caps, (B) paired backbone
|
|
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
|
|
They share the official split and physical cutoffs; the 8-s T/C row reuses
|
|
one run. The source retains unmeasured hypotheses and falsifiers.
|
|
|
|
\section{Results}
|
|
\label{sec:results}
|
|
|
|
\subsection{Global Context and Recent Audio Are Complementary}
|
|
|
|
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
|
|
slightly stronger when mAP is averaged to 2.0~s. Their fusion is better than
|
|
$\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at
|
|
0.2~s and 4.47 at 2.0~s. It also improves every class when averaged over
|
|
horizons (+8.72 background, +2.43 wearer, and +2.65 others). Relative to
|
|
released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by
|
|
3.92 points. The endpoint effect is asymmetric: offset AP rises by 7.23
|
|
points, while onset AP rises by 1.47. Dense forecasting gains therefore do
|
|
not imply equal progress at both behavior boundaries.
|
|
|
|
The global-only head has more trainable parameters and a longer context than
|
|
the local-only head, so this comparison demonstrates predictive
|
|
complementarity, not that semantic reasoning alone causes the gain. A wider
|
|
history-matched control is required for a representation-level causal claim.
|
|
|
|
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
|
|
% in the page-limited submission preview.
|
|
\iffalse
|
|
\begin{table*}[t]
|
|
\caption{Six-group \textbf{illustrative scenario, not measured results}.
|
|
These are conditional, pre-run interpretations for the pending matrix, not
|
|
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
|
|
cutoffs; signs and shape must be replaced by measured values even if contrary
|
|
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
|
|
\label{tab:scenario}
|
|
\centering
|
|
\normalsize
|
|
\renewcommand{\arraystretch}{1.12}
|
|
\setlength{\tabcolsep}{4.2pt}
|
|
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
|
|
\toprule
|
|
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
|
|
\midrule
|
|
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
|
|
past-delayed local cues; current-state persistence
|
|
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
|
|
near-horizon benefit; persistence does not recover the gain
|
|
& Width-only or persistence matches fusion, or delay does not hurt:
|
|
do not attribute gain to timely distinct evidence. \\
|
|
\addlinespace[2.5pt]
|
|
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
|
|
crossed with $\A$ on/off; causal replay and phase audit
|
|
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
|
|
future-only media never changes prior readouts
|
|
& No independent local gain under tail refresh implies redundancy;
|
|
replay/leakage failure invalidates timing claims. \\
|
|
\addlinespace[2.5pt]
|
|
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
|
|
add $\G$-only at each cap for a window-stability claim
|
|
& Fused performance plateaus around mid-range caps while
|
|
$\G$-only remains more cap-sensitive
|
|
& If long history substitutes for $\A$, local-evidence claim narrows;
|
|
without C2, no robustness comparison is justified. \\
|
|
\addlinespace[2.5pt]
|
|
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
|
|
interface audit, MiniCPM-o 4.5
|
|
& Positive within-backbone paired $\Delta$ on all admissible rows;
|
|
absolute backbone ranking is not inferred
|
|
& Failed causal/history match is unsupported, not zero gain;
|
|
one negative paired row defeats universal transfer. \\
|
|
\addlinespace[2.5pt]
|
|
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
|
|
same-horizon, cutoff-marked case selection
|
|
& Offset gains coexist with hard wearer-onset negatives; examples
|
|
explain, rather than replace, counted slice statistics
|
|
& Pure-negative slice AP is undefined; selected examples cannot
|
|
establish reduced false-entry rate. \\
|
|
\addlinespace[2.5pt]
|
|
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
|
|
encoding, tail recomputation, local path, and boundaries
|
|
& A feasible throughput--quality tradeoff can be plotted
|
|
against 200-ms output cadence
|
|
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
|
|
not real-time deployment. \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table*}
|
|
\fi
|
|
|
|
\subsection{Generic Vision Does Not Recover the Available Signal}
|
|
|
|
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
|
|
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
|
|
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
|
|
current-frame person-set model is stable across seeds but reaches lower mean
|
|
mAP than the historical pooled row under a non-matched selection protocol.
|
|
These results do not support a claim of learned speaker grounding. A separate
|
|
seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\%
|
|
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
|
|
causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a
|
|
shared GRU, while head capacity and training batch differ; raw boundary
|
|
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
|
|
difference does not isolate visual motion or floor behavior.
|
|
|
|
Table~\ref{tab:main}, panel B, locates the missing information. Aggregate visible
|
|
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
|
|
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
|
|
one-second-delayed activity by +0.164. Thus availability or extra parameters
|
|
cannot explain the oracle gain. Per-person activity exceeds shuffled identity
|
|
by only 0.200 mean-mAP points and is itself 0.291 below the aggregate
|
|
condition. The current target therefore supports timely visible-floor
|
|
activity as useful information, but not the necessity of identity continuity.
|
|
The majority of the oracle's 0.2-s macro gain comes from background and
|
|
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
|
|
dense mAP gain establishes fewer false wearer entries.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
|
|
\centering
|
|
\textbf{Q1 case slots: examples pending}\\[2pt]
|
|
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
|
|
Show source cutoff, observed activity and same-horizon wearer scores.
|
|
\end{minipage}}
|
|
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
|
|
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
|
|
behavioral improvement is asserted by this placeholder.}
|
|
\label{fig:hq_cases}
|
|
\end{figure}
|
|
|
|
\subsection{Pending Controls and Limits}
|
|
|
|
Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots
|
|
for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of
|
|
Table~\ref{tab:stream_context} separately reports a global-only chunk screen:
|
|
the 200-ms commits beat 2-s hold after startup exclusion but require about
|
|
ten times as many processor calls across 12 sessions. Freshness and chunk
|
|
construction both change, and the older tail bank is not extraction-matched;
|
|
these values do not fill the pending T/C factorial or establish real-time
|
|
throughput. L requires a
|
|
trained width-matched global control and past-delayed features on identical
|
|
valid pairs; T crosses tail refresh with local audio; C re-extracts each
|
|
causal context cap and pairs it with a global-only run. B compares the
|
|
within-backbone audio increment only after readout and history audits.
|
|
H/Q reports slice counts and cutoff-marked examples; S measures full-path
|
|
latency and memory. The oracle uses privileged labels, provided boxes omit
|
|
online detection, and aggregate others cannot establish addressee or
|
|
semantic floor ownership.
|
|
|
|
\subsection{Decision Rules and Reproducibility}
|
|
|
|
The unit of comparison is the same physical cutoff and future target time,
|
|
not an arbitrary feature row. For every new condition we will archive
|
|
input-source timestamps, common valid-pair counts, seeds, per-session AP,
|
|
trainable parameter counts, processor/model revisions, checkpoint hashes,
|
|
and prediction scores. Frames and horizons are correlated, not independent
|
|
replicates.
|
|
|
|
We retain the fixed four-epoch recipe for the matched matrix. Any loss
|
|
weight, threshold, context selection, or case-selection rule is frozen
|
|
using training-side evidence before looking at Sessions~1--3. Because
|
|
these test sessions have already informed this research program, even
|
|
new seed runs remain exploratory rather than restoring an unseen test set.
|
|
The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral
|
|
hypothesis, but an ENTRY/YIELD error-rate claim additionally requires
|
|
one-event-one-vote attribution with response windows censored at the next
|
|
other onset and yield windows at the next other offset. Until that
|
|
competing-risk evaluator passes real-timeline audit, we report only dense
|
|
forecasting and endpoint anticipation.
|
|
|
|
\section{Conclusion}
|
|
|
|
In this strictly causal audit, frozen Omni context and recent audio improve
|
|
wearer activity forecasting, especially future speech cessation. Generic
|
|
pooled vision does not recover the timely visible-activity signal exposed by
|
|
the analysis-only oracle. Matched controls and censored event evaluation
|
|
remain necessary before claiming improved floor behavior.
|
|
|
|
\section{Acknowledgment}
|
|
OpenAI Codex assisted with drafting and figure layout; the authors remain
|
|
responsible for all experimental claims and final content.
|
|
|
|
\clearpage
|
|
\bibliographystyle{IEEEbib}
|
|
\bibliography{references}
|
|
\end{document}
|