Update targeted ICASSP 2027 manuscript and results
This commit is contained in:
Xin Wang
+338
-136
@@ -1,42 +1,46 @@
|
||||
% Acceptance-target ICASSP 2027 draft grounded only in archived measurements.
|
||||
% The official 2027 author kit should replace the 2026-compatible style assets
|
||||
% before submission. Four technical pages plus an optional reference-only page.
|
||||
% Uses the repository's ICASSP 2027 author kit.
|
||||
% Four technical pages plus an optional reference-only page.
|
||||
\documentclass{article}
|
||||
\usepackage{spconf}
|
||||
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor}
|
||||
\usepackage{amsmath,amssymb,graphicx,booktabs,multirow,hyperref,xcolor,float}
|
||||
\hypersetup{hidelinks}
|
||||
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
||||
\newcommand{\G}{\mathrm{G}}
|
||||
\newcommand{\A}{\mathrm{A}}
|
||||
\newcommand{\V}{\mathrm{V}}
|
||||
\setlength{\emergencystretch}{2em}
|
||||
\setcounter{topnumber}{5}
|
||||
\setcounter{dbltopnumber}{5}
|
||||
\renewcommand{\topfraction}{0.95}
|
||||
\renewcommand{\dbltopfraction}{0.95}
|
||||
\renewcommand{\textfraction}{0.05}
|
||||
\renewcommand{\floatpagefraction}{0.75}
|
||||
\renewcommand{\dblfloatpagefraction}{0.75}
|
||||
\title{Does an Omni Model Know When the Wearer Will Speak?\\
|
||||
A Causal Global--Local Audit of Egocentric Speech Anticipation}
|
||||
\name{Authors withheld in this working draft}
|
||||
\address{}
|
||||
\name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,
|
||||
Xiaoju Dai, Li Zhang, and Guorun Chen}
|
||||
\address{School of Computer Science, Example University, Shanghai, China}
|
||||
|
||||
\begin{document}
|
||||
\ninept
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
An egocentric conversational agent must anticipate the future speech of an
|
||||
invisible camera wearer from a causal multiparty audio--video stream. We ask
|
||||
whether a frozen general-purpose Omni state exposes the sub-second evidence
|
||||
needed by this task. Our cutoff-aligned model combines an 8-s Qwen2.5-Omni
|
||||
context with the most recent 1-s audio and predicts three activity channels at
|
||||
ten horizons from 0.2 to 2.0~s. In a single-seed EasyCom screen, this model reaches
|
||||
81.72\% mAP at 0.2~s and 65.11\% averaged over horizons, compared with
|
||||
74.20\% and 61.19\% for released EgoSpeak. The gain is larger for wearer
|
||||
offset anticipation (+7.23 points) than onset anticipation (+1.47). Pooled
|
||||
visual features add only 0.29 mean-mAP points. A separate three-seed,
|
||||
strictly causal information audit explains this gap: privileged past activity
|
||||
of visible participants adds 4.83 points at 0.2~s and 1.77 mean-mAP points,
|
||||
whereas visibility, one-second-delayed activity, and identity shuffling show
|
||||
that timeliness matters more than identity continuity. The results isolate an
|
||||
evidence-access bottleneck: recent acoustic evidence is already useful, while
|
||||
current generic visual pooling does not recover the available visible-floor
|
||||
signal.
|
||||
We test whether a frozen Omni readout exposes sub-second evidence for
|
||||
forecasting an invisible camera wearer's speech from causal multiparty
|
||||
audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent
|
||||
1-s audio to predict three activity channels at ten 0.2--2.0-s horizons.
|
||||
On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\%
|
||||
mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP
|
||||
improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29
|
||||
mean-mAP points. A three-seed causal information audit finds that privileged
|
||||
past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP
|
||||
points; visibility and delayed activity add little. A separate no-box
|
||||
full-frame screen reaches 82.38/65.54\%, but unmatched training and head
|
||||
capacity prevent visual attribution. Timely evidence access, not semantic
|
||||
floor understanding, remains the supported conclusion.
|
||||
\end{abstract}
|
||||
|
||||
\begin{keywords}
|
||||
@@ -51,8 +55,18 @@ Conversational agents must decide not only \emph{what} to say, but also
|
||||
\emph{when} to participate. This decision is particularly asymmetric in an
|
||||
egocentric stream: the camera wearer is normally invisible, yet the model must
|
||||
forecast that wearer's behavior from mixed audio and observations of other
|
||||
people. A useful predictor must update before a turn boundary is known and
|
||||
must not use media from after its decision cutoff.
|
||||
people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known
|
||||
turn boundary and excludes media after the decision cutoff.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
|
||||
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
|
||||
only the past audio--video prefix is input; ten future activity states are
|
||||
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
|
||||
not semantic floor ownership.}
|
||||
\label{fig:task}
|
||||
\end{figure}
|
||||
|
||||
Voice Activity Projection (VAP) casts turn taking as continuous prediction of
|
||||
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to
|
||||
@@ -74,28 +88,22 @@ state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
|
||||
their value as an empirical question rather than assuming that a larger
|
||||
backbone solves temporal access.
|
||||
|
||||
This paper makes three contributions. First, we formulate a strictly causal
|
||||
global--local audit for invisible-wearer activity forecasting, with a common
|
||||
5-Hz cutoff and ten future horizons. Second, we realize the audit with a
|
||||
bounded Omni stream that combines completed history with a temporary observed
|
||||
tail and a separate one-second audio path. Third, we distinguish deployable
|
||||
feature gains from an analysis-only visible-activity oracle. This reveals a
|
||||
specific negative result---generic pooled or current-frame person features do
|
||||
not recover most of the available local-floor information---and identifies
|
||||
timely activity estimation, rather than identity continuity, as the next
|
||||
modeling target.
|
||||
We contribute a cutoff-aligned global--local audit, a bounded Omni stream
|
||||
with observed-tail refresh and recent audio, and a separation of deployable
|
||||
features from an analysis-only visible-activity oracle. Generic pooled or
|
||||
current-frame person features recover little of the available activity signal;
|
||||
timely activity estimation is the next modeling target.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=0.97\textwidth]{figures/system_overview.pdf}
|
||||
\caption{Global--local causal audit. At each 200-ms cutoff, the prediction
|
||||
path fuses an integrated Omni state with explicit recent audio and optional
|
||||
person-crop vision. Completed 2-s blocks are reused while only the observed
|
||||
partial tail is temporarily recomputed. The purple branch is a privileged
|
||||
past-activity audit used to test information value; it is never a deployable
|
||||
input and contains no future activity.}
|
||||
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
||||
\caption{Deployable global--local path and bounded stream. At each 200-ms
|
||||
cutoff, the head fuses the frozen integrated Omni state with explicit recent
|
||||
audio and optional pooled person-crop vision. Completed 2-s payloads are
|
||||
reused; only the observed partial tail is temporarily recomputed. Privileged
|
||||
past-activity audit labels are kept outside this graph and are not inputs.}
|
||||
\label{fig:overview}
|
||||
\end{figure*}
|
||||
\end{figure}
|
||||
|
||||
\section{Method}
|
||||
\label{sec:method}
|
||||
@@ -137,16 +145,13 @@ is frozen-backbone adaptation rather than end-to-end training.
|
||||
|
||||
\subsection{Cutoff-Aligned Streaming}
|
||||
|
||||
The global stream uses source-aligned 2-s blocks and an 8-s maximum
|
||||
presentation budget. After warm-up, three completed blocks and the observed
|
||||
part of the current block are available; the exact span varies with block
|
||||
phase. Completed modality payloads are reused when the bounded language-model
|
||||
cache is rebuilt. At every cutoff, the incomplete current block is cropped at
|
||||
$t_n$, appended to a temporary cache, and queried for $\vect g_n$. Neither
|
||||
the temporary tail nor the query is committed. At a block boundary the
|
||||
completed payload is committed once and a new tail begins. Sessions and
|
||||
source discontinuities reset all caches. Thus the logical state is refreshed
|
||||
at 5~Hz without backfilling a completed block into earlier rows.
|
||||
Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s
|
||||
presentation cap. After warm-up, three completed blocks and the observed
|
||||
current tail are available; span varies with block phase. Completed
|
||||
modality payloads are reused. At each cutoff, the tail is cropped at $t_n$
|
||||
and queried in a temporary cache, without committing it. Only completed
|
||||
blocks are committed; session or source discontinuities reset caches. This
|
||||
refreshes the logical state at 5~Hz without backfilling earlier rows.
|
||||
|
||||
\subsection{Visible-Activity Information Audit}
|
||||
|
||||
@@ -161,6 +166,148 @@ second, and (v) one OR-pooled aggregate-visible history. These conditions
|
||||
never expose future activity, but their speech labels make them privileged
|
||||
diagnostics, not deployable features or upper bounds.
|
||||
|
||||
\begin{table*}[t]
|
||||
\caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
|
||||
three-seed privileged past-activity audit, which is not deployable. Mean is
|
||||
ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
|
||||
F3 is a no-box, two-frame causal full-frame screen; it and the historical
|
||||
deployable rows are exploratory seed-0 runs with unmatched head/training
|
||||
details. F1 has three seeds. Dashes are unavailable, not zero.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\normalsize
|
||||
\renewcommand{\arraystretch}{0.87}
|
||||
\setlength{\tabcolsep}{5.0pt}
|
||||
\begin{tabular}{@{}lccccc@{}}
|
||||
\toprule
|
||||
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
||||
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
||||
\midrule
|
||||
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
||||
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
||||
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
||||
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
||||
Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
|
||||
& 65.400 & 26.019 & 45.371 \\\\
|
||||
No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\
|
||||
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
||||
& $64.951{\pm}0.095$ & -- & -- \\\\
|
||||
Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
|
||||
Past-delayed $\A$ (L3) & -- & -- & -- & -- & -- \\\\
|
||||
Current-state persistence (L4) & -- & -- & -- & -- & -- \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\par\smallskip
|
||||
\emph{Panel B: privileged past-activity audit (mean $\pm$ sample SD).}\par\smallskip
|
||||
\normalsize
|
||||
\setlength{\tabcolsep}{3.2pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
||||
\midrule
|
||||
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
||||
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
||||
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
||||
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
||||
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
||||
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
||||
& $\mathbf{66.951{\pm}.022}$ \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\begin{table*}[t]
|
||||
\caption{Chunk and context audit. Panel A is an unmeasured matched T/C
|
||||
matrix: each row pairs $\G$ and $\G+\A$ on common valid cutoffs. Panel B is
|
||||
a separate exploratory seed-0 global-only screen (10 epochs, 8-s cap,
|
||||
2-FPS video); its rows are not entries in Panel A. Dashes are pending.}
|
||||
\label{tab:stream_context}
|
||||
\centering\normalsize
|
||||
\renewcommand{\arraystretch}{0.87}
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}llccccc@{}}
|
||||
\toprule
|
||||
Group & Refresh / cap & $\G$ mAP & $\G+\A$ mAP & $\Delta$ mAP
|
||||
& $\Delta$ On & $\Delta$ Off \\
|
||||
\midrule
|
||||
T & Completed-only, 8 s & -- & -- & -- & -- & -- \\
|
||||
T & Observed-tail, 8 s & -- & -- & -- & -- & -- \\
|
||||
\midrule
|
||||
C & Tail-aware, 2 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 4 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 6 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 8 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 10 s & -- & -- & -- & -- & -- \\
|
||||
C & Tail-aware, 12 s & -- & -- & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\par\smallskip
|
||||
\emph{Panel B: reported 200-ms chunk screen; dense mAP (\%).}\par\smallskip
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Global feature construction & 0.2-s mAP & Mean mAP & Processor calls \\
|
||||
\midrule
|
||||
Native 2-s completed hold & 59.62 & 55.19 & 9,539 \\
|
||||
Non-native 200-ms commits & 67.05 & 57.84 & 95,453 \\
|
||||
Completed blocks + observed tail & 70.85 & 59.17 & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{B result slots: paired within-backbone local-audio gains (unmeasured).
|
||||
MiniCPM requires a matched readout/refresh/history audit before comparison.}
|
||||
\label{tab:backbone}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{3pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
|
||||
\midrule
|
||||
Qwen Omni 3B & -- & -- & -- \\
|
||||
Qwen Omni 7B & -- & -- & -- \\
|
||||
MiniCPM-o 4.5 & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{H result slots: predeclared activity slices (unmeasured). Report
|
||||
positive/negative counts and per-session scores; AP is undefined for a
|
||||
pure-negative slice, which instead uses calibrated FPR.}
|
||||
\label{tab:hard_slices}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{2.5pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Slice & $N_+/N_-$ & AP & FPR \\
|
||||
\midrule
|
||||
Wearer onset & -- & -- & -- \\
|
||||
Wearer offset & -- & -- & -- \\
|
||||
Other pause (negative) & -- & n/a & -- \\
|
||||
Overlap (negative) & -- & n/a & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{S result slots: full-path cost on the same hardware (unmeasured).
|
||||
Include I/O and block transitions; logical 5 Hz is not measured throughput.}
|
||||
\label{tab:efficiency}
|
||||
\centering\normalsize
|
||||
\setlength{\tabcolsep}{3pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Stage & P50 ms & P95 ms & Peak GB \\
|
||||
\midrule
|
||||
Completed block & -- & -- & -- \\
|
||||
Observed tail & -- & -- & -- \\
|
||||
Local path + head & -- & -- & -- \\
|
||||
Full path & -- & -- & -- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Experiments}
|
||||
\label{sec:experiments}
|
||||
|
||||
@@ -191,36 +338,16 @@ already sufficient. Because that screen used Session~12 model selection
|
||||
whereas the historical pooled row did not, their difference is diagnostic,
|
||||
not a matched causal attribution.
|
||||
|
||||
\textbf{Six pre-specified control families.}
|
||||
We freeze six pending control families: (L) capacity and recency, (T) tail
|
||||
refresh crossed with local audio, (C) six context caps, (B) paired backbone
|
||||
transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
|
||||
They share the official split and physical cutoffs; the 8-s T/C row reuses
|
||||
one run. The source retains unmeasured hypotheses and falsifiers.
|
||||
|
||||
\section{Results}
|
||||
\label{sec:results}
|
||||
|
||||
\begin{table*}[t]
|
||||
\caption{EasyCom Sessions~1--3 results (\%). $\mathrm{mAP}_{.2}$ and
|
||||
$\mathrm{WAP}_{.2}$ are dense 0.2-s metrics; Mean averages dense mAP over ten
|
||||
horizons. On/Off are raw wearer endpoint AP, not semantic event scores.
|
||||
Deployable rows are archived single-seed screens; $\pm$ is shown only for the
|
||||
three-seed F1 screen. Dashes denote unavailable measurements.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\scriptsize
|
||||
\setlength{\tabcolsep}{5.0pt}
|
||||
\begin{tabular}{@{}lccccc@{}}
|
||||
\toprule
|
||||
Method & $\mathrm{mAP}_{.2}\uparrow$ & $\mathrm{WAP}_{.2}\uparrow$
|
||||
& Mean$\uparrow$ & On$\uparrow$ & Off$\uparrow$ \\\\
|
||||
\midrule
|
||||
EgoSpeak MiniROADA A+V & 74.204 & 63.750 & 61.193 & 24.491 & 38.024 \\\\
|
||||
Recent Qwen audio $\A$ & 79.692 & 84.100 & 60.508 & -- & -- \\\\
|
||||
Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
|
||||
Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
|
||||
Omni $\G+\A+$pooled-$\V$ & \textbf{82.027} & \textbf{85.086}
|
||||
& \textbf{65.400} & \textbf{26.019} & 45.371 \\\\
|
||||
Current-frame person set F1 & $81.777{\pm}0.156$ & --
|
||||
& $64.951{\pm}0.095$ & -- & -- \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
|
||||
\subsection{Global Context and Recent Audio Are Complementary}
|
||||
|
||||
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
|
||||
@@ -238,6 +365,70 @@ the local-only head, so this comparison demonstrates predictive
|
||||
complementarity, not that semantic reasoning alone causes the gain. A wider
|
||||
history-matched control is required for a representation-level causal claim.
|
||||
|
||||
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
|
||||
% in the page-limited submission preview.
|
||||
\iffalse
|
||||
\begin{table*}[t]
|
||||
\caption{Six-group \textbf{illustrative scenario, not measured results}.
|
||||
These are conditional, pre-run interpretations for the pending matrix, not
|
||||
EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid
|
||||
cutoffs; signs and shape must be replaced by measured values even if contrary
|
||||
to the story. L/T/C/B require seeds 0--2 and per-session reporting.}
|
||||
\label{tab:scenario}
|
||||
\centering
|
||||
\normalsize
|
||||
\renewcommand{\arraystretch}{1.12}
|
||||
\setlength{\tabcolsep}{4.2pt}
|
||||
\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}}
|
||||
\toprule
|
||||
Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\
|
||||
\midrule
|
||||
L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$;
|
||||
past-delayed local cues; current-state persistence
|
||||
& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses
|
||||
near-horizon benefit; persistence does not recover the gain
|
||||
& Width-only or persistence matches fusion, or delay does not hurt:
|
||||
do not attribute gain to timely distinct evidence. \\
|
||||
\addlinespace[2.5pt]
|
||||
T1--2 & 2-s completed-only vs 200-ms observed-tail refresh,
|
||||
crossed with $\A$ on/off; causal replay and phase audit
|
||||
& Tail helps most without $\A$; local $\A$ still adds at each refresh;
|
||||
future-only media never changes prior readouts
|
||||
& No independent local gain under tail refresh implies redundancy;
|
||||
replay/leakage failure invalidates timing claims. \\
|
||||
\addlinespace[2.5pt]
|
||||
C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$;
|
||||
add $\G$-only at each cap for a window-stability claim
|
||||
& Fused performance plateaus around mid-range caps while
|
||||
$\G$-only remains more cap-sensitive
|
||||
& If long history substitutes for $\A$, local-evidence claim narrows;
|
||||
without C2, no robustness comparison is justified. \\
|
||||
\addlinespace[2.5pt]
|
||||
B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal
|
||||
interface audit, MiniCPM-o 4.5
|
||||
& Positive within-backbone paired $\Delta$ on all admissible rows;
|
||||
absolute backbone ranking is not inferred
|
||||
& Failed causal/history match is unsupported, not zero gain;
|
||||
one negative paired row defeats universal transfer. \\
|
||||
\addlinespace[2.5pt]
|
||||
H1/Q1 & Predefined onset/offset and other-pause/overlap slices;
|
||||
same-horizon, cutoff-marked case selection
|
||||
& Offset gains coexist with hard wearer-onset negatives; examples
|
||||
explain, rather than replace, counted slice statistics
|
||||
& Pure-negative slice AP is undefined; selected examples cannot
|
||||
establish reduced false-entry rate. \\
|
||||
\addlinespace[2.5pt]
|
||||
S1 & End-to-end p50/p95 latency and peak memory, including I/O,
|
||||
encoding, tail recomputation, local path, and boundaries
|
||||
& A feasible throughput--quality tradeoff can be plotted
|
||||
against 200-ms output cadence
|
||||
& If p95 exceeds cadence, describe logical 5-Hz prediction only,
|
||||
not real-time deployment. \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table*}
|
||||
\fi
|
||||
|
||||
\subsection{Generic Vision Does Not Recover the Available Signal}
|
||||
|
||||
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
|
||||
@@ -245,32 +436,15 @@ Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
|
||||
averaged over horizons; onset/offset changes are +0.059/+0.120. A richer
|
||||
current-frame person-set model is stable across seeds but reaches lower mean
|
||||
mAP than the historical pooled row under a non-matched selection protocol.
|
||||
These results do not support a claim of learned speaker grounding.
|
||||
These results do not support a claim of learned speaker grounding. A separate
|
||||
seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\%
|
||||
mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
|
||||
causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a
|
||||
shared GRU, while head capacity and training batch differ; raw boundary
|
||||
metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
|
||||
difference does not isolate visual motion or floor behavior.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Three-seed privileged-information audit (mean $\pm$ sample standard
|
||||
deviation, \%). Every row includes $\G+\A$; activity conditions use only
|
||||
past labels. Bold marks the best diagnostic, not a deployable model.}
|
||||
\label{tab:oracle}
|
||||
\centering
|
||||
\scriptsize
|
||||
\setlength{\tabcolsep}{3.2pt}
|
||||
\begin{tabular}{@{}lccc@{}}
|
||||
\toprule
|
||||
Extra condition & $\mathrm{mAP}_{.2}$ & $\mathrm{WAP}_{.2}$ & Mean \\\\
|
||||
\midrule
|
||||
None (audio baseline) & $81.869{\pm}.049$ & $85.264{\pm}.110$ & $65.177{\pm}.058$ \\\\
|
||||
Visibility only & $81.945{\pm}.098$ & $85.200{\pm}.144$ & $65.217{\pm}.048$ \\\\
|
||||
Activity delayed 1 s & $81.951{\pm}.086$ & $85.270{\pm}.137$ & $65.341{\pm}.051$ \\\\
|
||||
Shuffled person IDs & $85.917{\pm}.085$ & $85.577{\pm}.246$ & $66.460{\pm}.090$ \\\\
|
||||
Per-person activity & $86.266{\pm}.182$ & $85.608{\pm}.220$ & $66.660{\pm}.135$ \\\\
|
||||
Aggregate visible activity & $\mathbf{86.696{\pm}.085}$ & $\mathbf{85.836{\pm}.147}$
|
||||
& $\mathbf{66.951{\pm}.022}$ \\\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:oracle} locates the missing information. Aggregate visible
|
||||
Table~\ref{tab:main}, panel B, locates the missing information. Aggregate visible
|
||||
activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
|
||||
mean-mAP points. Visibility alone changes mean mAP by only +0.040, and
|
||||
one-second-delayed activity by +0.164. Thus availability or extra parameters
|
||||
@@ -282,45 +456,73 @@ The majority of the oracle's 0.2-s macro gain comes from background and
|
||||
others; wearer AP rises by only 0.572. This is why neither the oracle nor a
|
||||
dense mAP gain establishes fewer false wearer entries.
|
||||
|
||||
\subsection{What the Audit Establishes}
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth}
|
||||
\centering
|
||||
\textbf{Q1 case slots: examples pending}\\[2pt]
|
||||
Onset / other-pause hard negative \quad|\quad Offset / overlap\\
|
||||
Show source cutoff, observed activity and same-horizon wearer scores.
|
||||
\end{minipage}}
|
||||
\caption{Reserved H/Q case figure. Select examples by a frozen rule after
|
||||
the slice counts in Table~\ref{tab:hard_slices} are available. No case or
|
||||
behavioral improvement is asserted by this placeholder.}
|
||||
\label{fig:hq_cases}
|
||||
\end{figure}
|
||||
|
||||
Three conclusions survive the controls. First, a general-purpose global
|
||||
state and a one-second acoustic readout provide different useful timescales.
|
||||
Second, generic visual pooling has not converted visible people into reliable
|
||||
local floor evidence. Third, perfect timely aggregate activity contains
|
||||
recoverable information, whereas identity contributes little under the
|
||||
current target. A dedicated causal active-speaker front end---designed for
|
||||
mouth motion, blur, and short audio--visual synchrony as in egocentric ASD
|
||||
\cite{huh2025egoasd}---is consequently a better-motivated next step than a
|
||||
larger identity-aware fusion module.
|
||||
\subsection{Pending Controls and Limits}
|
||||
|
||||
The evidence remains bounded. EasyCom has only three official test sessions;
|
||||
the deployable matrix is single seed and has been examined repeatedly. The
|
||||
oracle uses labels and is not a performance upper bound. Provided boxes omit
|
||||
online detection cost, and logical 5-Hz output is not a measured throughput
|
||||
claim. Finally, aggregate others cannot supply addressee or semantic
|
||||
interruption ground truth. Confirmatory behavioral claims require a frozen
|
||||
training-side operating point, competing-risk event attribution, and
|
||||
multi-seed matched controls.
|
||||
Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots
|
||||
for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of
|
||||
Table~\ref{tab:stream_context} separately reports a global-only chunk screen:
|
||||
the 200-ms commits beat 2-s hold after startup exclusion but require about
|
||||
ten times as many processor calls across 12 sessions. Freshness and chunk
|
||||
construction both change, and the older tail bank is not extraction-matched;
|
||||
these values do not fill the pending T/C factorial or establish real-time
|
||||
throughput. L requires a
|
||||
trained width-matched global control and past-delayed features on identical
|
||||
valid pairs; T crosses tail refresh with local audio; C re-extracts each
|
||||
causal context cap and pairs it with a global-only run. B compares the
|
||||
within-backbone audio increment only after readout and history audits.
|
||||
H/Q reports slice counts and cutoff-marked examples; S measures full-path
|
||||
latency and memory. The oracle uses privileged labels, provided boxes omit
|
||||
online detection, and aggregate others cannot establish addressee or
|
||||
semantic floor ownership.
|
||||
|
||||
\subsection{Decision Rules and Reproducibility}
|
||||
|
||||
The unit of comparison is the same physical cutoff and future target time,
|
||||
not an arbitrary feature row. For every new condition we will archive
|
||||
input-source timestamps, common valid-pair counts, seeds, per-session AP,
|
||||
trainable parameter counts, processor/model revisions, checkpoint hashes,
|
||||
and prediction scores. Frames and horizons are correlated, not independent
|
||||
replicates.
|
||||
|
||||
We retain the fixed four-epoch recipe for the matched matrix. Any loss
|
||||
weight, threshold, context selection, or case-selection rule is frozen
|
||||
using training-side evidence before looking at Sessions~1--3. Because
|
||||
these test sessions have already informed this research program, even
|
||||
new seed runs remain exploratory rather than restoring an unseen test set.
|
||||
The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral
|
||||
hypothesis, but an ENTRY/YIELD error-rate claim additionally requires
|
||||
one-event-one-vote attribution with response windows censored at the next
|
||||
other onset and yield windows at the next other offset. Until that
|
||||
competing-risk evaluator passes real-timeline audit, we report only dense
|
||||
forecasting and endpoint anticipation.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
We presented a strictly causal global--local audit of egocentric wearer speech
|
||||
anticipation. Frozen Omni context and recent audio are strongly complementary,
|
||||
especially for future wearer cessation. In contrast, pooled or current-frame
|
||||
person features do not recover the local-floor signal revealed by a causal
|
||||
visible-activity oracle. The actionable conclusion is deliberately narrow:
|
||||
the next bottleneck is timely, learnable local activity evidence, not simply
|
||||
more multimodal capacity or persistent identity. This distinction provides a
|
||||
clearer target for future egocentric conversational agents and prevents dense
|
||||
forecasting gains from being mistaken for semantic floor control.
|
||||
In this strictly causal audit, frozen Omni context and recent audio improve
|
||||
wearer activity forecasting, especially future speech cessation. Generic
|
||||
pooled vision does not recover the timely visible-activity signal exposed by
|
||||
the analysis-only oracle. Matched controls and censored event evaluation
|
||||
remain necessary before claiming improved floor behavior.
|
||||
|
||||
\section{Acknowledgment}
|
||||
OpenAI Codex assisted with substantive drafting and figure layout in the
|
||||
abstract, introduction, and presentation of the archived analysis; the
|
||||
authors remain responsible for verification, experimental claims, and the
|
||||
final manuscript.
|
||||
OpenAI Codex assisted with drafting and figure layout; the authors remain
|
||||
responsible for all experimental claims and final content.
|
||||
|
||||
\clearpage
|
||||
\bibliographystyle{IEEEbib}
|
||||
\bibliography{references}
|
||||
\end{document}
|
||||
|
||||
Reference in New Issue
Block a user