wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reframe targeted manuscript around egocentric local evidence

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 14:34:35 +08:00
parent 3941770748
commit 82cd1b6abb
3 changed files with 310 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/targeted/README.md
+18 -3
View File
@@ -5,8 +5,18 @@ modify or replace `../main/`.
## Positioning ## Positioning
The paper is deliberately framed as a causal global--local **audit**, because The paper now starts from the egocentric conversational agent's when-to-speak
that is the strongest complete story supported by archived evidence: problem, positions it against 27 cited works in turn-taking, first-person
audio-visual conversation, active-speaker modeling, and streaming Omni
interaction, then studies two cutoff-aligned interfaces:
- **short-window encoder evidence (SWE)**: direct recent audio and optional
person-crop visual encoder features alongside the frozen Omni readout;
- **partial-tail refresh (PTR)**: query only the observed prefix of the current
two-second block at each 200-ms cutoff, reusing completed payloads.
The evidence remains a causal global--local **audit**, not a claim that all
Omni models lack local perception or that PTR's independent gain is proven:
- global Omni context and recent audio are complementary across all horizons; - global Omni context and recent audio are complementary across all horizons;
- most measured benefit is dense forecasting and wearer-offset anticipation; - most measured benefit is dense forecasting and wearer-offset anticipation;
@@ -24,6 +34,10 @@ independently audited here.
No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
claim is made. The deployable matrix is explicitly single-seed and exploratory. claim is made. The deployable matrix is explicitly single-seed and exploratory.
EasyCom has archived screening scores. The Ego4D manifest and extraction /
training workflow exist, but this checkout has no archived Ego4D test metrics;
the manuscript specifies its transfer protocol and explicitly leaves any
cross-dataset performance claim pending.
The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an
**illustrative, unmeasured planning aid**, excluded from the page-limited **illustrative, unmeasured planning aid**, excluded from the page-limited
PDF. Its directional statements are hypotheses and falsifiers, not EasyCom PDF. Its directional statements are hypotheses and falsifiers, not EasyCom
@@ -46,7 +60,8 @@ the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column.
The current PDF reserves five numbered result tables: measured deployable The current PDF reserves five numbered result tables: measured deployable
and oracle screens share Table 1; L controls extend its first panel. The and oracle screens share Table 1; L controls extend its first panel. The
unmeasured T/C matrix and a separate measured chunk screen share Table 2; unmeasured T/C matrix and a separate measured chunk screen share Table 2;
B, H, and S occupy Tables 3--5. Pending cells are explicitly unmeasured. B (including an Ego4D transfer slot), H, and S occupy Tables 3--5. Pending
cells are explicitly unmeasured.
Figure 3 is an H/Q case slot, not a selected or measured example. Its Figure 3 is an H/Q case slot, not a selected or measured example. Its
cutoff-marked timelines must be inserted only after the slice protocol and cutoff-marked timelines must be inserted only after the slice protocol and
case-selection rule are frozen. The single-column system figure is generated case-selection rule are frozen. The single-column system figure is generated
manuscript/targeted/main.tex
+147 -132
View File
@@ -17,10 +17,10 @@
\renewcommand{\textfraction}{0.05} \renewcommand{\textfraction}{0.05}
\renewcommand{\floatpagefraction}{0.75} \renewcommand{\floatpagefraction}{0.75}
\renewcommand{\dblfloatpagefraction}{0.75} \renewcommand{\dblfloatpagefraction}{0.75}
\title{Does an Omni Model Know When the Wearer Will Speak?\\ \title{Egocentric Speech Anticipation with Omni Models:\\
A Causal Global--Local Audit of Egocentric Speech Anticipation} Short-Window Evidence and Partial-Tail Refresh}
\name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu, \name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen} Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom} \address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom}
\begin{document} \begin{document}
@@ -28,19 +28,21 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}
\maketitle \maketitle
\begin{abstract} \begin{abstract}
We test whether a frozen Omni readout exposes sub-second evidence for An egocentric conversational agent must anticipate when its invisible wearer
forecasting an invisible camera wearer's speech from causal multiparty will speak while observing a causal multiparty audio--video stream. Although
audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent turn-taking prediction and omni-modal assistants have advanced, the temporal
1-s audio to predict three activity channels at ten 0.2--2.0-s horizons. accessibility of local evidence in an integrated Omni state remains unclear.
On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\% We study two interfaces: a short-window encoder-evidence path that exposes
mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP recent audio and optional person-crop vision to a frozen-Omni forecasting
improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29 head, and partial-tail refresh that queries the observed prefix of an
mean-mAP points. A three-seed causal information audit finds that privileged incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the
past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\%
points; visibility and delayed activity add little. A separate no-box ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled
full-frame screen reaches 82.38/65.54\%, but unmatched training and head vision adds only 0.29 mean-mAP points, whereas an analysis-only causal
capacity prevent visual attribution. Timely evidence access, not semantic visible-activity oracle reveals additional information. A separate chunk
floor understanding, remains the supported conclusion. screen suggests sensitivity to state freshness, but does not isolate the
partial-tail effect. These results motivate finer local evidence access;
they do not establish semantic floor control or cross-dataset transfer.
\end{abstract} \end{abstract}
\begin{keywords} \begin{keywords}
@@ -51,12 +53,16 @@ streaming, voice activity projection, conversational agents
\section{Introduction} \section{Introduction}
\label{sec:intro} \label{sec:intro}
Conversational agents must decide not only \emph{what} to say, but also An egocentric conversational agent must decide \emph{when} to speak while its
\emph{when} to participate. This decision is particularly asymmetric in an camera wearer is usually invisible. It sees other participants and hears a
egocentric stream: the camera wearer is normally invisible, yet the model must mixed conversation, but must predict the wearer's own future speech before a
forecast that wearer's behavior from mixed audio and observations of other turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models
people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known and voice-activity projection (VAP) show the value of anticipatory rather than
turn boundary and excludes media after the decision cutoff. silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap,
elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap,
jiang2026turnbench}. Yet these tasks differ in sensor placement, visible
speakers, dialogue roles, and whether the target is a future activity state
or a semantic intervention.
\begin{figure}[H] \begin{figure}[H]
\centering \centering
@@ -68,47 +74,56 @@ not semantic floor ownership.}
\label{fig:task} \label{fig:task}
\end{figure} \end{figure}
Voice Activity Projection (VAP) casts turn taking as continuous prediction of \textbf{Egocentric audiovisual conversation.}
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to EgoSpeak forecasts wearer speech from first-person RGB and audio using
untrimmed first-person video and defines the EasyCom protocol used here specialized encoders and a temporal fusion model \cite{kim2025egospeak};
\cite{kim2025egospeak,donley2021easycom}. Other work predicts future speakers EasyCom and Ego4D supply complementary egocentric conversation settings
from synchronized egocentric sensors \cite{serj2024threem}, reconstructs \cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker
egocentric conversational graphs \cite{jia2024avconv}, or grounds causal anticipation, egocentric speaking/listening graphs, and addressee detection
multiparty projections in visible face tracks \cite{qi2026muvap}. Those study related but different targets \cite{serj2024threem,jia2024avconv,
systems establish that neither multiparty audiovisual modeling nor kong2024sicnet}. Egocentric active-speaker localization and detection
first-person speech initiation is itself a novelty. Our question is narrower: \cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person
does a general-purpose Omni readout make timely evidence accessible for modeling \cite{wang2024loconet} estimate who is speaking now; they do not by
forecasting the unseen wearer? themselves forecast an unseen wearer's future activity. Visual cues can help
turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled
vision conveys the relevant local floor evidence here is empirical.
Qwen2.5-Omni integrates long-form audio and video in a Thinker representation \textbf{Omni and local evidence.}
\cite{xu2025qwenomni}. Such integration is attractive for conversational Omni systems integrate modalities and streaming interaction
context, but a blockwise state can blur or delay the short acoustic and visual \cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
cues needed at 200-ms intervals. We therefore separate an integrated global yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side
state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat audio--visual adaptation offers another route to accessible modality evidence
their value as an empirical question rather than assuming that a larger \cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither
backbone solves temporal access. large-context integration nor a nominally streaming interface guarantees a
fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni
extraction, completed 2-s blocks and an incomplete current block have
different update rules. We therefore test two distinct interfaces:
\emph{short-window encoder evidence} (SWE) exposes recent modality features
outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR)
queries only the observed current-block prefix. Neither term implies that
all Omni models use the same chunk duration or lack local perception.
We contribute a cutoff-aligned global--local audit, a bounded Omni stream We contribute a causal implementation of SWE and PTR for wearer activity
with observed-tail refresh and recent audio, and a separation of deployable anticipation; an EasyCom audit separating global, local-audio, pooled-visual,
features from an analysis-only visible-activity oracle. Generic pooled or and privileged information conditions; and a protocol for EasyCom and Ego4D
current-frame person features recover little of the available activity signal; transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D
timely activity estimation is the next modeling target. scores are still pending, so our measured claims remain confined to the
available EasyCom screens.
\begin{figure}[t] \begin{figure}[t]
\centering \centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Deployable global--local path and bounded stream. At each 200-ms \caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
cutoff, the head fuses the frozen integrated Omni state with explicit recent audio and optional pooled person-crop vision to a frozen-Omni head. PTR
audio and optional pooled person-crop vision. Completed 2-s payloads are temporarily queries the observed current-block prefix while reusing completed
reused; only the observed partial tail is temporarily recomputed. Privileged 2-s payloads. Privileged past-activity labels are not model inputs.}
past-activity audit labels are kept outside this graph and are not inputs.}
\label{fig:overview} \label{fig:overview}
\end{figure} \end{figure}
\section{Method} \section{Method}
\label{sec:method} \label{sec:method}
\subsection{Task and Causal Inputs} \subsection{Task and Short-Window Encoder Evidence}
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
@@ -116,17 +131,16 @@ horizon has three channels in fixed order: background, camera wearer, and
aggregate others. Wearer and others may overlap; no output identifies an aggregate others. Wearer and others may overlap; no output identifies an
individual speaker, addressee, or semantic end of turn. individual speaker, addressee, or semantic end of turn.
The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B. We The global feature $\vect g_n$ concatenates the final four frozen
concatenate the final four Thinker readout states and project them to 2048 Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
dimensions. The readout contains both audio and video context; $\G$ is not an already contains audio and video context; $\G$ is not audio-free. SWE
audio-free condition. The local audio feature $\vect a_n$ is obtained by adds encoder-side features without requiring the Thinker state to preserve
cropping the most recent one-second mixed waveform \emph{before} their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed
preprocessing, encoding it with the frozen Qwen audio encoder, and averaging waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and
valid output tokens. An optional visual feature $\vect v_n$ averages spatial averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens
tokens inside provided person boxes, people within each causal frame, and inside provided person boxes, then across people and valid causal frames in
valid frames from the most recent second. It preserves neither identity nor the latest second. This pooling has no identity or mouth-motion trajectory;
mouth-motion trajectories, and provided boxes do not constitute an online provided boxes do not imply an online detector.
detector.
Each enabled local feature is layer-normalized, projected to 256 dimensions, Each enabled local feature is layer-normalized, projected to 256 dimensions,
and paired with a validity bit. The trainable head is and paired with a validity bit. The trainable head is
@@ -143,15 +157,17 @@ head with the EgoSpeak multi-positive cross-entropy over causal-valid
source--horizon pairs. The Omni and modality encoders remain frozen, so this source--horizon pairs. The Omni and modality encoders remain frozen, so this
is frozen-backbone adaptation rather than end-to-end training. is frozen-backbone adaptation rather than end-to-end training.
\subsection{Cutoff-Aligned Streaming} \subsection{Causal Partial-Tail Refresh}
Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
presentation cap. After warm-up, three completed blocks and the observed (Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
current tail are available; span varies with block phase. Completed observed current tail are available; span varies with block phase. Reusing
modality payloads are reused. At each cutoff, the tail is cropped at $t_n$ completed modality payloads, we crop the tail at $t_n$ and query it in a
and queried in a temporary cache, without committing it. Only completed temporary cache. The tail and query are not committed. Only completed
blocks are committed; session or source discontinuities reset caches. This blocks enter persistent history; session and source discontinuities reset
refreshes the logical state at 5~Hz without backfilling earlier rows. caches. Thus each 5-Hz readout sees the observed prefix without future
backfill. PTR is a causal access rule, not by itself a proven accuracy or
real-time latency improvement.
\subsection{Visible-Activity Information Audit} \subsection{Visible-Activity Information Audit}
@@ -255,8 +271,8 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\
\end{table*} \end{table*}
\begin{table}[t] \begin{table}[t]
\caption{B result slots: paired within-backbone local-audio gains (unmeasured). \caption{Pending paired $\G\rightarrow\G+\A$ transfer (unmeasured).
MiniCPM requires a matched readout/refresh/history audit before comparison.} EC is EasyCom; MiniCPM first needs readout/refresh/history audits.}
\label{tab:backbone} \label{tab:backbone}
\centering\normalsize \centering\normalsize
\setlength{\tabcolsep}{3pt} \setlength{\tabcolsep}{3pt}
@@ -264,9 +280,10 @@ MiniCPM requires a matched readout/refresh/history audit before comparison.}
\toprule \toprule
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\ Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
\midrule \midrule
Qwen Omni 3B & -- & -- & -- \\ EC: Omni 3B & -- & -- & -- \\
Qwen Omni 7B & -- & -- & -- \\ EC: Omni 7B & -- & -- & -- \\
MiniCPM-o 4.5 & -- & -- & -- \\ EC: MiniCPM-o 4.5 & -- & -- & -- \\
Ego4D: Omni 3B & -- & -- & -- \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
@@ -320,6 +337,15 @@ single-seed screen; the information audit uses seeds 0, 1, and 2 with an
identical fixed recipe. We therefore report standard deviations only for the identical fixed recipe. We therefore report standard deviations only for the
latter and treat small differences in the former descriptively. latter and treat small differences in the former descriptively.
\textbf{Ego4D transfer protocol.}
Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak
manifest and targets, with 346 training and 87 test clips. The same 5-Hz
cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and
$\G+\A+\V$ comparisons; visual validity is retained rather than silently
dropping no-box rows. The repository contains the extraction and training
workflow but no archived Ego4D score or matched PTR control. Hence this
protocol is specified, not reported as an observed cross-dataset gain.
\textbf{Metrics.} \textbf{Metrics.}
At each horizon we compute AP for the three activity channels over common At each horizon we compute AP for the three activity channels over common
causal-valid rows, then average classes to obtain mAP and average all ten causal-valid rows, then average classes to obtain mAP and average all ten
@@ -345,25 +371,22 @@ transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
They share the official split and physical cutoffs; the 8-s T/C row reuses They share the official split and physical cutoffs; the 8-s T/C row reuses
one run. The source retains unmeasured hypotheses and falsifiers. one run. The source retains unmeasured hypotheses and falsifiers.
\section{Results} \section{Results and Analysis}
\label{sec:results} \label{sec:results}
\subsection{Global Context and Recent Audio Are Complementary} \subsection{EasyCom: Local Encoder Evidence}
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at
slightly stronger when mAP is averaged to 2.0~s. Their fusion is better than 0.2~s while integrated $\G$ is slightly stronger across all horizons.
$\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over
0.2~s and 4.47 at 2.0~s. It also improves every class when averaged over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
horizons (+8.72 background, +2.43 wearer, and +2.65 others). Relative to 0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by points, onset AP by only 1.47: dense gains do not imply equal improvement
3.92 points. The endpoint effect is asymmetric: offset AP rises by 7.23 at both activity boundaries.
points, while onset AP rises by 1.47. Dense forecasting gains therefore do
not imply equal progress at both behavior boundaries.
The global-only head has more trainable parameters and a longer context than The $\G$-only head has more trainable parameters and longer history than the
the local-only head, so this comparison demonstrates predictive $\A$-only head. Thus this screen shows predictive complementarity, not a
complementarity, not that semantic reasoning alone causes the gain. A wider capacity-matched proof that the Thinker lacks a specific local cue.
history-matched control is required for a representation-level causal claim.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong % Retained as a planning aid; illustrative, unmeasured patterns do not belong
% in the page-limited submission preview. % in the page-limited submission preview.
@@ -429,7 +452,7 @@ not real-time deployment. \\
\end{table*} \end{table*}
\fi \fi
\subsection{Generic Vision Does Not Recover the Available Signal} \subsection{EasyCom: Visual Evidence and Ablations}
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when +0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
@@ -470,53 +493,45 @@ behavioral improvement is asserted by this placeholder.}
\label{fig:hq_cases} \label{fig:hq_cases}
\end{figure} \end{figure}
\subsection{Pending Controls and Limits} \subsection{Chunk Timing and Cross-Dataset Limits}
Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots Table~\ref{tab:stream_context}, panel B, reports a separate global-only
for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
Table~\ref{tab:stream_context} separately reports a global-only chunk screen: at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
the 200-ms commits beat 2-s hold after startup exclusion but require about processor calls over 12 sessions. Completed blocks plus observed tail reach
ten times as many processor calls across 12 sessions. Freshness and chunk 70.85\%; that older bank is not extraction-matched. Freshness and chunk
construction both change, and the older tail bank is not extraction-matched; construction vary together, so this is motivation for, not evidence of, an
these values do not fill the pending T/C factorial or establish real-time isolated PTR benefit or real-time throughput. Panel A reserves the matched
throughput. L requires a tail $\times$ audio and context-cap tests. Tables~\ref{tab:backbone}--
trained width-matched global control and past-delayed features on identical \ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
valid pairs; T crosses tail refresh with local audio; C re-extracts each cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
causal context cap and pairs it with a global-only run. B compares the Ego4D has no archived test metrics in this checkout, so transfer cannot yet
within-backbone audio increment only after readout and history audits. be claimed. The oracle uses privileged labels, provided boxes omit online
H/Q reports slice counts and cutoff-marked examples; S measures full-path detection, and aggregate others cannot establish addressee or semantic floor
latency and memory. The oracle uses privileged labels, provided boxes omit ownership.
online detection, and aggregate others cannot establish addressee or
semantic floor ownership.
\subsection{Decision Rules and Reproducibility} \subsection{Decision Rules and Reproducibility}
The unit of comparison is the same physical cutoff and future target time, Comparisons require identical physical cutoffs and future targets. New runs
not an arbitrary feature row. For every new condition we will archive must archive source timestamps, common valid-pair counts, seeds, per-session
input-source timestamps, common valid-pair counts, seeds, per-session AP, AP, trainable counts, revisions, checkpoints, and predictions; correlated
trainable parameter counts, processor/model revisions, checkpoint hashes, frames are not independent replicates. The matched matrix keeps a fixed
and prediction scores. Frames and horizons are correlated, not independent four-epoch recipe, with selection rules frozen on training-side evidence.
replicates. Sessions~1--3 have already been inspected, so repeat seeds remain exploratory.
Raw endpoint AP is not one-event-one-vote ENTRY/YIELD: response windows must
We retain the fixed four-epoch recipe for the matched matrix. Any loss be censored at the next other onset and yield windows at the next other
weight, threshold, context selection, or case-selection rule is frozen offset. Until that evaluator passes a real-timeline audit, we report dense
using training-side evidence before looking at Sessions~1--3. Because forecasting and endpoint anticipation only.
these test sessions have already informed this research program, even
new seed runs remain exploratory rather than restoring an unseen test set.
The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral
hypothesis, but an ENTRY/YIELD error-rate claim additionally requires
one-event-one-vote attribution with response windows censored at the next
other onset and yield windows at the next other offset. Until that
competing-risk evaluator passes real-timeline audit, we report only dense
forecasting and endpoint anticipation.
\section{Conclusion} \section{Conclusion}
In this strictly causal audit, frozen Omni context and recent audio improve On EasyCom, a short-window audio-encoder path complements the frozen Omni
wearer activity forecasting, especially future speech cessation. Generic readout for future wearer activity, most clearly at speech offset. Pooled
pooled vision does not recover the timely visible-activity signal exposed by vision recovers little of the timely visible-activity information exposed by
the analysis-only oracle. Matched controls and censored event evaluation the analysis-only audit. Partial-tail refresh provides a causal 5-Hz
remain necessary before claiming improved floor behavior. interface, but its independent predictive and efficiency effects await
matched tests. Ego4D transfer and censored event evaluation remain necessary
before claiming generalizable or improved semantic floor behavior.
\section{Acknowledgment} \section{Acknowledgment}
OpenAI Codex assisted with drafting and figure layout; the authors remain OpenAI Codex assisted with drafting and figure layout; the authors remain
manuscript/targeted/references.bib
+145
View File
@@ -61,3 +61,148 @@
year = {2025}, year = {2025},
doi = {10.1109/ICASSP49660.2025.10888166} doi = {10.1109/ICASSP49660.2025.10888166}
} }
@inproceedings{russell2025mmvap,
author = {Sam O'Connor Russell and Naomi Harte},
title = {Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction},
booktitle = {Findings of ACL},
pages = {209--221},
year = {2025},
doi = {10.18653/v1/2025.findings-acl.12}
}
@inproceedings{elmers2025triadic,
author = {Mikey Elmers and Koji Inoue and Divesh Lala and Tatsuya Kawahara},
title = {Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems},
booktitle = {Proc. Interspeech},
year = {2025},
url = {https://www.isca-archive.org/interspeech_2025/elmers25_interspeech.html}
}
@inproceedings{lin2025backchannel,
author = {Yuxin Lin and Yinglin Zheng and Ming Zeng and Wangzheng Shi},
title = {Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals},
booktitle = {Proc. ACL},
pages = {15310--15322},
year = {2025},
doi = {10.18653/v1/2025.acl-long.743}
}
@inproceedings{lee2024eot,
author = {Meng-Chen Lee and Zhigang Deng},
title = {Online Multimodal End-of-Turn Prediction for Three-party Conversations},
booktitle = {Proc. International Conference on Multimodal Interaction},
pages = {57--65},
year = {2024},
doi = {10.1145/3678957.3685742}
}
@article{jiang2026turnbench,
author = {Freeman Jiang and others},
title = {{TurnBench}: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue},
journal = {arXiv preprint arXiv:2608.25218},
year = {2026}
}
@inproceedings{grauman2022ego4d,
author = {Kristen Grauman and Andrew Westbury and Eugene Byrne and others},
title = {{Ego4D}: Around the World in 3,000 Hours of Egocentric Video},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages = {18995--19012},
year = {2022}
}
@inproceedings{kong2024sicnet,
author = {Deqian Kong and Furqan Khan and Xu Zhang and Prateek Singhal and Ying Nian Wu},
title = {Long-Term Social Interaction Context: The Key to Egocentric Addressee Detection},
booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2024}
}
@inproceedings{jiang2022asl,
author = {Hao Jiang and Calvin Murdock and Vamsi Krishna Ithapu},
title = {Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2022}
}
@inproceedings{clarke2025asd,
author = {Jason Clarke and Yoshihiko Gotoh and Stefan Goetze},
title = {Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings},
booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2025},
url = {https://arxiv.org/abs/2502.06012}
}
@inproceedings{wang2024loconet,
author = {Xizi Wang and Feng Cheng and Gedas Bertasius and David Crandall},
title = {{LoCoNet}: Long-Short Context Network for Active Speaker Detection},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2024}
}
@inproceedings{oconnorrussell2025noise,
author = {Sam O'Connor Russell and Naomi Harte},
title = {Visual Cues Support Robust Turn-taking Prediction in Noise},
booktitle = {Proc. Interspeech},
year = {2025},
url = {https://www.isca-archive.org/interspeech_2025/oconnorrussell25_interspeech.html}
}
@article{cui2026minicpmo45,
author = {Junbo Cui and Bokai Xu and Chongyi Wang and others},
title = {{MiniCPM-o 4.5}: Towards Real-Time Full-Duplex Omni-Modal Interaction},
journal = {arXiv preprint arXiv:2604.27393},
year = {2026}
}
@article{tian2026roma,
author = {Xueyun Tian and Wei Li and Bingbing Xu and Heng Dong and Yuanzhuo Wang and Huawei Shen},
title = {{ROMA}: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding},
journal = {arXiv preprint arXiv:2601.10323},
year = {2026}
}
@article{yan2026duplug,
author = {Ruiqi Yan and Wenxi Chen and Zhanxun Liu and others},
title = {{SoulX-Duplug}: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation},
journal = {arXiv preprint arXiv:2603.14877},
year = {2026}
}
@article{yu2025salmonnomni,
author = {Wenyi Yu and Siyin Wang and Xiaoyu Yang and others},
title = {{SALMONN-omni}: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation},
journal = {arXiv preprint arXiv:2505.17060},
year = {2025}
}
@inproceedings{chen2024videollmonline,
author = {Joya Chen and others},
title = {{VideoLLM-online}: Online Video Large Language Model for Streaming Video},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2024}
}
@inproceedings{wang2025omnimmi,
author = {Yuxuan Wang and others},
title = {{OmniMMI}: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2025}
}
@inproceedings{lin2023lavish,
author = {Yan-Bo Lin and Yi-Lin Sung and Jie Lei and Mohit Bansal and Gedas Bertasius},
title = {Vision Transformers Are Parameter-Efficient Audio-Visual Learners},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages = {2299--2309},
year = {2023}
}
@inproceedings{rouditchenko2024whisperflamingo,
author = {Andrew Rouditchenko and Yuan Gong and Samuel Thomas and Leonid Karlinsky and Hilde Kuehne and Rogerio Feris and James Glass},
title = {{Whisper-Flamingo}: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation},
booktitle = {Proc. Interspeech},
pages = {2420--2424},
year = {2024}
}