wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reframe targeted manuscript around egocentric local evidence

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 14:34:35 +08:00
parent 3941770748
commit 82cd1b6abb
3 changed files with 310 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/targeted/README.md
+18 -3
View File
@@ -5,8 +5,18 @@ modify or replace `../main/`.
## Positioning
The paper is deliberately framed as a causal global--local **audit**, because
that is the strongest complete story supported by archived evidence:
The paper now starts from the egocentric conversational agent's when-to-speak
problem, positions it against 27 cited works in turn-taking, first-person
audio-visual conversation, active-speaker modeling, and streaming Omni
interaction, then studies two cutoff-aligned interfaces:
- **short-window encoder evidence (SWE)**: direct recent audio and optional
person-crop visual encoder features alongside the frozen Omni readout;
- **partial-tail refresh (PTR)**: query only the observed prefix of the current
two-second block at each 200-ms cutoff, reusing completed payloads.
The evidence remains a causal global--local **audit**, not a claim that all
Omni models lack local perception or that PTR's independent gain is proven:
- global Omni context and recent audio are complementary across all horizons;
- most measured benefit is dense forecasting and wearer-offset anticipation;
@@ -24,6 +34,10 @@ independently audited here.
No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput
claim is made. The deployable matrix is explicitly single-seed and exploratory.
EasyCom has archived screening scores. The Ego4D manifest and extraction /
training workflow exist, but this checkout has no archived Ego4D test metrics;
the manuscript specifies its transfer protocol and explicitly leaves any
cross-dataset performance claim pending.
The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an
**illustrative, unmeasured planning aid**, excluded from the page-limited
PDF. Its directional statements are hypotheses and falsifiers, not EasyCom
@@ -46,7 +60,8 @@ the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column.
The current PDF reserves five numbered result tables: measured deployable
and oracle screens share Table 1; L controls extend its first panel. The
unmeasured T/C matrix and a separate measured chunk screen share Table 2;
B, H, and S occupy Tables 3--5. Pending cells are explicitly unmeasured.
B (including an Ego4D transfer slot), H, and S occupy Tables 3--5. Pending
cells are explicitly unmeasured.
Figure 3 is an H/Q case slot, not a selected or measured example. Its
cutoff-marked timelines must be inserted only after the slice protocol and
case-selection rule are frozen. The single-column system figure is generated
manuscript/targeted/main.tex
+147 -132
View File
@@ -17,10 +17,10 @@
\renewcommand{\textfraction}{0.05}
\renewcommand{\floatpagefraction}{0.75}
\renewcommand{\dblfloatpagefraction}{0.75}
\title{Does an Omni Model Know When the Wearer Will Speak?\\
A Causal Global--Local Audit of Egocentric Speech Anticipation}
\name{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}
\title{Egocentric Speech Anticipation with Omni Models:\\
Short-Window Evidence and Partial-Tail Refresh}
\name{\shortstack{Xin Wang, Dingtian Yan, Shengxiang Hu, Changdong Liu,\\
Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}}
\address{Shanghai Ideal Information Industry (Group) Co., Ltd., China Telecom}
\begin{document}
@@ -28,19 +28,21 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}
\maketitle
\begin{abstract}
We test whether a frozen Omni readout exposes sub-second evidence for
forecasting an invisible camera wearer's speech from causal multiparty
audio--video. Our 5-Hz model combines 8-s Qwen2.5-Omni context and recent
1-s audio to predict three activity channels at ten 0.2--2.0-s horizons.
On EasyCom, a seed-0 screen reaches 81.72\% mAP at 0.2~s and 65.11\%
mean mAP, versus 74.20/61.19\% for released EgoSpeak. Wearer offset AP
improves by 7.23 points, onset AP by 1.47. Pooled vision adds only 0.29
mean-mAP points. A three-seed causal information audit finds that privileged
past visible-person activity adds 4.83 points at 0.2~s and 1.77 mean-mAP
points; visibility and delayed activity add little. A separate no-box
full-frame screen reaches 82.38/65.54\%, but unmatched training and head
capacity prevent visual attribution. Timely evidence access, not semantic
floor understanding, remains the supported conclusion.
An egocentric conversational agent must anticipate when its invisible wearer
will speak while observing a causal multiparty audio--video stream. Although
turn-taking prediction and omni-modal assistants have advanced, the temporal
accessibility of local evidence in an integrated Omni state remains unclear.
We study two interfaces: a short-window encoder-evidence path that exposes
recent audio and optional person-crop vision to a frozen-Omni forecasting
head, and partial-tail refresh that queries the observed prefix of an
incomplete audiovisual block at each 200-ms cutoff. On EasyCom, the
exploratory global-plus-audio screen obtains 81.72\% 0.2-s mAP and 65.11\%
ten-horizon mean mAP, versus 74.20/61.19\% for released EgoSpeak. Pooled
vision adds only 0.29 mean-mAP points, whereas an analysis-only causal
visible-activity oracle reveals additional information. A separate chunk
screen suggests sensitivity to state freshness, but does not isolate the
partial-tail effect. These results motivate finer local evidence access;
they do not establish semantic floor control or cross-dataset transfer.
\end{abstract}
\begin{keywords}
@@ -51,12 +53,16 @@ streaming, voice activity projection, conversational agents
\section{Introduction}
\label{sec:intro}
Conversational agents must decide not only \emph{what} to say, but also
\emph{when} to participate. This decision is particularly asymmetric in an
egocentric stream: the camera wearer is normally invisible, yet the model must
forecast that wearer's behavior from mixed audio and observations of other
people. As Fig.~\ref{fig:task} illustrates, prediction precedes any known
turn boundary and excludes media after the decision cutoff.
An egocentric conversational agent must decide \emph{when} to speak while its
camera wearer is usually invisible. It sees other participants and hears a
mixed conversation, but must predict the wearer's own future speech before a
turn boundary is known (Fig.~\ref{fig:task}). Continuous turn-taking models
and voice-activity projection (VAP) show the value of anticipatory rather than
silence-triggered decisions \cite{ekstedt2022vap,russell2025mmvap,
elmers2025triadic,lin2025backchannel,lee2024eot,qi2026muvap,
jiang2026turnbench}. Yet these tasks differ in sensor placement, visible
speakers, dialogue roles, and whether the target is a future activity state
or a semantic intervention.
\begin{figure}[H]
\centering
@@ -68,47 +74,56 @@ not semantic floor ownership.}
\label{fig:task}
\end{figure}
Voice Activity Projection (VAP) casts turn taking as continuous prediction of
future activity \cite{ekstedt2022vap}. EgoSpeak brings this objective to
untrimmed first-person video and defines the EasyCom protocol used here
\cite{kim2025egospeak,donley2021easycom}. Other work predicts future speakers
from synchronized egocentric sensors \cite{serj2024threem}, reconstructs
egocentric conversational graphs \cite{jia2024avconv}, or grounds causal
multiparty projections in visible face tracks \cite{qi2026muvap}. Those
systems establish that neither multiparty audiovisual modeling nor
first-person speech initiation is itself a novelty. Our question is narrower:
does a general-purpose Omni readout make timely evidence accessible for
forecasting the unseen wearer?
\textbf{Egocentric audiovisual conversation.}
EgoSpeak forecasts wearer speech from first-person RGB and audio using
specialized encoders and a temporal fusion model \cite{kim2025egospeak};
EasyCom and Ego4D supply complementary egocentric conversation settings
\cite{donley2021easycom,grauman2022ego4d}. Multi-view next-speaker
anticipation, egocentric speaking/listening graphs, and addressee detection
study related but different targets \cite{serj2024threem,jia2024avconv,
kong2024sicnet}. Egocentric active-speaker localization and detection
\cite{jiang2022asl,huh2025egoasd,clarke2025asd} and long--short person
modeling \cite{wang2024loconet} estimate who is speaking now; they do not by
themselves forecast an unseen wearer's future activity. Visual cues can help
turn prediction in noise \cite{oconnorrussell2025noise}, but whether pooled
vision conveys the relevant local floor evidence here is empirical.
Qwen2.5-Omni integrates long-form audio and video in a Thinker representation
\cite{xu2025qwenomni}. Such integration is attractive for conversational
context, but a blockwise state can blur or delay the short acoustic and visual
cues needed at 200-ms intervals. We therefore separate an integrated global
state $\G$ from explicit recent evidence $\A$ and optional $\V$, and treat
their value as an empirical question rather than assuming that a larger
backbone solves temporal access.
\textbf{Omni and local evidence.}
Omni systems integrate modalities and streaming interaction
\cite{xu2025qwenomni,cui2026minicpmo45,tian2026roma,yan2026duplug,
yu2025salmonnomni,chen2024videollmonline,wang2025omnimmi}. Encoder-side
audio--visual adaptation offers another route to accessible modality evidence
\cite{lin2023lavish,rouditchenko2024whisperflamingo}. However, neither
large-context integration nor a nominally streaming interface guarantees a
fresh representation at every 200-ms prediction cutoff. In our Qwen2.5-Omni
extraction, completed 2-s blocks and an incomplete current block have
different update rules. We therefore test two distinct interfaces:
\emph{short-window encoder evidence} (SWE) exposes recent modality features
outside the integrated Thinker state, and \emph{partial-tail refresh} (PTR)
queries only the observed current-block prefix. Neither term implies that
all Omni models use the same chunk duration or lack local perception.
We contribute a cutoff-aligned global--local audit, a bounded Omni stream
with observed-tail refresh and recent audio, and a separation of deployable
features from an analysis-only visible-activity oracle. Generic pooled or
current-frame person features recover little of the available activity signal;
timely activity estimation is the next modeling target.
We contribute a causal implementation of SWE and PTR for wearer activity
anticipation; an EasyCom audit separating global, local-audio, pooled-visual,
and privileged information conditions; and a protocol for EasyCom and Ego4D
transfer with endpoint and hard-slice diagnostics. Matched tail and Ego4D
scores are still pending, so our measured claims remain confined to the
available EasyCom screens.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Deployable global--local path and bounded stream. At each 200-ms
cutoff, the head fuses the frozen integrated Omni state with explicit recent
audio and optional pooled person-crop vision. Completed 2-s payloads are
reused; only the observed partial tail is temporarily recomputed. Privileged
past-activity audit labels are kept outside this graph and are not inputs.}
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
audio and optional pooled person-crop vision to a frozen-Omni head. PTR
temporarily queries the observed current-block prefix while reusing completed
2-s payloads. Privileged past-activity labels are not model inputs.}
\label{fig:overview}
\end{figure}
\section{Method}
\label{sec:method}
\subsection{Task and Causal Inputs}
\subsection{Task and Short-Window Encoder Evidence}
At cutoff $t_n=0.2n$~s, the model observes only media available by $t_n$ and
predicts activity at $t_n+h$, where $h\in\{0.2,0.4,\ldots,2.0\}$~s. Each
@@ -116,17 +131,16 @@ horizon has three channels in fixed order: background, camera wearer, and
aggregate others. Wearer and others may overlap; no output identifies an
individual speaker, addressee, or semantic end of turn.
The global feature $\vect g_n$ is extracted from frozen Qwen2.5-Omni-3B. We
concatenate the final four Thinker readout states and project them to 2048
dimensions. The readout contains both audio and video context; $\G$ is not an
audio-free condition. The local audio feature $\vect a_n$ is obtained by
cropping the most recent one-second mixed waveform \emph{before}
preprocessing, encoding it with the frozen Qwen audio encoder, and averaging
valid output tokens. An optional visual feature $\vect v_n$ averages spatial
tokens inside provided person boxes, people within each causal frame, and
valid frames from the most recent second. It preserves neither identity nor
mouth-motion trajectories, and provided boxes do not constitute an online
detector.
The global feature $\vect g_n$ concatenates the final four frozen
Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
already contains audio and video context; $\G$ is not audio-free. SWE
adds encoder-side features without requiring the Thinker state to preserve
their fine timing. Local audio $\vect a_n$ crops the latest one-second mixed
waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and
averages valid tokens. Optional $\vect v_n$ averages frozen vision tokens
inside provided person boxes, then across people and valid causal frames in
the latest second. This pooling has no identity or mouth-motion trajectory;
provided boxes do not imply an online detector.
Each enabled local feature is layer-normalized, projected to 256 dimensions,
and paired with a validity bit. The trainable head is
@@ -143,15 +157,17 @@ head with the EgoSpeak multi-positive cross-entropy over causal-valid
source--horizon pairs. The Omni and modality encoders remain frozen, so this
is frozen-backbone adaptation rather than end-to-end training.
\subsection{Cutoff-Aligned Streaming}
\subsection{Causal Partial-Tail Refresh}
Figure~\ref{fig:overview} uses source-aligned 2-s blocks under an 8-s
presentation cap. After warm-up, three completed blocks and the observed
current tail are available; span varies with block phase. Completed
modality payloads are reused. At each cutoff, the tail is cropped at $t_n$
and queried in a temporary cache, without committing it. Only completed
blocks are committed; session or source discontinuities reset caches. This
refreshes the logical state at 5~Hz without backfilling earlier rows.
PTR uses source-aligned 2-s blocks under an 8-s maximum presentation cap
(Fig.~\ref{fig:overview}). After warm-up, three completed blocks and the
observed current tail are available; span varies with block phase. Reusing
completed modality payloads, we crop the tail at $t_n$ and query it in a
temporary cache. The tail and query are not committed. Only completed
blocks enter persistent history; session and source discontinuities reset
caches. Thus each 5-Hz readout sees the observed prefix without future
backfill. PTR is a causal access rule, not by itself a proven accuracy or
real-time latency improvement.
\subsection{Visible-Activity Information Audit}
@@ -255,8 +271,8 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\
\end{table*}
\begin{table}[t]
\caption{B result slots: paired within-backbone local-audio gains (unmeasured).
MiniCPM requires a matched readout/refresh/history audit before comparison.}
\caption{Pending paired $\G\rightarrow\G+\A$ transfer (unmeasured).
EC is EasyCom; MiniCPM first needs readout/refresh/history audits.}
\label{tab:backbone}
\centering\normalsize
\setlength{\tabcolsep}{3pt}
@@ -264,9 +280,10 @@ MiniCPM requires a matched readout/refresh/history audit before comparison.}
\toprule
Backbone & $\G$ & $\G+\A$ & $\Delta$ \\
\midrule
Qwen Omni 3B & -- & -- & -- \\
Qwen Omni 7B & -- & -- & -- \\
MiniCPM-o 4.5 & -- & -- & -- \\
EC: Omni 3B & -- & -- & -- \\
EC: Omni 7B & -- & -- & -- \\
EC: MiniCPM-o 4.5 & -- & -- & -- \\
Ego4D: Omni 3B & -- & -- & -- \\
\bottomrule
\end{tabular}
\end{table}
@@ -320,6 +337,15 @@ single-seed screen; the information audit uses seeds 0, 1, and 2 with an
identical fixed recipe. We therefore report standard deviations only for the
latter and treat small differences in the former descriptively.
\textbf{Ego4D transfer protocol.}
Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak
manifest and targets, with 346 training and 87 test clips. The same 5-Hz
cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and
$\G+\A+\V$ comparisons; visual validity is retained rather than silently
dropping no-box rows. The repository contains the extraction and training
workflow but no archived Ego4D score or matched PTR control. Hence this
protocol is specified, not reported as an observed cross-dataset gain.
\textbf{Metrics.}
At each horizon we compute AP for the three activity channels over common
causal-valid rows, then average classes to obtain mAP and average all ten
@@ -345,25 +371,22 @@ transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
They share the official split and physical cutoffs; the 8-s T/C row reuses
one run. The source retains unmeasured hypotheses and falsifiers.
\section{Results}
\section{Results and Analysis}
\label{sec:results}
\subsection{Global Context and Recent Audio Are Complementary}
\subsection{EasyCom: Local Encoder Evidence}
Table~\ref{tab:main} shows that $\A$ is strong at 0.2~s, while $\G$ is
slightly stronger when mAP is averaged to 2.0~s. Their fusion is better than
$\A$ at every horizon: the mean gain is 4.60 points, with gains of 2.03 at
0.2~s and 4.47 at 2.0~s. It also improves every class when averaged over
horizons (+8.72 background, +2.43 wearer, and +2.65 others). Relative to
released EgoSpeak, $\G+\A$ improves 0.2-s mAP by 7.52 points and mean mAP by
3.92 points. The endpoint effect is asymmetric: offset AP rises by 7.23
points, while onset AP rises by 1.47. Dense forecasting gains therefore do
not imply equal progress at both behavior boundaries.
Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at
0.2~s while integrated $\G$ is slightly stronger across all horizons.
Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over
$\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23
points, onset AP by only 1.47: dense gains do not imply equal improvement
at both activity boundaries.
The global-only head has more trainable parameters and a longer context than
the local-only head, so this comparison demonstrates predictive
complementarity, not that semantic reasoning alone causes the gain. A wider
history-matched control is required for a representation-level causal claim.
The $\G$-only head has more trainable parameters and longer history than the
$\A$-only head. Thus this screen shows predictive complementarity, not a
capacity-matched proof that the Thinker lacks a specific local cue.
% Retained as a planning aid; illustrative, unmeasured patterns do not belong
% in the page-limited submission preview.
@@ -429,7 +452,7 @@ not real-time deployment. \\
\end{table*}
\fi
\subsection{Generic Vision Does Not Recover the Available Signal}
\subsection{EasyCom: Visual Evidence and Ablations}
Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when
@@ -470,53 +493,45 @@ behavioral improvement is asserted by this placeholder.}
\label{fig:hq_cases}
\end{figure}
\subsection{Pending Controls and Limits}
\subsection{Chunk Timing and Cross-Dataset Limits}
Tables~\ref{tab:stream_context}--\ref{tab:efficiency} reserve result slots
for T/C, B, H, and S; Fig.~\ref{fig:hq_cases} reserves Q cases. Panel B of
Table~\ref{tab:stream_context} separately reports a global-only chunk screen:
the 200-ms commits beat 2-s hold after startup exclusion but require about
ten times as many processor calls across 12 sessions. Freshness and chunk
construction both change, and the older tail bank is not extraction-matched;
these values do not fill the pending T/C factorial or establish real-time
throughput. L requires a
trained width-matched global control and past-delayed features on identical
valid pairs; T crosses tail refresh with local audio; C re-extracts each
causal context cap and pairs it with a global-only run. B compares the
within-backbone audio increment only after readout and history audits.
H/Q reports slice counts and cutoff-marked examples; S measures full-path
latency and memory. The oracle uses privileged labels, provided boxes omit
online detection, and aggregate others cannot establish addressee or
semantic floor ownership.
Table~\ref{tab:stream_context}, panel B, reports a separate global-only
chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold
at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539
processor calls over 12 sessions. Completed blocks plus observed tail reach
70.85\%; that older bank is not extraction-matched. Freshness and chunk
construction vary together, so this is motivation for, not evidence of, an
isolated PTR benefit or real-time throughput. Panel A reserves the matched
tail $\times$ audio and context-cap tests. Tables~\ref{tab:backbone}--
\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the
cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured.
Ego4D has no archived test metrics in this checkout, so transfer cannot yet
be claimed. The oracle uses privileged labels, provided boxes omit online
detection, and aggregate others cannot establish addressee or semantic floor
ownership.
\subsection{Decision Rules and Reproducibility}
The unit of comparison is the same physical cutoff and future target time,
not an arbitrary feature row. For every new condition we will archive
input-source timestamps, common valid-pair counts, seeds, per-session AP,
trainable parameter counts, processor/model revisions, checkpoint hashes,
and prediction scores. Frames and horizons are correlated, not independent
replicates.
We retain the fixed four-epoch recipe for the matched matrix. Any loss
weight, threshold, context selection, or case-selection rule is frozen
using training-side evidence before looking at Sessions~1--3. Because
these test sessions have already informed this research program, even
new seed runs remain exploratory rather than restoring an unseen test set.
The raw endpoint AP in Table~\ref{tab:main} may motivate a behavioral
hypothesis, but an ENTRY/YIELD error-rate claim additionally requires
one-event-one-vote attribution with response windows censored at the next
other onset and yield windows at the next other offset. Until that
competing-risk evaluator passes real-timeline audit, we report only dense
forecasting and endpoint anticipation.
Comparisons require identical physical cutoffs and future targets. New runs
must archive source timestamps, common valid-pair counts, seeds, per-session
AP, trainable counts, revisions, checkpoints, and predictions; correlated
frames are not independent replicates. The matched matrix keeps a fixed
four-epoch recipe, with selection rules frozen on training-side evidence.
Sessions~1--3 have already been inspected, so repeat seeds remain exploratory.
Raw endpoint AP is not one-event-one-vote ENTRY/YIELD: response windows must
be censored at the next other onset and yield windows at the next other
offset. Until that evaluator passes a real-timeline audit, we report dense
forecasting and endpoint anticipation only.
\section{Conclusion}
In this strictly causal audit, frozen Omni context and recent audio improve
wearer activity forecasting, especially future speech cessation. Generic
pooled vision does not recover the timely visible-activity signal exposed by
the analysis-only oracle. Matched controls and censored event evaluation
remain necessary before claiming improved floor behavior.
On EasyCom, a short-window audio-encoder path complements the frozen Omni
readout for future wearer activity, most clearly at speech offset. Pooled
vision recovers little of the timely visible-activity information exposed by
the analysis-only audit. Partial-tail refresh provides a causal 5-Hz
interface, but its independent predictive and efficiency effects await
matched tests. Ego4D transfer and censored event evaluation remain necessary
before claiming generalizable or improved semantic floor behavior.
\section{Acknowledgment}
OpenAI Codex assisted with drafting and figure layout; the authors remain
manuscript/targeted/references.bib
+145
View File
@@ -61,3 +61,148 @@
year = {2025},
doi = {10.1109/ICASSP49660.2025.10888166}
}
@inproceedings{russell2025mmvap,
author = {Sam O'Connor Russell and Naomi Harte},
title = {Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction},
booktitle = {Findings of ACL},
pages = {209--221},
year = {2025},
doi = {10.18653/v1/2025.findings-acl.12}
}
@inproceedings{elmers2025triadic,
author = {Mikey Elmers and Koji Inoue and Divesh Lala and Tatsuya Kawahara},
title = {Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems},
booktitle = {Proc. Interspeech},
year = {2025},
url = {https://www.isca-archive.org/interspeech_2025/elmers25_interspeech.html}
}
@inproceedings{lin2025backchannel,
author = {Yuxin Lin and Yinglin Zheng and Ming Zeng and Wangzheng Shi},
title = {Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals},
booktitle = {Proc. ACL},
pages = {15310--15322},
year = {2025},
doi = {10.18653/v1/2025.acl-long.743}
}
@inproceedings{lee2024eot,
author = {Meng-Chen Lee and Zhigang Deng},
title = {Online Multimodal End-of-Turn Prediction for Three-party Conversations},
booktitle = {Proc. International Conference on Multimodal Interaction},
pages = {57--65},
year = {2024},
doi = {10.1145/3678957.3685742}
}
@article{jiang2026turnbench,
author = {Freeman Jiang and others},
title = {{TurnBench}: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue},
journal = {arXiv preprint arXiv:2608.25218},
year = {2026}
}
@inproceedings{grauman2022ego4d,
author = {Kristen Grauman and Andrew Westbury and Eugene Byrne and others},
title = {{Ego4D}: Around the World in 3,000 Hours of Egocentric Video},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages = {18995--19012},
year = {2022}
}
@inproceedings{kong2024sicnet,
author = {Deqian Kong and Furqan Khan and Xu Zhang and Prateek Singhal and Ying Nian Wu},
title = {Long-Term Social Interaction Context: The Key to Egocentric Addressee Detection},
booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2024}
}
@inproceedings{jiang2022asl,
author = {Hao Jiang and Calvin Murdock and Vamsi Krishna Ithapu},
title = {Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2022}
}
@inproceedings{clarke2025asd,
author = {Jason Clarke and Yoshihiko Gotoh and Stefan Goetze},
title = {Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings},
booktitle = {Proc. IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2025},
url = {https://arxiv.org/abs/2502.06012}
}
@inproceedings{wang2024loconet,
author = {Xizi Wang and Feng Cheng and Gedas Bertasius and David Crandall},
title = {{LoCoNet}: Long-Short Context Network for Active Speaker Detection},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2024}
}
@inproceedings{oconnorrussell2025noise,
author = {Sam O'Connor Russell and Naomi Harte},
title = {Visual Cues Support Robust Turn-taking Prediction in Noise},
booktitle = {Proc. Interspeech},
year = {2025},
url = {https://www.isca-archive.org/interspeech_2025/oconnorrussell25_interspeech.html}
}
@article{cui2026minicpmo45,
author = {Junbo Cui and Bokai Xu and Chongyi Wang and others},
title = {{MiniCPM-o 4.5}: Towards Real-Time Full-Duplex Omni-Modal Interaction},
journal = {arXiv preprint arXiv:2604.27393},
year = {2026}
}
@article{tian2026roma,
author = {Xueyun Tian and Wei Li and Bingbing Xu and Heng Dong and Yuanzhuo Wang and Huawei Shen},
title = {{ROMA}: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding},
journal = {arXiv preprint arXiv:2601.10323},
year = {2026}
}
@article{yan2026duplug,
author = {Ruiqi Yan and Wenxi Chen and Zhanxun Liu and others},
title = {{SoulX-Duplug}: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation},
journal = {arXiv preprint arXiv:2603.14877},
year = {2026}
}
@article{yu2025salmonnomni,
author = {Wenyi Yu and Siyin Wang and Xiaoyu Yang and others},
title = {{SALMONN-omni}: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation},
journal = {arXiv preprint arXiv:2505.17060},
year = {2025}
}
@inproceedings{chen2024videollmonline,
author = {Joya Chen and others},
title = {{VideoLLM-online}: Online Video Large Language Model for Streaming Video},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2024}
}
@inproceedings{wang2025omnimmi,
author = {Yuxuan Wang and others},
title = {{OmniMMI}: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2025}
}
@inproceedings{lin2023lavish,
author = {Yan-Bo Lin and Yi-Lin Sung and Jie Lei and Mohit Bansal and Gedas Bertasius},
title = {Vision Transformers Are Parameter-Efficient Audio-Visual Learners},
booktitle = {Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages = {2299--2309},
year = {2023}
}
@inproceedings{rouditchenko2024whisperflamingo,
author = {Andrew Rouditchenko and Yuan Gong and Samuel Thomas and Leonid Karlinsky and Hilde Kuehne and Rogerio Feris and James Glass},
title = {{Whisper-Flamingo}: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation},
booktitle = {Proc. Interspeech},
pages = {2420--2424},
year = {2024}
}