wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revise manuscript positioning around causal egocentric floor entry

Browse Source
This commit is contained in:
Xin Wang
2026-09-08 13:25:01 +08:00
parent 4908d00c79
commit 0534c9d3d0
2 changed files with 144 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/main/main.tex
+103 -114
View File
@@ -10,7 +10,7 @@
\newcommand{\vect}[1]{\boldsymbol{#1}} \newcommand{\vect}[1]{\boldsymbol{#1}}
\setlength{\emergencystretch}{2em} \setlength{\emergencystretch}{2em}
\title{Adapting Omni Models for Egocentric Multiparty Floor-Entry Forecasting} \title{Adapting Omni Models for Causal Egocentric Multiparty Floor-Entry Forecasting}
% TODO: Insert the final author list and affiliations after the ICASSP 2027 % TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized. % review policy and author order are finalized.
@@ -22,24 +22,26 @@
\maketitle \maketitle
\begin{abstract} \begin{abstract}
An egocentric conversational agent operating among several people must decide In multiparty conversation, a pause need not invite an egocentric participant
whether to speak or keep listening before generating a response. Following to speak: it may instead mark a handover between two other people. We study
EgoSpeak, we use causal first-person audio--video as observation and the future causal egocentric multiparty floor-entry forecasting, where the target is the
speaking behavior of the human camera wearer as an imitation target for this future speech participation of the first-person camera wearer---the one
agent decision. We ask whether a jointly pretrained Omni model exposes the participant who is necessarily absent from the visible speaker set. This
fine-grained floor cues required every 200~ms. Our global--local formulation wearer-relative target distinguishes our setting from recent multimodal VAP
uses a causal global Omni readout for broad scene and conversational context, models that forecast visible floor holders or observe human--human turn
while a causal local branch preserves recent mixed audio and a variable set of dynamics. Following EgoSpeak, we use causal first-person audio--video and the
track-organized person observations. Anonymous tracks are an organizational human wearer's future speaking behavior as an imitation target, and ask how a
interface whose incremental value must be established, not an assumption that jointly pretrained audio--visual Omni model should be adapted for predictions
the model knows speaker identity. On EasyCom development data, a single-seed every 200~ms. Our global--local formulation combines a causal Omni readout with
oracle analysis finds that perfectly aligned aggregate activity among visible a lightweight pathway for recent mixed audio and visible-person observations.
non-wearers improves 0.2-s mAP by 6.97 points and ten-horizon mean mAP by 2.91 On EasyCom development data, a single-seed oracle analysis finds that perfectly
points over global Omni plus recent audio. Person-resolved histories do not aligned aggregate activity among visible non-wearers improves 0.2-s mAP by
outperform an identity-shuffled control. This result motivates timely local 6.97 points and ten-horizon mean mAP by 2.91 points over global Omni plus recent
floor evidence, while explicitly withholding a person-continuity claim. The audio, whereas person-resolved histories do not outperform an
full study compares frozen, parameter-efficient, and learned local variants identity-shuffled control. This motivates timely local floor evidence without
and evaluates false agent entry at matched wearer-onset recall. assuming that person continuity is useful. The full study compares frozen,
parameter-efficient, and learned local adaptation and evaluates false wearer
entry at matched wearer-onset recall together with yield behavior.
\todo{After the train-side study is locked, replace the final two sentences \todo{After the train-side study is locked, replace the final two sentences
with the selected learned local variant and insert: its oracle-gap recovery, with the selected learned local variant and insert: its oracle-gap recovery,
multi-other false-entry reduction at 80\% onset recall, onset/yield multi-other false-entry reduction at 80\% onset recall, onset/yield
@@ -55,107 +57,94 @@ omni models
\section{Introduction} \section{Introduction}
\label{sec:intro} \label{sec:intro}
We consider a conversational agent equipped with an egocentric camera and Natural turn-taking requires a conversational system to anticipate not only
microphone and deployed in a scene where several people may speak. Before when speech will stop, but whether an emerging floor opportunity actually
response generation and speech synthesis, its causal floor controller must belongs to it. This distinction is particularly important in multiparty
decide every 200~ms whether to continue listening, begin speaking, or yield. A interaction. In a dyadic conversation, a pause after one interlocutor finishes
pause in dyadic dialogue may invite a response; in multiparty interaction it often provides an opportunity for the other to respond; in a group
may instead mark a handover between two other people. An agent that reacts only conversation, the same pause may simply mark a handover between two other
to acoustic availability can therefore interrupt a conversation that it participants. An egocentric conversational agent that reacts only to silence
should continue observing. or aggregate speech activity may therefore enter the floor while the
surrounding conversation should continue without it. We study this problem as
\emph{egocentric multiparty floor-entry forecasting}: continuously predicting
the near-future speech participation of the first-person participant from
causal audio--visual observations.
Existing datasets do not contain autonomous agents making these decisions at Voice Activity Projection (VAP) provides a predictive formulation of
scale. Following EgoSpeak \cite{kim2025egospeak}, we use human egocentric turn-taking by continuously projecting the joint future voice activity of
recordings to construct supervision: the camera wearer defines the first-person conversational participants, from which turn shifts, holds, and backchannels
sensor origin, and the wearer's observed speaking behavior defines the desired can be inferred \cite{ekstedt2022vap}. EgoSpeak extends predictive speech
floor-action target. This is an imitation-learning proxy for a future agent initiation to the first-person setting, using causal, untrimmed egocentric
under comparable observations. Throughout the paper, \emph{wearer} names this audio--visual observations and the future speaking behavior of the camera
dataset role and target channel; it is neither the deployed agent nor a wearer as supervision \cite{kim2025egospeak}. This behavior is a useful
semantic model input. Response content and closed-loop effects of deploying an imitation target for the perceptual decision faced by an egocentric agent:
agent remain outside our scope. given everything observed so far, is the first-person participant likely to
enter the floor next? EgoSpeak establishes that this prediction can be learned
from real-world egocentric streams, but leaves open how recent jointly
pretrained audio--visual Omni models should be used for such fine-grained
causal forecasting.
EgoSpeak already establishes real-time speech-initiation prediction in causal, In parallel, recent work has expanded predictive turn-taking along multimodal
untrimmed egocentric video. It combines separately pretrained ResNet-50 and and multiparty dimensions. MM-VAP incorporates facial expression, head pose,
wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba and gaze into predictive turn-taking for two-party visual interaction
temporal models, forecasting the camera wearer and aggregate other-person \cite{russell2025mmvap}. A recent social-robot MM-VAP framework adapts
activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is voice-activity-related audio--visual encoders for anticipating human--human
not the task definition or the use of first-person data. The unresolved model turn dynamics in observer or mediation settings \cite{cano2026mmvap}. MuVAP
question is whether a jointly pretrained audio--visual Omni backbone, further extends VAP to multiparty interaction using a monaural mixed-audio
operating on causal raw media, exposes an adequate state for this dense stream and a single camera, grounding acoustic dynamics in visible face tracks
floor-action controller. To our knowledge, such omni representations have not been and predicting current and future visible floor holders \cite{qi2026muvap}.
systematically evaluated or adapted for the EgoSpeak task. These formulations are complementary to our egocentric setting. In MuVAP, the
candidate future speakers belong to the set visible to the camera; in
mediation-oriented MM-VAP, the system observes turn dynamics between human
participants. Here, by contrast, the participant whose future floor entry must
be predicted is located \emph{behind the camera} and is therefore absent from
the visible participant set.
Our working hypothesis is a representational decomposition. A global causal This leaves an underexplored intersection: continuous causal prediction of an
Omni state may retain broad scene, speech, and semantic context while making egocentric participant's own future floor participation in a changing
recent prosody, activity boundaries, and localized person motion difficult for multiparty scene using a general-purpose audio--visual Omni model. The defining
a shallow 5-Hz controller to recover. A causal local branch therefore difference is its viewpoint and prediction target: the model must project the
preserves recent mixed audio and a variable-size set of person observations activity of the unseen first-person participant from the causal stream used to
before fusing them back into the global state. The local set can be pooled observe everyone else. We investigate how pretrained Omni representations can
within a frame or organized into anonymous tracks across time. Whether that be adapted under the EgoSpeak wearer-relative forecasting objective. Current
continuity adds useful information beyond aggregate activity is itself a Qwen2.5-Omni results motivate this study, but do not establish a limitation
falsifiable question. Frozen, parameter-efficiently tuned, and locally shared by Omni models in general.
augmented variants must consequently be compared within each backbone.
An initial causal oracle screen sharpens this hypothesis. On held-out EasyCom Our working hypothesis separates broad context from timely floor evidence. A
Session~12, perfectly timed aggregate activity from currently visible causal global Omni state can represent the scene and conversational history,
non-wearers improves a matched global-plus-audio model by 6.97 points at 0.2~s while a lightweight local pathway preserves recent mixed audio and a changing
and 2.91 points in mean mAP. Visibility alone and activity delayed by one set of visible-person observations. The latter may be aggregated per frame or
second remain near the audio baseline. However, person-resolved histories are organized into anonymous tracks; track continuity becomes a contribution only
weaker than aggregate histories and indistinguishable from histories whose if it outperforms aggregate, no-ID, shuffled, and capacity-matched controls. An
identity continuity is shuffled. Thus the current evidence supports the value analysis-only oracle screen on held-out EasyCom Session~12 supports the weaker
of timely local floor information, not a claim that person identity or track hypothesis: perfectly timed aggregate visible-other activity improves a
continuity has already been exploited. matched global-plus-audio model by 6.97 points at 0.2~s and 2.91 points in
ten-horizon mean mAP, whereas person-resolved histories do not outperform an
identity-shuffled control. Thus the current evidence motivates timely local
floor information, not speaker identity or addressee understanding.
Voice Activity Projection (VAP) provides the broader future-activity view of We retain EgoSpeak-compatible dense future-activity prediction for
turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves comparability and add a behavior-aligned event analysis. In particular,
compatibility with that view, but its aggregate ``others'' channel discards per-frame average precision can hide the failure that motivates this work:
whether the same or different visible people produced recent activity. entering during an exchange among other participants. We therefore measure
Our oracle result shows that this discarded distinction is not automatically false wearer entries during multi-other opportunities at matched wearer-onset
useful under the dense target: identity continuity must beat aggregate and recall, together with response latency and wearer-yield behavior. These events
shuffled controls before it enters the method claim. are observable timing-derived interaction proxies, not semantic EOT, INT, or
Moreover, conventional per-frame average precision can hide the behavioral addressee labels; the final protocol uses competing-risk censoring so an event
failure of entering during an exchange between others. We therefore retain the cannot claim a later action across an incompatible floor change. This follows
dense EgoSpeak target while adding event evaluation of agent entry and yield; the broader event-oriented emphasis on recall, false alarms, and latency
event-oriented work likewise emphasizes recall, false alarms, and latency \cite{jiang2026turnbench}.
rather than a single frame-level score \cite{jiang2026turnbench}.
Related work supplies complementary parts of the problem. Egocentric Our contributions are: (1) a controlled study of frozen and
talking-to-me detection predicts whether a completed utterance addresses the parameter-efficient Omni adaptation for EgoSpeak-style causal forecasting;
camera wearer and benefits from long-term social context (2) an oracle information audit that distinguishes timely aggregate local
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face floor evidence from person continuity; (3) a global--local adaptation selected
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}, through causal, alignment, and capacity-matched controls; and (4) an event
while AV-CONV reconstructs current person-level speaking/listening relations protocol that tests whether dense gains reduce false first-person floor entry
from egocentric audio--video \cite{jia2024avconv}. The former talking-to-me without suppressing valid entry or yield. EasyCom supplies synchronized
task is generally utterance-level rather than continuous anticipation; egocentric audio--video, per-person tracks, and voice activity
MuVAP predicts visible participants from a static third-person view; and \cite{donley2021easycom}; target-of-speech annotations, when used, serve only
AV-CONV estimates a current interaction graph rather than strictly causal as analysis strata and never as model input or required output.
future wearer action. Our gap is therefore narrower than ``audio--visual
multiparty turn taking'': we study whether omni representations support
direct, continuous agent floor-action prediction and whether causal local
evidence helps when the visible participant set changes.
We call this task \emph{egocentric floor-action projection}. The study asks
four questions: (RQ1) how accurately a frozen global Omni state predicts the
wearer-derived agent target and other-person activity; (RQ2) whether timely
causal local audio and visible-person evidence complement that state; (RQ3)
whether aggregate person-set evidence is sufficient or anonymous track
continuity adds measurable value; and (RQ4) whether any dense gain reduces
false agent entries without making the controller globally passive.
Parameter-efficient tuning is a matched adaptation control rather than an
assumed remedy. EasyCom is suitable because
it provides synchronized egocentric audio--video, per-person tracks, and voice
activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
analysis strata only and are never model input or required output.
The intended contributions are therefore: a controlled study of global Omni
representations for EgoSpeak-style floor action; an oracle information audit
that separates timely aggregate floor evidence from person continuity; a
global--local adaptation selected from progressively structured candidates;
and a behavior-aligned event evaluation that maps wearer demonstrations to
agent entry and yield decisions. Current Qwen2.5-Omni observations are
motivation, not a claim that all Omni models intrinsically fail. Likewise,
track-consistent cross-attention becomes the final method only if it beats
aggregate, shuffled/no-ID, and capacity-matched controls.
\begin{figure}[t] \begin{figure}[t]
\centering \centering
manuscript/main/references.bib
+41 -10
View File
@@ -1,16 +1,45 @@
@inproceedings{ekstedt2022vap, @inproceedings{ekstedt2022vap,
author = {Erik Ekstedt and Gabriel Skantze}, author = {Erik Ekstedt and Gabriel Skantze},
title = {Voice Activity Projection: Self-Supervised Learning of Turn-Taking Events}, title = {{Voice Activity Projection: Self-Supervised Learning of Turn-Taking Events}},
booktitle = {Proc. Interspeech}, booktitle = {Proc. Interspeech 2022},
year = {2022} pages = {5190--5194},
year = {2022},
doi = {10.21437/Interspeech.2022-10955}
} }
@inproceedings{kim2025egospeak, @inproceedings{kim2025egospeak,
author = {Junhyeok Kim and Min Soo Kim and Jiwan Chung and Jungbin Cho and Jisoo Kim and Sungwoong Kim and Gyeongbo Sim and Youngjae Yu}, author = {Junhyeok Kim and Min Soo Kim and Jiwan Chung and Jungbin Cho and Jisoo Kim and Sungwoong Kim and Gyeongbo Sim and Youngjae Yu},
title = {{EgoSpeak}: Learning When to Speak for Egocentric Conversational Agents in the Wild}, title = {{EgoSpeak}: Learning When to Speak for Egocentric Conversational Agents in the Wild},
booktitle = {Findings of the Association for Computational Linguistics: NAACL}, booktitle = {Findings of the Association for Computational Linguistics: NAACL 2025},
publisher = {Association for Computational Linguistics},
address = {Albuquerque, New Mexico},
pages = {2990--3005}, pages = {2990--3005},
year = {2025} year = {2025},
doi = {10.18653/v1/2025.findings-naacl.163},
url = {https://aclanthology.org/2025.findings-naacl.163/}
}
@inproceedings{russell2025mmvap,
author = {Sam O'Connor Russell and Naomi Harte},
title = {Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
publisher = {Association for Computational Linguistics},
address = {Vienna, Austria},
pages = {209--221},
year = {2025},
doi = {10.18653/v1/2025.findings-acl.12},
url = {https://aclanthology.org/2025.findings-acl.12/}
}
@misc{cano2026mmvap,
author = {Antonio Cano and Guillermo P{\'e}rez and Luis Merino and Randy Gomez},
title = {Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders},
year = {2026},
eprint = {2607.07294},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
note = {Accepted at IEEE RO-MAN 2026},
url = {https://arxiv.org/abs/2607.07294}
} }
@article{jiang2026turnbench, @article{jiang2026turnbench,
@@ -27,11 +56,13 @@
year = {2024} year = {2024}
} }
@article{qi2026muvap, @inproceedings{qi2026muvap,
author = {Haotian Qi and Gabriel Skantze}, author = {Haotian Qi and Gabriel Skantze},
title = {{MuVAP}: Multimodal Multiparty Voice Activity Projection for Turn-Taking Prediction in the Wild}, title = {{MuVAP}: Multimodal Multiparty Voice Activity Projection for Turn-Taking Prediction in the Wild},
journal = {arXiv preprint arXiv:2606.16731}, booktitle = {Proc. Interspeech 2026},
year = {2026} year = {2026},
note = {arXiv:2606.16731},
url = {https://arxiv.org/abs/2606.16731}
} }
@article{donley2021easycom, @article{donley2021easycom,