wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build ICASSP manuscript and experiment framework

Browse Source
This commit is contained in:
Xin Wang
2026-08-31 14:48:36 +08:00
parent 87aff8f00e
commit 69cec4f6bf
3 changed files with 487 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/main/main.tex
+323 -74
View File
@@ -1,12 +1,17 @@
% Draft ICASSP 2027 manuscript using the official ICASSP 2026 template
% currently bundled in this repository. Replace the template when the ICASSP
% 2027 author kit is released.
% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
% in this repository. Replace the class/style files when the 2027 author kit
% is released, and remove every visible \todo{} marker before submission.
% Compile from this directory with: make
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,graphicx,hyperref}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
\title{NOT EVERY TURN IS MINE: ADDRESSEE-GROUNDED FLOOR-ENTRY PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
\newcommand{\method}{WearerSet}
\newcommand{\vect}[1]{\boldsymbol{#1}}
\title{NOT EVERY OPEN FLOOR IS MINE: ADDRESSEE-CONDITIONED FLOOR-ENTRY
PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
% TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized.
@@ -18,22 +23,23 @@
\maketitle
\begin{abstract}
Egocentric conversational agents must decide not only when a speaking
opportunity arises, but also whether that opportunity belongs to them.
Existing speech-initiation forecasting collapses all non-wearer speakers, so
strong frame-level accuracy can coexist with intrusive entries while two
bystanders converse. We study addressee-grounded floor-entry projection in
causal multiparty audio-video streams. Our formulation retains dense forecasts
of wearer and non-wearer activity at 200-ms resolution over a two-second
horizon, while grounding the wearer-entry decision in person-aligned visual
context. Using EasyCom target-of-speech annotations, we introduce an event-level
protocol that separates addressed-to-wearer responses from third-party
exchanges and measures false entries at matched response recall, together with
decision latency. Wrong-person and time-shifted visual controls test whether
any improvement follows correct social grounding rather than additional model
capacity. Experiments on EasyCom, with Ego4D as an activity-forecasting
generalization set, will evaluate this hypothesis under a strictly causal
protocol.
An egocentric conversational agent must decide not only when a speaking
opportunity arises, but also whether that opportunity belongs to it. Existing
speech-activity projection collapses all non-wearer speakers, allowing strong
frame-level accuracy to coexist with intrusive entries during conversations
between bystanders. We formulate addressee-conditioned floor-entry projection:
dense causal forecasts of wearer and non-wearer activity at 200-ms resolution,
conditioned on a variable-cardinality set of visible participants. Our model
augments a frozen streaming omni-model with person-aligned audio--visual
representations and permutation-invariant role queries for floor state,
wearer address, and third-party interaction. We further introduce an EasyCom
event protocol that separates addressed-to-wearer responses from third-party
exchanges and reports false entries at matched response recall and decision
latency. Correct-, wrong-person, time-shifted, and removed-video controls test
whether a gain follows social grounding rather than model capacity.
\todo{Replace this sentence with the main result: FP/min reduction at 80\%
response recall, response latency/recall non-degradation, and 95\% confidence
interval.}
\end{abstract}
\begin{keywords}
@@ -45,65 +51,308 @@ audio-visual learning
\label{sec:intro}
Conversational agents must continuously decide whether to speak or keep
listening. In a dyadic exchange, a pause or an apparent end of turn is
often treated as an invitation to respond. This assumption fails in
multiparty interaction: one participant may finish an utterance addressed to
another participant, and an egocentric agent that reacts only to acoustic
availability will enter a floor that was never offered to it. The resulting
error is socially more disruptive than a small framewise prediction error.
Accordingly, a useful model must distinguish \emph{when the floor may change}
from \emph{whether the change is directed to the camera wearer}.
listening. In a dyadic exchange, a pause or an apparent end of turn is often
treated as an invitation to respond. This assumption fails in multiparty
interaction: one participant may finish an utterance addressed to another
participant, and an egocentric agent that reacts only to acoustic availability
will enter a floor that was never offered to it. Such an error is socially more
disruptive than a small framewise prediction error. A useful agent must
therefore distinguish \emph{when the floor may change} from \emph{whether the
change is directed to the camera wearer}.
Voice Activity Projection (VAP) models future joint speech activity and derives
turn-taking events without manually labeling every event
Voice Activity Projection (VAP) models future joint speech activity and
derives turn-taking events without manually labeling each event
\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
untrimmed egocentric video, forecasting wearer and non-wearer activity over the
next two seconds \cite{kim2025egospeak}. Its role-relative output is attractive
for a wearable agent, but aggregating all visible participants into a single
``others'' channel removes the distinction between an utterance addressed to
the wearer and an exchange among bystanders. Consequently, conventional
per-frame average precision can be dominated by ordinary speech activity and
can hide precisely the false-entry behavior that visual social cues should
prevent. Recent event-oriented evaluation likewise emphasizes the joint
trade-off among recall, false positives, and latency rather than a single
frame-level score \cite{jiang2026turnbench}.
untrimmed egocentric video, forecasting wearer and aggregate non-wearer
activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
output is attractive for a wearable agent, but an ``others'' channel cannot
distinguish an utterance addressed to the wearer from an exchange between
bystanders. Conventional per-frame average precision can consequently be
dominated by ordinary activity while hiding precisely the false-entry behavior
that visual social cues should prevent. Event-oriented evaluation likewise
emphasizes recall, false alarms, and latency rather than a single frame-level
score \cite{jiang2026turnbench}.
Related work provides two complementary pieces of the problem but does not
unify them. Egocentric talking-to-me detection predicts whether a completed
utterance addresses the camera wearer and benefits from long-term social
context \cite{kong2024sicnet}. Multiparty VAP grounds future activity in
visible face tracks to predict floor shifts and next speakers
\cite{qi2026muvap}. The former is generally evaluated as utterance-level
addressee classification rather than continuous anticipation, whereas the
latter selects among visible participants and does not ask whether an
invisible camera wearer should enter. We target the intersection: causal
floor-entry forecasting for an egocentric agent whose decision depends on the
speaker--addressee relation among the people in view.
Related work supplies complementary parts of the problem. Egocentric
talking-to-me detection predicts whether a completed utterance addresses the
camera wearer and benefits from long-term social context
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
The former is generally utterance-level rather than continuous anticipation;
the latter does not ask whether an invisible camera wearer should enter. Our
gap is therefore narrower than ``audio--visual multiparty turn taking'': we
study continuous, wearer-relative floor action when the candidate responder is
behind the camera and the number of visible participants changes over time.
We formulate this problem as \emph{addressee-conditioned egocentric activity
projection}. A global streaming audio-video representation predicts future
wearer and non-wearer activity, while a person-aligned visual pathway preserves
participant identity and recent social context. The final wearer-entry score
can therefore represent both floor availability and evidence that the current
utterance is directed to the wearer. EasyCom is particularly suitable for this
analysis because it provides synchronized egocentric audio-video, per-person
tracks, voice activity, and target-of-speech annotations
\cite{donley2021easycom}. Target annotations are used for training-side
auxiliary supervision and evaluation strata, never as model input at test
time.
We call this task \emph{addressee-conditioned egocentric floor-entry
projection}. A global streaming audio--video representation supplies broad
context, while a set branch preserves individual visual histories and their
alignment to recent mixed audio. Fixed semantic role queries compress a
variable participant set into wearer-address and third-party evidence without
assuming a fixed seating order. EasyCom is suitable because it provides
synchronized egocentric audio--video, per-person tracks, voice activity, and
target-of-speech annotations \cite{donley2021easycom}. Target annotations are
used only for training-side supervision and evaluation strata, never as input
at test time.
Our contributions are threefold. First, we cast speech initiation as
recipient-aware floor-entry projection, connecting dense egocentric activity
forecasting with multiparty addressee grounding. Second, we propose an
event-level evaluation that contrasts addressed-to-wearer response events with
third-party exchanges and reports false entries at matched response recall,
alongside latency and the original EgoSpeak metrics. Third, we use
wrong-person, temporally shifted, and visibility-conditioned visual controls to
test whether improvements arise from the correct social evidence instead of
extra feature capacity or globally conservative predictions.
Our contributions are threefold. (1) We formulate continuous speech
initiation as recipient-aware floor-entry projection for an invisible camera
wearer. (2) We propose a causal set-based fusion module that grounds mixed
audio in a variable number of visible participants and preserves the original
dense EgoSpeak target. (3) We introduce an event evaluation that reports
third-party false entries at matched addressed-response recall, accompanied by
latency and visual counterfactual controls. This design makes the scientific
claim falsifiable: visual fusion is useful only if \emph{correctly aligned}
vision improves the operating trade-off without suppressing legitimate wearer
responses.
% TODO: Add Method, Experimental Setup, Results, and Conclusion after the
% event protocol and person-aligned model are frozen.
\begin{figure}[t]
\centering
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
\todo{Draw the final method figure: causal omni state $\vect g_t$; recent
mixed-audio tokens; a masked set of face-track histories; shared per-person
audio--visual grounding; three role queries; zero-initialized residual; and
the $10\times3$ activity head. Mark all causal cutoffs and masks.}}}
\caption{Proposed wearer-relative set fusion. The number and ordering of
visible participants may change, while the output remains fixed-size.}
\label{fig:method}
\end{figure}
\section{Task and Method}
\label{sec:method}
\subsection{Wearer-relative activity projection}
At decision time $t$, the model observes only the causal stream
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
For bin $k$, the target
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed; hence this is not a three-way softmax. The
output is
\begin{equation}
\hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
\qquad q_t=\hat y^{\mathrm{w}}_{t,1},
\label{eq:task}
\end{equation}
where $q_t$ is the immediate wearer-entry score used by the event protocol.
It is interpreted as an \emph{entry} score only in wearer-silent opportunity
windows; elsewhere it remains a future-activity probability. This retains the
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
signal.
\subsection{Frozen global stream and local participant set}
We obtain a causal global state $\vect{g}_t$ from the streaming Thinker of
Qwen2.5-Omni \cite{xu2025qwenomni}. The backbone is frozen in the main
experiment because preliminary LoRA adaptation caused negative transfer
(Sec.~\ref{sec:results}). From the most recent $L_a$ seconds, a shared audio
encoder produces token sequence $\vect{A}_t$. For each visible track
$i\in\{1,\ldots,N_t\}$, a shared temporal visual encoder produces
$\vect{v}_{t,i}$ from its recent face history. Person-aligned evidence is
\begin{equation}
\vect{p}_{t,i}=\vect{v}_{t,i}+
\mathrm{CA}(\vect{v}_{t,i},\vect{A}_t,\vect{A}_t),
\label{eq:person}
\end{equation}
so every participant queries the same mixed audio rather than requiring
unavailable separated microphones. A masked set encoder models interactions
among $\mathcal{P}_t=\{\vect{p}_{t,i}\}_{i=1}^{N_t}$ and is permutation
equivariant \cite{lee2019settransformer}. Missing detections use a learned
\textsc{offscreen/null} element. In minibatches, sets are padded to
$N_{\max}$ with an attention mask; no slot-ID embedding is used, and track
slots are randomly permuted during training.
\todo{Implement and verify the shared track encoder, mixed-audio
cross-attention, OFFSCREEN token, padding mask, and numerical invariance under
track-slot permutation. Freeze $L_a$, visual crop history, $N_{\max}$, and
face-track confidence filtering from train-side experiments only.}
\subsection{Role-query fusion}
Mean pooling can erase which participant is speaking to whom. Instead, three
learned semantic queries represent floor state, address-to-wearer evidence,
and third-party interaction. Each query is conditioned on $\vect{g}_t$ and
cross-attends to the participant set:
\begin{align}
\vect{Q}_t &= [\vect{q}_{\mathrm{floor}};
\vect{q}_{\mathrm{addr}};
\vect{q}_{\mathrm{third}}]+W_g\vect{g}_t,\\
\vect{R}_t &= \mathrm{MHA}(\vect{Q}_t,\widetilde{\mathcal P}_t,
\widetilde{\mathcal P}_t),\\
\vect{z}_t &= \vect{g}_t+\tanh(\alpha)W_r\mathrm{vec}(\vect{R}_t),
\label{eq:fusion}
\end{align}
where $\alpha$ is initialized to zero. The zero-initialized residual makes the
starting model identical to the frozen global baseline and allows the visual
branch to be removed exactly. A two-layer prediction head maps $\vect{z}_t$ to
Eq.~\eqref{eq:task}.
\subsection{Training objectives}
The main loss is class-balanced binary cross entropy over all horizons and
activity channels. When annotations are available, auxiliary heads predict
per-track active-speaker state and whether the current source addresses the
wearer:
\begin{equation}
\mathcal{L}=\mathcal{L}_{\mathrm{act}}+
\lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}+
\lambda_{\mathrm{addr}}\mathcal{L}_{\mathrm{addr}}.
\label{eq:loss}
\end{equation}
Ambiguous, group-addressed, and offscreen-target samples are masked from
$\mathcal{L}_{\mathrm{addr}}$, but remain in activity training. No language
model or pseudo-label is used to define primary test events.
\todo{Run the staged ablation: activity only; +ASD; +addressee. Select
$\lambda$ values and early stopping on a training-side calibration fold. Do
not inspect Sessions 1--3 during selection.}
\section{Experimental Setup}
\label{sec:setup}
\textbf{Data and splits.}
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
and hyperparameter choices use training-side folds only. Inputs contain the
camera stream and shared array audio available to the wearer; close-talking
microphones and target-of-speech labels are supervision only. We also report aggregate
activity forecasting on the EgoSpeak Ego4D split \cite{grauman2022ego4d}, but
do not claim addressee grounding there because comparable target annotations
are absent.
\textbf{Event construction.}
An \emph{addressed response} starts from a pure other-to-wearer utterance and
requires a wearer onset before an unrelated other onset. A \emph{third-party
exchange} starts from a pure other-to-other target while the wearer remains
silent. Group, none, multiple, and unknown targets form a separate diagnostic
stratum. Threshold $\tau_r$ is selected on the development split to obtain
addressed-response recall $r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary
metric is false wearer entries per minute of third-party opportunity at
matched recall; the headline operating point is $r=0.8$. We also report
addressed-response recall, signed detection latency, wearer yield recall, and
the original ten-horizon per-frame mAP.
\todo{Freeze the event pairing and competing-risk rules, including pre/post
windows, minimum silence, maximum response delay, overlap handling, and one
detection per event. Publish counts for train/test positive and negative
events, visibility, duration, and excluded target types. Treat all previously
viewed Sessions 1--3 scores as exploratory; lock the final protocol and model
before the confirmatory three-seed run.}
\textbf{Baselines and controls.}
We compare: global omni state (G); G plus recent audio (G+A); G+A plus pooled
visual context (G+A+V-pool); a parameter-matched temporal branch without
person identity; and the proposed set branch. Required counterfactuals replace
correct tracks with (i) wrong participants from the same scene, (ii)
time-shifted tracks, and (iii) zeros. These controls share parameter count,
audio history, and optimization budget.
\textbf{Implementation and statistics.}
All models run at 5~Hz with an 8-s causal memory. The 3B Thinker is frozen and
local audio/face histories are encoded causally. We report three seeds and
mean$\pm$standard deviation. Confidence intervals for event metrics use paired
cluster bootstrap over sessions/conversation blocks rather than treating
adjacent frames as independent. Runtime includes P50/P95 decision latency,
peak memory, and trainable parameter count.
\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
count, hardware, and real-time latency after the architecture is frozen.}
\section{Results and Analysis}
\label{sec:results}
\begin{table}[t]
\caption{Current aggregate evidence on EasyCom test Sessions 1--3. Values are
percentages from one seed and are motivation, not the final main result.}
\label{tab:preliminary}
\centering
\small
\begin{tabular}{lccc}
\hline
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
\hline
G+A & 81.72 & 85.07 & 65.11\\
G+A+V-pool & 82.03 & 85.09 & 65.40\\
\hline
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
\hline
\end{tabular}
\end{table}
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
Pooled vision gives a small gain, while immediate wearer AP is nearly
unchanged. This is compatible with either weak but useful social grounding or
generic visual regularization; only matched-recall event evaluation and
counterfactual identity controls can distinguish them.
\begin{table}[t]
\caption{Primary EasyCom event evaluation. Lower $\mathrm{FP}_{\mathrm{3p}}$
is better; all rows use the same development-selected operating protocol.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Model & $\mathrm{FP}_{\mathrm{3p}}$/min@$R=.8$ & $R_{\mathrm{addr}}$ &
Latency & Yield R & Mean mAP\\
\hline
G & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
G+A & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
G+A+V-pool & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Temporal control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\method{} & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\method{}+ASD & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\method{}+Addr & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}
\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
seeds, plus 95\% paired cluster-bootstrap confidence intervals for the
proposed-vs-G+A difference. The paper's main claim is supported only if
third-party FP/min decreases at matched recall without worse response latency
or wearer-yield recall.}
\begin{table}[t]
\caption{Visual grounding counterfactual on the frozen best model.}
\label{tab:counterfactual}
\centering
\small
\begin{tabular}{lccc}
\hline
Visual evidence & $\mathrm{FP}_{\mathrm{3p}}$/min & Addr. recall & 0.2-s mAP\\
\hline
Correct tracks & \todo{--} & \todo{--} & \todo{--}\\
Wrong person & \todo{--} & \todo{--} & \todo{--}\\
Time shifted & \todo{--} & \todo{--} & \todo{--}\\
Removed & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}
\end{table}
Table~\ref{tab:counterfactual} is the identity test. A true social-grounding
effect should be largest for correct tracks and should collapse for wrong or
time-shifted people. If every visual variant improves equally, the defensible
conclusion is additional capacity/history, not addressee grounding.
\todo{Add one compact error analysis split by target visibility, number of
visible people, overlap, and backchannel/interruption proxy. Include event
counts with every rate. Move negative LoRA/7B/ROMA diagnostics to supplementary
material unless they directly explain the final method.}
\section{Conclusion}
\label{sec:conclusion}
We study when an egocentric agent should enter a multiparty floor, not merely
whether someone will speak. The proposed formulation retains dense future
activity prediction while grounding wearer entry in a variable set of visible
participants. Its event protocol tests the intended behavior directly:
reducing intrusions into third-party exchanges at matched legitimate-response
recall. \todo{Replace with one quantitative conclusion and one limitation;
do not claim success from aggregate mAP alone.}
\bibliographystyle{IEEEbib}
\bibliography{references}