icassp2027-experiment/manuscript/main/main.tex

% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled
% in this repository. Replace the class/style files when the 2027 author kit
% is released, and remove every visible \todo{} marker before submission.
% Compile from this directory with: make
\documentclass{article}
\usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}

\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
\newcommand{\method}{WearerSet}
\newcommand{\vect}[1]{\boldsymbol{#1}}

\title{NOT EVERY OPEN FLOOR IS MINE: ADDRESSEE-CONDITIONED FLOOR-ENTRY
PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}

% TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized.
\name{Internal Manuscript Draft}
\address{}

\begin{document}
\ninept
\maketitle

\begin{abstract}
An egocentric conversational agent must decide not only when a speaking
opportunity arises, but also whether that opportunity belongs to it. Existing
speech-activity projection collapses all non-wearer speakers, allowing strong
frame-level accuracy to coexist with intrusive entries during conversations
between bystanders. We formulate addressee-conditioned floor-entry projection:
dense causal forecasts of wearer and non-wearer activity at 200-ms resolution,
conditioned on a variable-cardinality set of visible participants. Our model
augments a frozen streaming omni-model with person-aligned audio--visual
representations and permutation-invariant role queries for floor state,
wearer address, and third-party interaction. We further introduce an EasyCom
event protocol that separates addressed-to-wearer responses from third-party
exchanges and reports false entries at matched response recall and decision
latency. Correct-, wrong-person, time-shifted, and removed-video controls test
whether a gain follows social grounding rather than model capacity.
\todo{Replace this sentence with the main result: FP/min reduction at 80\%
response recall, response latency/recall non-degradation, and 95\% confidence
interval.}
\end{abstract}

\begin{keywords}
turn-taking, speech initiation, addressee detection, egocentric video,
audio-visual learning
\end{keywords}

\section{Introduction}
\label{sec:intro}

Conversational agents must continuously decide whether to speak or keep
listening. In a dyadic exchange, a pause or an apparent end of turn is often
treated as an invitation to respond. This assumption fails in multiparty
interaction: one participant may finish an utterance addressed to another
participant, and an egocentric agent that reacts only to acoustic availability
will enter a floor that was never offered to it. Such an error is socially more
disruptive than a small framewise prediction error. A useful agent must
therefore distinguish \emph{when the floor may change} from \emph{whether the
change is directed to the camera wearer}.

Voice Activity Projection (VAP) models future joint speech activity and
derives turn-taking events without manually labeling each event
\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
untrimmed egocentric video, forecasting wearer and aggregate non-wearer
activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
output is attractive for a wearable agent, but an ``others'' channel cannot
distinguish an utterance addressed to the wearer from an exchange between
bystanders. Conventional per-frame average precision can consequently be
dominated by ordinary activity while hiding precisely the false-entry behavior
that visual social cues should prevent. Event-oriented evaluation likewise
emphasizes recall, false alarms, and latency rather than a single frame-level
score \cite{jiang2026turnbench}.

Related work supplies complementary parts of the problem. Egocentric
talking-to-me detection predicts whether a completed utterance addresses the
camera wearer and benefits from long-term social context
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
The former is generally utterance-level rather than continuous anticipation;
the latter does not ask whether an invisible camera wearer should enter. Our
gap is therefore narrower than ``audio--visual multiparty turn taking'': we
study continuous, wearer-relative floor action when the candidate responder is
behind the camera and the number of visible participants changes over time.

We call this task \emph{addressee-conditioned egocentric floor-entry
projection}. A global streaming audio--video representation supplies broad
context, while a set branch preserves individual visual histories and their
alignment to recent mixed audio. Fixed semantic role queries compress a
variable participant set into wearer-address and third-party evidence without
assuming a fixed seating order. EasyCom is suitable because it provides
synchronized egocentric audio--video, per-person tracks, voice activity, and
target-of-speech annotations \cite{donley2021easycom}. Target annotations are
used only for training-side supervision and evaluation strata, never as input
at test time.

Our contributions are threefold. (1) We formulate continuous speech
initiation as recipient-aware floor-entry projection for an invisible camera
wearer. (2) We propose a causal set-based fusion module that grounds mixed
audio in a variable number of visible participants and preserves the original
dense EgoSpeak target. (3) We introduce an event evaluation that reports
third-party false entries at matched addressed-response recall, accompanied by
latency and visual counterfactual controls. This design makes the scientific
claim falsifiable: visual fusion is useful only if \emph{correctly aligned}
vision improves the operating trade-off without suppressing legitimate wearer
responses.

\begin{figure}[t]
\centering
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
\todo{Draw the final method figure: causal omni state $\vect g_t$; recent
mixed-audio tokens; a masked set of face-track histories; shared per-person
audio--visual grounding; three role queries; zero-initialized residual; and
the $10\times3$ activity head. Mark all causal cutoffs and masks.}}}
\caption{Proposed wearer-relative set fusion. The number and ordering of
visible participants may change, while the output remains fixed-size.}
\label{fig:method}
\end{figure}

\section{Task and Method}
\label{sec:method}

\subsection{Wearer-relative activity projection}

At decision time $t$, the model observes only the causal stream
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s.
For bin $k$, the target
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed; hence this is not a three-way softmax. The
output is
\begin{equation}
  \hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3},
  \qquad q_t=\hat y^{\mathrm{w}}_{t,1},
  \label{eq:task}
\end{equation}
where $q_t$ is the immediate wearer-entry score used by the event protocol.
It is interpreted as an \emph{entry} score only in wearer-silent opportunity
windows; elsewhere it remains a future-activity probability. This retains the
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
signal.

\subsection{Frozen global stream and local participant set}

We obtain a causal global state $\vect{g}_t$ from the streaming Thinker of
Qwen2.5-Omni \cite{xu2025qwenomni}. The backbone is frozen in the main
experiment because preliminary LoRA adaptation caused negative transfer
(Sec.~\ref{sec:results}). From the most recent $L_a$ seconds, a shared audio
encoder produces token sequence $\vect{A}_t$. For each visible track
$i\in\{1,\ldots,N_t\}$, a shared temporal visual encoder produces
$\vect{v}_{t,i}$ from its recent face history. Person-aligned evidence is
\begin{equation}
 \vect{p}_{t,i}=\vect{v}_{t,i}+
 \mathrm{CA}(\vect{v}_{t,i},\vect{A}_t,\vect{A}_t),
 \label{eq:person}
\end{equation}
so every participant queries the same mixed audio rather than requiring
unavailable separated microphones. A masked set encoder models interactions
among $\mathcal{P}_t=\{\vect{p}_{t,i}\}_{i=1}^{N_t}$ and is permutation
equivariant \cite{lee2019settransformer}. Missing detections use a learned
\textsc{offscreen/null} element. In minibatches, sets are padded to
$N_{\max}$ with an attention mask; no slot-ID embedding is used, and track
slots are randomly permuted during training.

\todo{Implement and verify the shared track encoder, mixed-audio
cross-attention, OFFSCREEN token, padding mask, and numerical invariance under
track-slot permutation. Freeze $L_a$, visual crop history, $N_{\max}$, and
face-track confidence filtering from train-side experiments only.}

\subsection{Role-query fusion}

Mean pooling can erase which participant is speaking to whom. Instead, three
learned semantic queries represent floor state, address-to-wearer evidence,
and third-party interaction. Each query is conditioned on $\vect{g}_t$ and
cross-attends to the participant set:
\begin{align}
 \vect{Q}_t &= [\vect{q}_{\mathrm{floor}};
               \vect{q}_{\mathrm{addr}};
               \vect{q}_{\mathrm{third}}]+W_g\vect{g}_t,\\
 \vect{R}_t &= \mathrm{MHA}(\vect{Q}_t,\widetilde{\mathcal P}_t,
                            \widetilde{\mathcal P}_t),\\
 \vect{z}_t &= \vect{g}_t+\tanh(\alpha)W_r\mathrm{vec}(\vect{R}_t),
 \label{eq:fusion}
\end{align}
where $\alpha$ is initialized to zero. The zero-initialized residual makes the
starting model identical to the frozen global baseline and allows the visual
branch to be removed exactly. A two-layer prediction head maps $\vect{z}_t$ to
Eq.~\eqref{eq:task}.

\subsection{Training objectives}

The main loss is class-balanced binary cross entropy over all horizons and
activity channels. When annotations are available, auxiliary heads predict
per-track active-speaker state and whether the current source addresses the
wearer:
\begin{equation}
 \mathcal{L}=\mathcal{L}_{\mathrm{act}}+
 \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}+
 \lambda_{\mathrm{addr}}\mathcal{L}_{\mathrm{addr}}.
 \label{eq:loss}
\end{equation}
Ambiguous, group-addressed, and offscreen-target samples are masked from
$\mathcal{L}_{\mathrm{addr}}$, but remain in activity training. No language
model or pseudo-label is used to define primary test events.

\todo{Run the staged ablation: activity only; +ASD; +addressee. Select
$\lambda$ values and early stopping on a training-side calibration fold. Do
not inspect Sessions 1--3 during selection.}

\section{Experimental Setup}
\label{sec:setup}

\textbf{Data and splits.}
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
and hyperparameter choices use training-side folds only. Inputs contain the
camera stream and shared array audio available to the wearer; close-talking
microphones and target-of-speech labels are supervision only. We also report aggregate
activity forecasting on the EgoSpeak Ego4D split \cite{grauman2022ego4d}, but
do not claim addressee grounding there because comparable target annotations
are absent.

\textbf{Event construction.}
An \emph{addressed response} starts from a pure other-to-wearer utterance and
requires a wearer onset before an unrelated other onset. A \emph{third-party
exchange} starts from a pure other-to-other target while the wearer remains
silent. Group, none, multiple, and unknown targets form a separate diagnostic
stratum. Threshold $\tau_r$ is selected on the development split to obtain
addressed-response recall $r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary
metric is false wearer entries per minute of third-party opportunity at
matched recall; the headline operating point is $r=0.8$. We also report
addressed-response recall, signed detection latency, wearer yield recall, and
the original ten-horizon per-frame mAP.

\todo{Freeze the event pairing and competing-risk rules, including pre/post
windows, minimum silence, maximum response delay, overlap handling, and one
detection per event. Publish counts for train/test positive and negative
events, visibility, duration, and excluded target types. Treat all previously
viewed Sessions 1--3 scores as exploratory; lock the final protocol and model
before the confirmatory three-seed run.}

\textbf{Baselines and controls.}
We compare: global omni state (G); G plus recent audio (G+A); G+A plus pooled
visual context (G+A+V-pool); a parameter-matched temporal branch without
person identity; and the proposed set branch. Required counterfactuals replace
correct tracks with (i) wrong participants from the same scene, (ii)
time-shifted tracks, and (iii) zeros. These controls share parameter count,
audio history, and optimization budget.

\textbf{Implementation and statistics.}
All models run at 5~Hz with an 8-s causal memory. The 3B Thinker is frozen and
local audio/face histories are encoded causally. We report three seeds and
mean$\pm$standard deviation. Confidence intervals for event metrics use paired
cluster bootstrap over sessions/conversation blocks rather than treating
adjacent frames as independent. Runtime includes P50/P95 decision latency,
peak memory, and trainable parameter count.

\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
count, hardware, and real-time latency after the architecture is frozen.}

\section{Results and Analysis}
\label{sec:results}

\begin{table}[t]
\caption{Current aggregate evidence on EasyCom test Sessions 1--3. Values are
percentages from one seed and are motivation, not the final main result.}
\label{tab:preliminary}
\centering
\small
\begin{tabular}{lccc}
\hline
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
\hline
G+A             & 81.72 & 85.07 & 65.11\\
G+A+V-pool      & 82.03 & 85.09 & 65.40\\
\hline
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
\hline
\end{tabular}
\end{table}

Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
Pooled vision gives a small gain, while immediate wearer AP is nearly
unchanged. This is compatible with either weak but useful social grounding or
generic visual regularization; only matched-recall event evaluation and
counterfactual identity controls can distinguish them.

\begin{table}[t]
\caption{Primary EasyCom event evaluation. Lower $\mathrm{FP}_{\mathrm{3p}}$
is better; all rows use the same development-selected operating protocol.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Model & $\mathrm{FP}_{\mathrm{3p}}$/min@$R=.8$ & $R_{\mathrm{addr}}$ &
Latency & Yield R & Mean mAP\\
\hline
G                 & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
G+A               & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
G+A+V-pool        & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Temporal control  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\method{}         & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\method{}+ASD     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\method{}+Addr    & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}

\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
seeds, plus 95\% paired cluster-bootstrap confidence intervals for the
proposed-vs-G+A difference. The paper's main claim is supported only if
third-party FP/min decreases at matched recall without worse response latency
or wearer-yield recall.}

\begin{table}[t]
\caption{Visual grounding counterfactual on the frozen best model.}
\label{tab:counterfactual}
\centering
\small
\begin{tabular}{lccc}
\hline
Visual evidence & $\mathrm{FP}_{\mathrm{3p}}$/min & Addr. recall & 0.2-s mAP\\
\hline
Correct tracks    & \todo{--} & \todo{--} & \todo{--}\\
Wrong person      & \todo{--} & \todo{--} & \todo{--}\\
Time shifted      & \todo{--} & \todo{--} & \todo{--}\\
Removed           & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}
\end{table}

Table~\ref{tab:counterfactual} is the identity test. A true social-grounding
effect should be largest for correct tracks and should collapse for wrong or
time-shifted people. If every visual variant improves equally, the defensible
conclusion is additional capacity/history, not addressee grounding.

\todo{Add one compact error analysis split by target visibility, number of
visible people, overlap, and backchannel/interruption proxy. Include event
counts with every rate. Move negative LoRA/7B/ROMA diagnostics to supplementary
material unless they directly explain the final method.}

\section{Conclusion}
\label{sec:conclusion}

We study when an egocentric agent should enter a multiparty floor, not merely
whether someone will speak. The proposed formulation retains dense future
activity prediction while grounding wearer entry in a variable set of visible
participants. Its event protocol tests the intended behavior directly:
reducing intrusions into third-party exchanges at matched legitimate-response
recall. \todo{Replace with one quantitative conclusion and one limitation;
do not claim success from aggregate mAP alone.}

\bibliographystyle{IEEEbib}
\bibliography{references}

\end{document}