wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Align manuscript with EOT INT evaluation

Browse Source
This commit is contained in:
Xin Wang
2026-09-14 15:44:01 +08:00
parent f14026e82f
commit 7292fea7fb
3 changed files with 122 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3
View File
@@ -15,5 +15,8 @@ __pycache__/
# Local editors and environments.
.idea/
.vscode/
!.vscode/
.vscode/*
!.vscode/settings.json
.venv/
venv/
.vscode/settings.json
Vendored
+3
View File
@@ -0,0 +1,3 @@
{
"latex-workshop.latex.outDir": "%DIR%/build"
}
manuscript/main/main.tex
+114 -79
View File
@@ -33,9 +33,9 @@ not explicitly trained to expose sub-second, wearer-relative floor evidence.
We therefore study a candidate causal local floor adapter that combines the global
Omni state with short-window mixed audio and an anonymous set of visible-person
observations, without assuming persistent speaker identity. On EasyCom,
\todo{report the final three-seed dense and competing-risk-censored event
results: false-entry reduction at matched wearer-onset recall, onset latency,
yield behavior, confidence intervals, and oracle-gap recovery}. Controlled
\todo{report the final three-seed dense and committed-event
results: EOT-response and INT-yield recall, false-positive rate, latency,
confidence intervals, and oracle-gap recovery}. Controlled
\todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
show that \todo{state the supported mechanism, without claiming addressee or
speaker identity}. These results demonstrate \todo{the narrow conclusion
@@ -102,9 +102,9 @@ corresponding TODO gates pass:
\item We evaluate a wearer-relative causal local floor adapter that combines
long-context Omni information with short-window audio and anonymous
visible-person evidence.
\item We evaluate whether the adapter reduces false wearer entries during
multi-other exchanges at matched valid-onset recall, using causal alignment
interventions and competing-risk-censored entry/yield analysis.
\item We evaluate whether the adapter reduces erroneous wearer commits
during multiparty floor transfers under an EOT/INT event protocol with
explicitly annotated hard-negative regions.
\end{itemize}
\begin{figure}[t]
@@ -112,7 +112,7 @@ corresponding TODO gates pass:
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
\todo{Draw only the selected method: causal AV to the long-context Omni path;
the short-window mixed-audio and anonymous person-set path; residual fusion;
the common $10\times3$ head; and the wearer-silent/active event interpretation.
the common $10\times3$ head; and the separate EOT-response/INT-yield evaluation.
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
\caption{Long-context Omni understanding and short-window local floor evidence
@@ -136,13 +136,13 @@ Wearer--other overlap is allowed. The model outputs
\in[0,1]^{K\times3}.
\label{eq:task}
\end{equation}
The same wearer-activity forecast has two controller interpretations. In a
wearer-silent opportunity, $\hat y^{\mathrm w}_{t,1}$ is an entry score. While
the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ is a near-term yield score.
The current self-state is used to define these observable event strata, not as
an unreported oracle model input. \todo{If the final adapter explicitly uses
self-state, define its strictly causal and deployment-available source and add
a matched no-state ablation; otherwise keep self-state evaluation-only.}
The wearer-activity forecast also provides a causal control score. When the
wearer is silent, $\hat y^{\mathrm w}_{t,1}$ acts as a near-term take-turn
score; while the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ acts as a
yield score. These scores are trained only from future activity targets.
Committed EOT-response and INT-yield events used for behavioral evaluation are
defined separately from participant-resolved annotations in
Sec.~\ref{sec:experiments}.
\subsection{Global Omni baseline}
@@ -216,41 +216,80 @@ per-person speech label enters the deployable model. \todo{Add an active-speaker
or event auxiliary loss only if it improves the frozen primary protocol and
its label/input boundary is fully reported.}
\section{Experiments}
\label{sec:experiments}
\subsection{Dataset and Data Preparation}
We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.}
We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 as
the training/development pool and Sessions 1--3 only for final evaluation
\cite{donley2021easycom}. The participant-resolved speech annotations are
converted into two parallel representations. First, they are downsampled to
5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background,
camera-wearer, and aggregate non-wearer activity. Second, the original
participant timeline is retained to derive committed interaction events for
behavioral evaluation.
We consider two wearer-relative events. An \emph{EOT-response} denotes a
non-wearer floor release followed by the wearer taking the floor, whereas an
\emph{INT-yield} denotes a non-wearer takeover while the wearer is speaking,
followed by the wearer relinquishing the floor. Candidate events are generated
automatically from participant-level onset/offset annotations. Mid-turn pauses,
other-to-other handovers, backchannels, and non-takeover overlaps are retained
as hard-negative regions rather than positive events. Annotation-corrupted or
unresolvable boundaries are excluded from scoring. Ambiguous candidates are
manually adjudicated.
Participant identity is used only to construct these evaluation annotations
and is never provided to the prediction model. Provided person boxes are used
only for the causal visible-person input; transcripts, target-of-speech labels,
close-microphone audio, and future observations are unavailable to the model.
\todo{Report duration and event statistics: EOT positives/hard negatives,
INT positives/hard negatives, excluded candidates, and manual-adjudication
count for train/development and test. Freeze the event-construction and
adjudication protocol before reporting semantic EOT/INT results; until then,
retain the timing-derived proxy terminology.}
\subsection{Model Configuration and Training}
Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.}
Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout,
matched PEFT/LoRA adaptation, global features with recent local audio, a
parameter- and history-matched temporal local branch, and the proposed causal
local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling
control. All learned variants use identical causal examples, prediction
horizons, prediction heads, and the class-balanced binary cross-entropy
objective in Eq.~\eqref{eq:loss}.
\subsection{Evaluation Metrics}
Model selection is performed using training-side data only. Event operating
points are selected from session-level out-of-fold predictions over
Sessions~4--12; the final model is then retrained on all nine sessions before
evaluation on Sessions~1--3. Final results are averaged over three random
seeds.
\todo{Freeze context length, local window/rates, optimizer, epochs, LoRA
configuration, trainable parameter counts, and inference latency.}
We evaluate both dense future-activity prediction and event-level floor
control. For comparability with EgoSpeak, we report per-frame AP at
$0.2$--$2.0$~s horizons and their mean mAP.
\subsection{Evaluation Protocol}
For event-level evaluation, a \emph{wearer entry} is a transition from wearer
silence to speech, while a \emph{wearer yield} is a transition from speech to
silence. An entry prediction is considered correct if it is committed within
the predefined response window of a ground-truth wearer onset. We report entry
recall, false entries per minute of eligible wearer-silent time, and signed
onset latency. Yield events are evaluated analogously using recall and latency.
We first retain dense future-activity evaluation for direct comparison with
EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.
Multiparty interaction introduces a competing-risk condition. After another
speaker releases the floor, the current wearer-entry opportunity is terminated
if another non-wearer participant starts speaking before the wearer does.
Predictions after this competing onset are therefore not matched to the
previous opportunity. This prevents an other-to-other handover from being
treated as an open floor for the wearer.
Behavioral evaluation follows a committed-event protocol. For both
EOT-response and INT-yield, each gold event is matched to at most one committed
prediction and is scored by recall and signed detection latency. A prediction
inside a task-specific hard-negative region is counted as a false positive.
EOT hard negatives include mid-turn continuations and competing
other-to-other floor transfers; INT hard negatives include backchannels and
overlaps in which the wearer retains the floor. Regions excluded during
adjudication do not contribute to either the positive or negative denominator.
Operating thresholds are selected on the development split and frozen for test
evaluation. Our primary behavioral metric is false wearer entries per minute at
a matched wearer-entry recall of $0.8$; dense mAP and wearer-yield performance
are reported as complementary measures.
We report the recall--false-positive trade-off and a common operating point at
matched recall $R=0.8$. The primary behavioral quantities are therefore EOT
false-positive rate, INT false-positive rate, and signed latency at the frozen
operating point. We additionally report performance on the other-to-other
subset to test whether local visual evidence specifically reduces erroneous
wearer turn-taking during multiparty floor transfers. \todo{Freeze the FPR
denominators and report signed median latency separately for EOT and INT.}
\section{Results and Analysis}
@@ -268,25 +307,25 @@ frozen before test evaluation. \todo{Freeze all baseline implementations and
verify identical targets, causal validity, and evaluation.}
\begin{table}[t]
\caption{Confirmatory EasyCom results. Higher mAP and yield recall are better;
lower false entries and onset latency are better. FP/min is measured at
matched valid-onset recall $R=0.8$.}
\caption{Confirmatory EasyCom results. EOT and INT false-positive rates are
measured at matched recall $R=0.8$. E/I Lat. gives signed median EOT/INT
latency in seconds; values closer to zero are better.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ & FP/min $\downarrow$
& Onset lat. $\downarrow$ & Yield R $\uparrow$\\
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
& EOT FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I Lat.\\
\hline
EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
\hline
\end{tabular}}
\end{table}
@@ -299,14 +338,13 @@ The gain is largest at \todo{short prediction horizons / transition-adjacent
frames / specify actual observation}, indicating that local audiovisual
evidence primarily improves \todo{the interpretation supported by the data}.
\textbf{Floor-entry behavior.}
At matched valid-onset recall $R=0.8$, the proposed adapter reduces
multi-other false entries from \todo{$X$} to \todo{$Y$} FP/min
(\todo{$Z\%$ relative reduction}).
This reduction is obtained while \todo{maintaining / improving} onset latency
and yield recall.
Thus, the improvement cannot be explained solely by a more conservative
decision threshold.
\textbf{Committed behavior.}
At matched EOT-response recall $R=0.8$, the proposed adapter reduces EOT FPR
from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR
from \todo{$A$} to \todo{$B$}. Its signed median EOT/INT latencies are
\todo{$+0.XX/+0.XX$~s}. These recall-matched error rates and signed latencies
test whether fewer erroneous commits arise from better floor decisions rather
than indefinite waiting.
\textbf{Comparison with adaptation baselines.}
Matched PEFT yields \todo{result}, whereas adding recent local audio alone
@@ -351,36 +389,32 @@ evaluate the controls in Table~\ref{tab:ablation}.
\begin{table}[t]
\caption{Mechanism controls for the selected Omni-3B local adapter. All
variants use the same prediction head and comparable temporal capacity.}
variants use the same prediction head and comparable temporal capacity; FPRs
are measured at matched recall $R=0.8$.}
\label{tab:ablation}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\begin{tabular}{lcccc}
\hline
Local condition & Mean mAP $\uparrow$ & FP/min@$R=.8$ $\downarrow$
& Yield R $\uparrow$\\
Local condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$
& O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\
\hline
Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\
Remove local vision & \todo{--} & \todo{--} & \todo{--}\\
Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\
Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\
Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\
Remove vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Time-shift vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Full-frame vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Correct local AV & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}}
\end{table}
Removing local vision changes \todo{metric} by \todo{$X$}, while temporally
shifting the visual stream by \todo{$\Delta t$} changes it by \todo{$Y$}.
Replacing person-local visual evidence with a global/full-frame visual pool
produces \todo{result}.
Together, these comparisons \todo{support / do not support} the hypothesis
that the benefit depends specifically on correctly timed local visual
evidence.
The same-rate/capacity control achieves \todo{result}, showing that
\todo{the gain cannot / can largely} be attributed to additional temporal
parameters or higher-rate processing alone.
Removing local vision changes \todo{EOT/O$\rightarrow$O/INT FPR by $X/Y/Z$},
while shifting the visual stream by \todo{$\Delta t$} produces \todo{result}.
Replacing person-local evidence with full-frame vision produces \todo{result}.
If aligned local AV improves estimates of surrounding floor occupancy, its
largest selective reduction should appear on the O$\rightarrow$O hard-negative
subset. Together, these comparisons \todo{support / do not support} that
mechanism beyond generic visual context.
\todo{Report paired intervals and the supported interpretation. If correct
and shifted/removed vision do not separate, drop the audiovisual
@@ -391,7 +425,8 @@ not describe lower false entry as success.}
\subsection{Scaling Control}
Increasing the frozen Omni backbone from 3B to 7B yields
\todo{$X$ points} in 0.2-s mAP and \todo{$Y$} FP/min at matched recall.
\todo{$X$ points} in 0.2-s mAP and \todo{EOT/INT FPR of $Y/Z$} at matched
recall.
The proposed 3B adapter \todo{remains better than / approaches / is surpassed
by} the 7B scaling control.
@@ -422,8 +457,8 @@ This study asks whether a general-purpose Omni representation can be adapted
for causal floor forecasting of an invisible egocentric wearer. The candidate
method separates long-context Omni understanding from short-window local floor
evidence without assuming persistent speaker identity. \todo{State the final
learned-method improvement, matched-recall false-entry result, onset/yield
non-degradation, causal-control evidence, and main limitation. If no learned
learned-method improvement, matched-recall EOT/INT false-positive rates,
signed latency, causal-control evidence, and main limitation. If no learned
variant passes these tests, reframe the submission as an information audit or
do not make the method claim.}