wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update manuscript results section

Browse Source
This commit is contained in:
Xin Wang
2026-09-14 14:23:37 +08:00
parent f16107d38b
commit f14026e82f
1 changed files with 115 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/main/main.tex
+115 -39
View File
@@ -223,7 +223,7 @@ its label/input boundary is fully reported.}
We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.}
\subsection{Model Configuration and Training}
\subsection{Model Configuration and Training}
Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.}
@@ -256,26 +256,29 @@ are reported as complementary measures.
\section{Results and Analysis}
\label{sec:results}
\subsection{Baselines}
The confirmatory comparison contains: an EgoSpeak reproduction; the frozen
Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local
audio; a parameter- and history-matched temporal local branch; and the proposed
audio--visible-person adapter. Qwen2.5-Omni-7B is a capacity-scaling control,
not a second paper storyline. \todo{Freeze all baseline implementations and
ensure the EgoSpeak comparison uses identical targets, split, causal validity,
and evaluation.}
\subsection{Main Comparison}
Table~\ref{tab:main} compares the proposed causal local audiovisual adapter
with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$,
matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched
temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling
control. All methods use the same split, causal inputs, targets, and evaluation
protocol. Event thresholds are selected on training-side calibration data and
frozen before test evaluation. \todo{Freeze all baseline implementations and
verify identical targets, causal validity, and evaluation.}
\begin{table}[t]
\caption{Confirmatory EasyCom results. Lower multi-other false entries are
better; all event rows use a training-side selected operating point.}
\caption{Confirmatory EasyCom results. Higher mAP and yield recall are better;
lower false entries and onset latency are better. FP/min is measured at
matched valid-onset recall $R=0.8$.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\hline
Method & 0.2-s mAP & Mean mAP & FP/min@$R=.8$ & Onset lat. & Yield R\\
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ & FP/min $\downarrow$
& Onset lat. $\downarrow$ & Yield R $\uparrow$\\
\hline
EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
@@ -288,33 +291,75 @@ Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \t
\end{tabular}}
\end{table}
\textbf{Information opportunity.}
On development Session~12, an analysis-only seed-0 oracle supplied perfectly
timed visible-other speech activity through an explicit non-deployable path.
Aggregate visible activity improved 0.2-s mAP by 6.97 points and ten-horizon
mean mAP by 2.91 points over matched global-plus-local-audio. Visibility-only
and one-second-delayed activity remained near audio, while per-person histories
did not outperform identity-shuffled histories. This motivates timely local
floor evidence, not participant identity. \todo{After seeds 1--2, report the
interval or remove the exact oracle numbers from the four-page paper.}
\textbf{Dense forecasting.}
The proposed local AV adapter improves 0.2-s mAP from
\todo{$X$ to $Y$} over the strongest matched baseline and improves
mean mAP by \todo{$Z$ points}.
The gain is largest at \todo{short prediction horizons / transition-adjacent
frames / specify actual observation}, indicating that local audiovisual
evidence primarily improves \todo{the interpretation supported by the data}.
\textbf{Main result.}
Table~\ref{tab:main} should establish one result: \todo{the selected local
adapter reduces multi-other false entry at matched valid-onset recall, retains
onset latency/yield behavior, improves dense forecasting, and recovers a
meaningful fraction of the oracle opportunity}. The 7B row should determine
whether \todo{capacity alone closes the gap or the adaptation structure remains
useful at larger scale}. No conclusion is drawn across all Omni families from
Qwen-only evidence.
\textbf{Floor-entry behavior.}
At matched valid-onset recall $R=0.8$, the proposed adapter reduces
multi-other false entries from \todo{$X$} to \todo{$Y$} FP/min
(\todo{$Z\%$ relative reduction}).
This reduction is obtained while \todo{maintaining / improving} onset latency
and yield recall.
Thus, the improvement cannot be explained solely by a more conservative
decision threshold.
\textbf{Comparison with adaptation baselines.}
Matched PEFT yields \todo{result}, whereas adding recent local audio alone
yields \todo{result}.
The parameter- and history-matched temporal branch achieves
\todo{result}.
The remaining gap between this control and the proposed adapter therefore
suggests that \todo{correctly timed audiovisual local evidence contributes
beyond additional temporal capacity}.
\todo{If this difference is not statistically supported, state instead that
the local temporal branch accounts for most of the gain.}
\subsection{Information Opportunity}
Before designing the deployable adapter, we measured whether local floor
information was useful at all using an analysis-only oracle.
The oracle receives perfectly timed visible-other speech activity through a
non-deployable pathway and is never used in the main comparison.
On development Session~12, aggregate visible activity improved 0.2-s mAP by
6.97 points and ten-horizon mean mAP by 2.91 points over the matched
global-plus-local-audio baseline.
In contrast, visibility alone and one-second-delayed activity remained close
to the audio baseline, while person-specific histories did not outperform
identity-shuffled histories.
These controls indicate that the useful signal is primarily
\emph{timely local floor evidence}, rather than participant identity or
static visibility.
The proposed adapter is therefore designed to approximate this information
using only causally available audiovisual observations.
\todo{After completing all seeds, report mean $\pm$ confidence interval here.
If the oracle effect is unstable, describe it qualitatively and omit the exact
numbers from the four-page version.}
\subsection{Mechanism Analysis}
To determine whether the gain arises from correctly aligned visual floor
evidence rather than additional parameters or generic visual context, we
evaluate the controls in Table~\ref{tab:ablation}.
\begin{table}[t]
\caption{Mechanism controls for the selected Omni-3B local adapter.}
\caption{Mechanism controls for the selected Omni-3B local adapter. All
variants use the same prediction head and comparable temporal capacity.}
\label{tab:ablation}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Local condition & Mean mAP & FP/min@$R=.8$ & Yield R\\
Local condition & Mean mAP $\uparrow$ & FP/min@$R=.8$ $\downarrow$
& Yield R $\uparrow$\\
\hline
Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\
Remove local vision & \todo{--} & \todo{--} & \todo{--}\\
@@ -322,17 +367,48 @@ Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\
Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\
Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\
\hline
\end{tabular}
\end{tabular}}
\end{table}
\textbf{Mechanism and limitations.}
Table~\ref{tab:ablation} should test whether gains require correctly timed local
visual evidence rather than extra parameters, regularization, or generic scene
context. \todo{Report paired intervals and the supported interpretation. If
correct and shifted/removed vision do not separate, drop the audiovisual
Removing local vision changes \todo{metric} by \todo{$X$}, while temporally
shifting the visual stream by \todo{$\Delta t$} changes it by \todo{$Y$}.
Replacing person-local visual evidence with a global/full-frame visual pool
produces \todo{result}.
Together, these comparisons \todo{support / do not support} the hypothesis
that the benefit depends specifically on correctly timed local visual
evidence.
The same-rate/capacity control achieves \todo{result}, showing that
\todo{the gain cannot / can largely} be attributed to additional temporal
parameters or higher-rate processing alone.
\todo{Report paired intervals and the supported interpretation. If correct
and shifted/removed vision do not separate, drop the audiovisual
evidence-routing claim. If dense mAP improves but event behavior does not, do
not claim improved floor control. If legitimate onset/yield degrades, stop the
adapter rather than describing lower false entry as success.}
not claim improved floor control. If legitimate onset or yield degrades, do
not describe lower false entry as success.}
\subsection{Scaling Control}
Increasing the frozen Omni backbone from 3B to 7B yields
\todo{$X$ points} in 0.2-s mAP and \todo{$Y$} FP/min at matched recall.
The proposed 3B adapter \todo{remains better than / approaches / is surpassed
by} the 7B scaling control.
This result suggests that \todo{architectural access to local causal evidence
provides a complementary benefit to backbone scaling}.
Because both backbones belong to the Qwen2.5-Omni family, we do not claim that
this conclusion generalizes to all omni-modal models.
\subsection{Limitations and Qualitative Analysis}
The results support \todo{precise supported claim}, but several limitations
remain.
First, \todo{EasyCom size/session diversity limitation}.
Second, the adapter relies on \todo{state the actual visible-person box or
tracking assumption}.
Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor
exchange, etc.} remains challenging.
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
one multi-other false-entry improvement paired with one preserved real wearer