Update manuscript
This commit is contained in:
@@ -1,8 +1,4 @@
|
||||
LATEXMK ?= latexmk
|
||||
TEMPLATE_DIR := $(abspath ../ICASSP2026_Paper_Templates)
|
||||
|
||||
export TEXINPUTS := $(TEMPLATE_DIR):
|
||||
export BSTINPUTS := $(TEMPLATE_DIR):
|
||||
|
||||
.PHONY: all clean
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
This is the editable paper directory. It uses the unmodified official ICASSP
|
||||
2026 LaTeX assets stored in `../ICASSP2026_Paper_Templates/`. When the ICASSP
|
||||
2027 author kit is released, replace or add the official template and update
|
||||
`TEMPLATE_DIR` in the Makefile only after checking the new instructions.
|
||||
`template_dir` in `latexmkrc` only after checking the new instructions.
|
||||
|
||||
Compile from this directory:
|
||||
|
||||
@@ -14,12 +14,19 @@ make
|
||||
The PDF and all intermediate files are written to `build/`; only source files
|
||||
belong in Git.
|
||||
|
||||
VS Code's LaTeX Workshop extension is supported when the repository is opened
|
||||
as the workspace. Open `main.tex` and run **LaTeX Workshop: Build LaTeX
|
||||
project**. The workspace setting writes output to `build/`, while `latexmkrc`
|
||||
provides the template and bibliography-style search paths to both LaTeX
|
||||
Workshop and command-line builds.
|
||||
|
||||
Current source layout:
|
||||
|
||||
```text
|
||||
main/
|
||||
main.tex
|
||||
references.bib
|
||||
latexmkrc
|
||||
Makefile
|
||||
README.md
|
||||
build/ # generated, ignored
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
my $path_separator = $^O eq 'MSWin32' ? ';' : ':';
|
||||
my $template_dir = '../ICASSP2026_Paper_Templates';
|
||||
|
||||
$ENV{'TEXINPUTS'} =
|
||||
$template_dir . $path_separator . ($ENV{'TEXINPUTS'} // '');
|
||||
$ENV{'BSTINPUTS'} =
|
||||
$template_dir . $path_separator . ($ENV{'BSTINPUTS'} // '');
|
||||
|
||||
$out_dir = 'build';
|
||||
+40
-46
@@ -169,7 +169,7 @@ $\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
|
||||
a shared visual encoder and masked set operator produce
|
||||
\begin{equation}
|
||||
\vect v_\tau=\mathrm{SetPool}
|
||||
\left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})}_{i=1}^{N_\tau}
|
||||
\left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}
|
||||
\cup\{\vect v_\varnothing\}\right),
|
||||
\label{eq:set}
|
||||
\end{equation}
|
||||
@@ -216,19 +216,47 @@ per-person speech label enters the deployable model. \todo{Add an active-speaker
|
||||
or event auxiliary loss only if it improves the frozen primary protocol and
|
||||
its label/input boundary is fully reported.}
|
||||
|
||||
\section{Experiments}
|
||||
\label{sec:experiments}
|
||||
\section{Experiments}
|
||||
\label{sec:experiments}
|
||||
|
||||
\textbf{Data and split.}
|
||||
We follow the EgoSpeak EasyCom split: Sessions 4--12 for training-side model
|
||||
development and Sessions 1--3 for final reporting \cite{donley2021easycom}.
|
||||
Thresholds, windows, checkpoints, loss weights, and early stopping are selected
|
||||
without using the official test sessions. Already observed test results are
|
||||
treated as exploratory internal evidence and are not included in the paper.
|
||||
\todo{Report duration, frame/event counts, class prevalence, visibility and
|
||||
box coverage, source discontinuities, and the frozen calibration fold.}
|
||||
\subsection{Dataset and Data Preparation}
|
||||
|
||||
\textbf{Baselines.}
|
||||
We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.}
|
||||
|
||||
\subsection{Model Configuration and Training}
|
||||
|
||||
Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.}
|
||||
|
||||
\subsection{Evaluation Metrics}
|
||||
|
||||
We evaluate both dense future-activity prediction and event-level floor
|
||||
control. For comparability with EgoSpeak, we report per-frame AP at
|
||||
$0.2$--$2.0$~s horizons and their mean mAP.
|
||||
|
||||
For event-level evaluation, a \emph{wearer entry} is a transition from wearer
|
||||
silence to speech, while a \emph{wearer yield} is a transition from speech to
|
||||
silence. An entry prediction is considered correct if it is committed within
|
||||
the predefined response window of a ground-truth wearer onset. We report entry
|
||||
recall, false entries per minute of eligible wearer-silent time, and signed
|
||||
onset latency. Yield events are evaluated analogously using recall and latency.
|
||||
|
||||
Multiparty interaction introduces a competing-risk condition. After another
|
||||
speaker releases the floor, the current wearer-entry opportunity is terminated
|
||||
if another non-wearer participant starts speaking before the wearer does.
|
||||
Predictions after this competing onset are therefore not matched to the
|
||||
previous opportunity. This prevents an other-to-other handover from being
|
||||
treated as an open floor for the wearer.
|
||||
|
||||
Operating thresholds are selected on the development split and frozen for test
|
||||
evaluation. Our primary behavioral metric is false wearer entries per minute at
|
||||
a matched wearer-entry recall of $0.8$; dense mAP and wearer-yield performance
|
||||
are reported as complementary measures.
|
||||
|
||||
|
||||
\section{Results and Analysis}
|
||||
\label{sec:results}
|
||||
|
||||
\subsection{Baselines}
|
||||
The confirmatory comparison contains: an EgoSpeak reproduction; the frozen
|
||||
Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local
|
||||
audio; a parameter- and history-matched temporal local branch; and the proposed
|
||||
@@ -237,40 +265,6 @@ not a second paper storyline. \todo{Freeze all baseline implementations and
|
||||
ensure the EgoSpeak comparison uses identical targets, split, causal validity,
|
||||
and evaluation.}
|
||||
|
||||
\textbf{Event protocol.}
|
||||
Dense ten-horizon AP/mAP is retained for EgoSpeak comparability. The implemented
|
||||
competing-risk evaluator counts each eligible aggregate-other boundary once,
|
||||
ends a response window before the next other onset, and ends a yield window
|
||||
before the next other offset. Pending the remaining P0 real-data audit and
|
||||
scenario freeze, the intended primary behavior metric is false wearer entries per
|
||||
minute of multi-other opportunity at a training-side selected wearer-onset
|
||||
recall, with $R=.8$ as the headline operating point. The intended analysis also
|
||||
reports onset latency and wearer-yield recall/latency, following the
|
||||
event-oriented emphasis on recall, false alarms, and latency
|
||||
\cite{jiang2026turnbench}. The protocol will censor a response window after an
|
||||
other offset at the next other onset, and a yield window after an other onset
|
||||
at the next other offset.
|
||||
|
||||
\todo{P0 blocker: run the tested competing-risk evaluator on real predictions,
|
||||
manually audit sampled timelines, and freeze pairing before treating any event
|
||||
score as a paper result. Specify overlap,
|
||||
minimum duration, timeouts, duplicate detections, source boundaries, offscreen
|
||||
cases, and event counts. Until this is complete, call existing values
|
||||
timing-derived proxies rather than final event results.}
|
||||
|
||||
\textbf{Controls and statistics.}
|
||||
The main intervention compares correctly aligned local evidence with
|
||||
time-shifted evidence, removed vision, global-frame visual pooling, and the
|
||||
capacity-matched temporal branch. A same-rate control is required for a
|
||||
dual-rate claim. Track/no-ID/broken-continuity controls are added only if a
|
||||
track-aware variant first wins on training-side data. We report three seeds,
|
||||
mean$\pm$sample standard deviation, and paired cluster-bootstrap intervals over
|
||||
sessions or conversation blocks. \todo{Freeze seeds, bootstrap unit/count,
|
||||
optimizer, epochs, hardware, throughput, P50/P95 latency, and trainable
|
||||
parameters.}
|
||||
|
||||
\section{Results and Analysis}
|
||||
\label{sec:results}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Confirmatory EasyCom results. Lower multi-other false entries are
|
||||
|
||||
Reference in New Issue
Block a user