diff --git a/.gitignore b/.gitignore index 7cf1785..1dd5f12 100644 --- a/.gitignore +++ b/.gitignore @@ -15,5 +15,8 @@ __pycache__/ # Local editors and environments. .idea/ .vscode/ +!.vscode/ +.vscode/* +!.vscode/settings.json .venv/ venv/ diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 0000000..59a960d --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "latex-workshop.latex.outDir": "%DIR%/build" +} diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index cc75d25..7be85f4 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -33,9 +33,9 @@ not explicitly trained to expose sub-second, wearer-relative floor evidence. We therefore study a candidate causal local floor adapter that combines the global Omni state with short-window mixed audio and an anonymous set of visible-person observations, without assuming persistent speaker identity. On EasyCom, -\todo{report the final three-seed dense and competing-risk-censored event -results: false-entry reduction at matched wearer-onset recall, onset latency, -yield behavior, confidence intervals, and oracle-gap recovery}. Controlled +\todo{report the final three-seed dense and committed-event +results: EOT-response and INT-yield recall, false-positive rate, latency, +confidence intervals, and oracle-gap recovery}. Controlled \todo{time-shift/remove-vision/same-rate or capacity-matched} interventions show that \todo{state the supported mechanism, without claiming addressee or speaker identity}. These results demonstrate \todo{the narrow conclusion @@ -102,9 +102,9 @@ corresponding TODO gates pass: \item We evaluate a wearer-relative causal local floor adapter that combines long-context Omni information with short-window audio and anonymous visible-person evidence. - \item We evaluate whether the adapter reduces false wearer entries during - multi-other exchanges at matched valid-onset recall, using causal alignment - interventions and competing-risk-censored entry/yield analysis. + \item We evaluate whether the adapter reduces erroneous wearer commits + during multiparty floor transfers under an EOT/INT event protocol with + explicitly annotated hard-negative regions. \end{itemize} \begin{figure}[t] @@ -112,7 +112,7 @@ corresponding TODO gates pass: \fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering \todo{Draw only the selected method: causal AV to the long-context Omni path; the short-window mixed-audio and anonymous person-set path; residual fusion; -the common $10\times3$ head; and the wearer-silent/active event interpretation. +the common $10\times3$ head; and the separate EOT-response/INT-yield evaluation. Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN. Do not draw oracle labels, rejected track variants, or multiple backbones.}}} \caption{Long-context Omni understanding and short-window local floor evidence @@ -136,13 +136,13 @@ Wearer--other overlap is allowed. The model outputs \in[0,1]^{K\times3}. \label{eq:task} \end{equation} -The same wearer-activity forecast has two controller interpretations. In a -wearer-silent opportunity, $\hat y^{\mathrm w}_{t,1}$ is an entry score. While -the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ is a near-term yield score. -The current self-state is used to define these observable event strata, not as -an unreported oracle model input. \todo{If the final adapter explicitly uses -self-state, define its strictly causal and deployment-available source and add -a matched no-state ablation; otherwise keep self-state evaluation-only.} +The wearer-activity forecast also provides a causal control score. When the +wearer is silent, $\hat y^{\mathrm w}_{t,1}$ acts as a near-term take-turn +score; while the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ acts as a +yield score. These scores are trained only from future activity targets. +Committed EOT-response and INT-yield events used for behavioral evaluation are +defined separately from participant-resolved annotations in +Sec.~\ref{sec:experiments}. \subsection{Global Omni baseline} @@ -216,41 +216,80 @@ per-person speech label enters the deployable model. \todo{Add an active-speaker or event auxiliary loss only if it improves the frozen primary protocol and its label/input boundary is fully reported.} -\section{Experiments} -\label{sec:experiments} + +\section{Experiments} +\label{sec:experiments} \subsection{Dataset and Data Preparation} -We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.} +We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 as +the training/development pool and Sessions 1--3 only for final evaluation +\cite{donley2021easycom}. The participant-resolved speech annotations are +converted into two parallel representations. First, they are downsampled to +5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background, +camera-wearer, and aggregate non-wearer activity. Second, the original +participant timeline is retained to derive committed interaction events for +behavioral evaluation. + +We consider two wearer-relative events. An \emph{EOT-response} denotes a +non-wearer floor release followed by the wearer taking the floor, whereas an +\emph{INT-yield} denotes a non-wearer takeover while the wearer is speaking, +followed by the wearer relinquishing the floor. Candidate events are generated +automatically from participant-level onset/offset annotations. Mid-turn pauses, +other-to-other handovers, backchannels, and non-takeover overlaps are retained +as hard-negative regions rather than positive events. Annotation-corrupted or +unresolvable boundaries are excluded from scoring. Ambiguous candidates are +manually adjudicated. + +Participant identity is used only to construct these evaluation annotations +and is never provided to the prediction model. Provided person boxes are used +only for the causal visible-person input; transcripts, target-of-speech labels, +close-microphone audio, and future observations are unavailable to the model. +\todo{Report duration and event statistics: EOT positives/hard negatives, +INT positives/hard negatives, excluded candidates, and manual-adjudication +count for train/development and test. Freeze the event-construction and +adjudication protocol before reporting semantic EOT/INT results; until then, +retain the timing-derived proxy terminology.} \subsection{Model Configuration and Training} -Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.} +Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, +matched PEFT/LoRA adaptation, global features with recent local audio, a +parameter- and history-matched temporal local branch, and the proposed causal +local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling +control. All learned variants use identical causal examples, prediction +horizons, prediction heads, and the class-balanced binary cross-entropy +objective in Eq.~\eqref{eq:loss}. -\subsection{Evaluation Metrics} +Model selection is performed using training-side data only. Event operating +points are selected from session-level out-of-fold predictions over +Sessions~4--12; the final model is then retrained on all nine sessions before +evaluation on Sessions~1--3. Final results are averaged over three random +seeds. +\todo{Freeze context length, local window/rates, optimizer, epochs, LoRA +configuration, trainable parameter counts, and inference latency.} -We evaluate both dense future-activity prediction and event-level floor -control. For comparability with EgoSpeak, we report per-frame AP at -$0.2$--$2.0$~s horizons and their mean mAP. +\subsection{Evaluation Protocol} -For event-level evaluation, a \emph{wearer entry} is a transition from wearer -silence to speech, while a \emph{wearer yield} is a transition from speech to -silence. An entry prediction is considered correct if it is committed within -the predefined response window of a ground-truth wearer onset. We report entry -recall, false entries per minute of eligible wearer-silent time, and signed -onset latency. Yield events are evaluated analogously using recall and latency. +We first retain dense future-activity evaluation for direct comparison with +EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP. -Multiparty interaction introduces a competing-risk condition. After another -speaker releases the floor, the current wearer-entry opportunity is terminated -if another non-wearer participant starts speaking before the wearer does. -Predictions after this competing onset are therefore not matched to the -previous opportunity. This prevents an other-to-other handover from being -treated as an open floor for the wearer. +Behavioral evaluation follows a committed-event protocol. For both +EOT-response and INT-yield, each gold event is matched to at most one committed +prediction and is scored by recall and signed detection latency. A prediction +inside a task-specific hard-negative region is counted as a false positive. +EOT hard negatives include mid-turn continuations and competing +other-to-other floor transfers; INT hard negatives include backchannels and +overlaps in which the wearer retains the floor. Regions excluded during +adjudication do not contribute to either the positive or negative denominator. -Operating thresholds are selected on the development split and frozen for test -evaluation. Our primary behavioral metric is false wearer entries per minute at -a matched wearer-entry recall of $0.8$; dense mAP and wearer-yield performance -are reported as complementary measures. +We report the recall--false-positive trade-off and a common operating point at +matched recall $R=0.8$. The primary behavioral quantities are therefore EOT +false-positive rate, INT false-positive rate, and signed latency at the frozen +operating point. We additionally report performance on the other-to-other +subset to test whether local visual evidence specifically reduces erroneous +wearer turn-taking during multiparty floor transfers. \todo{Freeze the FPR +denominators and report signed median latency separately for EOT and INT.} \section{Results and Analysis} @@ -268,25 +307,25 @@ frozen before test evaluation. \todo{Freeze all baseline implementations and verify identical targets, causal validity, and evaluation.} \begin{table}[t] -\caption{Confirmatory EasyCom results. Higher mAP and yield recall are better; -lower false entries and onset latency are better. FP/min is measured at -matched valid-onset recall $R=0.8$.} +\caption{Confirmatory EasyCom results. EOT and INT false-positive rates are +measured at matched recall $R=0.8$. E/I Lat. gives signed median EOT/INT +latency in seconds; values closer to zero are better.} \label{tab:main} \centering \small \resizebox{\columnwidth}{!}{% \begin{tabular}{lccccc} \hline -Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ & FP/min $\downarrow$ -& Onset lat. $\downarrow$ & Yield R $\uparrow$\\ +Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ +& EOT FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I Lat.\\ \hline -EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ \hline \end{tabular}} \end{table} @@ -299,14 +338,13 @@ The gain is largest at \todo{short prediction horizons / transition-adjacent frames / specify actual observation}, indicating that local audiovisual evidence primarily improves \todo{the interpretation supported by the data}. -\textbf{Floor-entry behavior.} -At matched valid-onset recall $R=0.8$, the proposed adapter reduces -multi-other false entries from \todo{$X$} to \todo{$Y$} FP/min -(\todo{$Z\%$ relative reduction}). -This reduction is obtained while \todo{maintaining / improving} onset latency -and yield recall. -Thus, the improvement cannot be explained solely by a more conservative -decision threshold. +\textbf{Committed behavior.} +At matched EOT-response recall $R=0.8$, the proposed adapter reduces EOT FPR +from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR +from \todo{$A$} to \todo{$B$}. Its signed median EOT/INT latencies are +\todo{$+0.XX/+0.XX$~s}. These recall-matched error rates and signed latencies +test whether fewer erroneous commits arise from better floor decisions rather +than indefinite waiting. \textbf{Comparison with adaptation baselines.} Matched PEFT yields \todo{result}, whereas adding recent local audio alone @@ -351,36 +389,32 @@ evaluate the controls in Table~\ref{tab:ablation}. \begin{table}[t] \caption{Mechanism controls for the selected Omni-3B local adapter. All -variants use the same prediction head and comparable temporal capacity.} +variants use the same prediction head and comparable temporal capacity; FPRs +are measured at matched recall $R=0.8$.} \label{tab:ablation} \centering \small \resizebox{\columnwidth}{!}{% -\begin{tabular}{lccc} +\begin{tabular}{lcccc} \hline -Local condition & Mean mAP $\uparrow$ & FP/min@$R=.8$ $\downarrow$ -& Yield R $\uparrow$\\ +Local condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$ +& O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\ \hline -Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\ -Remove local vision & \todo{--} & \todo{--} & \todo{--}\\ -Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\ -Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\ -Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\ +Remove vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Time-shift vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Full-frame vision & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Correct local AV & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ \hline \end{tabular}} \end{table} -Removing local vision changes \todo{metric} by \todo{$X$}, while temporally -shifting the visual stream by \todo{$\Delta t$} changes it by \todo{$Y$}. -Replacing person-local visual evidence with a global/full-frame visual pool -produces \todo{result}. -Together, these comparisons \todo{support / do not support} the hypothesis -that the benefit depends specifically on correctly timed local visual -evidence. - -The same-rate/capacity control achieves \todo{result}, showing that -\todo{the gain cannot / can largely} be attributed to additional temporal -parameters or higher-rate processing alone. +Removing local vision changes \todo{EOT/O$\rightarrow$O/INT FPR by $X/Y/Z$}, +while shifting the visual stream by \todo{$\Delta t$} produces \todo{result}. +Replacing person-local evidence with full-frame vision produces \todo{result}. +If aligned local AV improves estimates of surrounding floor occupancy, its +largest selective reduction should appear on the O$\rightarrow$O hard-negative +subset. Together, these comparisons \todo{support / do not support} that +mechanism beyond generic visual context. \todo{Report paired intervals and the supported interpretation. If correct and shifted/removed vision do not separate, drop the audiovisual @@ -391,7 +425,8 @@ not describe lower false entry as success.} \subsection{Scaling Control} Increasing the frozen Omni backbone from 3B to 7B yields -\todo{$X$ points} in 0.2-s mAP and \todo{$Y$} FP/min at matched recall. +\todo{$X$ points} in 0.2-s mAP and \todo{EOT/INT FPR of $Y/Z$} at matched +recall. The proposed 3B adapter \todo{remains better than / approaches / is surpassed by} the 7B scaling control. @@ -422,8 +457,8 @@ This study asks whether a general-purpose Omni representation can be adapted for causal floor forecasting of an invisible egocentric wearer. The candidate method separates long-context Omni understanding from short-window local floor evidence without assuming persistent speaker identity. \todo{State the final -learned-method improvement, matched-recall false-entry result, onset/yield -non-degradation, causal-control evidence, and main limitation. If no learned +learned-method improvement, matched-recall EOT/INT false-positive rates, +signed latency, causal-control evidence, and main limitation. If no learned variant passes these tests, reframe the submission as an information audit or do not make the method claim.}