Align manuscript with EOT INT evaluation

.gitignore

@@ -15,5 +15,8 @@ __pycache__/
 # Local editors and environments.
 .idea/
 .vscode/
+!.vscode/
+.vscode/*
+!.vscode/settings.json
 .venv/
 venv/

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+  "latex-workshop.latex.outDir": "%DIR%/build"
+}

manuscript/main/main.tex

@@ -33,9 +33,9 @@ not explicitly trained to expose sub-second, wearer-relative floor evidence.
 We therefore study a candidate causal local floor adapter that combines the global
 Omni state with short-window mixed audio and an anonymous set of visible-person
 observations, without assuming persistent speaker identity. On EasyCom,
-\todo{report the final three-seed dense and competing-risk-censored event
+\todo{report the final three-seed dense and committed-event
-results: false-entry reduction at matched wearer-onset recall, onset latency,
+results: EOT-response and INT-yield recall, false-positive rate, latency,
-yield behavior, confidence intervals, and oracle-gap recovery}. Controlled
+confidence intervals, and oracle-gap recovery}. Controlled
 \todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
 show that \todo{state the supported mechanism, without claiming addressee or
 speaker identity}. These results demonstrate \todo{the narrow conclusion
@@ -102,9 +102,9 @@ corresponding TODO gates pass:
   \item We evaluate a wearer-relative causal local floor adapter that combines
   long-context Omni information with short-window audio and anonymous
   visible-person evidence.
-  \item We evaluate whether the adapter reduces false wearer entries during
+  \item We evaluate whether the adapter reduces erroneous wearer commits
-  multi-other exchanges at matched valid-onset recall, using causal alignment
+  during multiparty floor transfers under an EOT/INT event protocol with
-  interventions and competing-risk-censored entry/yield analysis.
+  explicitly annotated hard-negative regions.
 \end{itemize}
 
 \begin{figure}[t]
@@ -112,7 +112,7 @@ corresponding TODO gates pass:
 \fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
 \todo{Draw only the selected method: causal AV to the long-context Omni path;
 the short-window mixed-audio and anonymous person-set path; residual fusion;
-the common $10\times3$ head; and the wearer-silent/active event interpretation.
+the common $10\times3$ head; and the separate EOT-response/INT-yield evaluation.
 Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
 Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
 \caption{Long-context Omni understanding and short-window local floor evidence
@@ -136,13 +136,13 @@ Wearer--other overlap is allowed. The model outputs
  \in[0,1]^{K\times3}.
  \label{eq:task}
 \end{equation}
-The same wearer-activity forecast has two controller interpretations. In a
+The wearer-activity forecast also provides a causal control score. When the
-wearer-silent opportunity, $\hat y^{\mathrm w}_{t,1}$ is an entry score. While
+wearer is silent, $\hat y^{\mathrm w}_{t,1}$ acts as a near-term take-turn
-the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ is a near-term yield score.
+score; while the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ acts as a
-The current self-state is used to define these observable event strata, not as
+yield score. These scores are trained only from future activity targets.
-an unreported oracle model input. \todo{If the final adapter explicitly uses
+Committed EOT-response and INT-yield events used for behavioral evaluation are
-self-state, define its strictly causal and deployment-available source and add
+defined separately from participant-resolved annotations in
-a matched no-state ablation; otherwise keep self-state evaluation-only.}
+Sec.~\ref{sec:experiments}.
 
 \subsection{Global Omni baseline}
 
@@ -216,41 +216,80 @@ per-person speech label enters the deployable model. \todo{Add an active-speaker
 or event auxiliary loss only if it improves the frozen primary protocol and
 its label/input boundary is fully reported.}
 
-\section{Experiments} 
+
-\label{sec:experiments} 
+\section{Experiments}
+\label{sec:experiments}
 
 \subsection{Dataset and Data Preparation}
 
-We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 for training and development and Sessions 1--3 for final evaluation \cite{donley2021easycom}. All hyperparameters, event windows, operating thresholds, and checkpoints are selected without access to the test sessions. The input stream is converted to strictly causal examples at 5~Hz. Each decision point is paired with ten future 0.2-s activity bins for background, camera wearer, and aggregate non-wearer speech. Provided person boxes are used only to construct the visible-person inputs; participant identity, close-mic audio, transcripts, and future observations are not available to the model. \todo{Report dataset duration, number of training/test examples and events, and the frozen development/calibration split.}
+We evaluate on EasyCom following the EgoSpeak split, using Sessions 4--12 as
+the training/development pool and Sessions 1--3 only for final evaluation
+\cite{donley2021easycom}. The participant-resolved speech annotations are
+converted into two parallel representations. First, they are downsampled to
+5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background,
+camera-wearer, and aggregate non-wearer activity. Second, the original
+participant timeline is retained to derive committed interaction events for
+behavioral evaluation.
+
+We consider two wearer-relative events. An \emph{EOT-response} denotes a
+non-wearer floor release followed by the wearer taking the floor, whereas an
+\emph{INT-yield} denotes a non-wearer takeover while the wearer is speaking,
+followed by the wearer relinquishing the floor. Candidate events are generated
+automatically from participant-level onset/offset annotations. Mid-turn pauses,
+other-to-other handovers, backchannels, and non-takeover overlaps are retained
+as hard-negative regions rather than positive events. Annotation-corrupted or
+unresolvable boundaries are excluded from scoring. Ambiguous candidates are
+manually adjudicated.
+
+Participant identity is used only to construct these evaluation annotations
+and is never provided to the prediction model. Provided person boxes are used
+only for the causal visible-person input; transcripts, target-of-speech labels,
+close-microphone audio, and future observations are unavailable to the model.
+\todo{Report duration and event statistics: EOT positives/hard negatives,
+INT positives/hard negatives, excluded candidates, and manual-adjudication
+count for train/development and test. Freeze the event-construction and
+adjudication protocol before reporting semantic EOT/INT results; until then,
+retain the timing-derived proxy terminology.}
 
 \subsection{Model Configuration and Training}
 
-Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout, matched PEFT/LoRA adaptation, global features with recent local audio, a parameter- and history-matched temporal local branch, and the proposed causal local audio--visual adapter. Qwen2.5-Omni-7B is used only as a capacity-scaling control. All learned variants use the same causal context, prediction head, training examples, and class-balanced binary cross-entropy objective in Eq.~\eqref{eq:loss}. Unless otherwise stated, model selection is performed on the development data and final numbers are averaged over three random seeds. \todo{Freeze context length, local window/rates, optimizer, epochs, LoRA configuration, trainable parameter counts, and inference latency.}
+Our main backbone is Qwen2.5-Omni-3B. We compare a frozen global Omni readout,
+matched PEFT/LoRA adaptation, global features with recent local audio, a
+parameter- and history-matched temporal local branch, and the proposed causal
+local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling
+control. All learned variants use identical causal examples, prediction
+horizons, prediction heads, and the class-balanced binary cross-entropy
+objective in Eq.~\eqref{eq:loss}.
 
-\subsection{Evaluation Metrics}
+Model selection is performed using training-side data only. Event operating
+points are selected from session-level out-of-fold predictions over
+Sessions~4--12; the final model is then retrained on all nine sessions before
+evaluation on Sessions~1--3. Final results are averaged over three random
+seeds.
+\todo{Freeze context length, local window/rates, optimizer, epochs, LoRA
+configuration, trainable parameter counts, and inference latency.}
 
-We evaluate both dense future-activity prediction and event-level floor
+\subsection{Evaluation Protocol}
-control. For comparability with EgoSpeak, we report per-frame AP at
-$0.2$--$2.0$~s horizons and their mean mAP.
 
-For event-level evaluation, a \emph{wearer entry} is a transition from wearer
+We first retain dense future-activity evaluation for direct comparison with
-silence to speech, while a \emph{wearer yield} is a transition from speech to
+EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.
-silence. An entry prediction is considered correct if it is committed within
-the predefined response window of a ground-truth wearer onset. We report entry
-recall, false entries per minute of eligible wearer-silent time, and signed
-onset latency. Yield events are evaluated analogously using recall and latency.
 
-Multiparty interaction introduces a competing-risk condition. After another
+Behavioral evaluation follows a committed-event protocol. For both
-speaker releases the floor, the current wearer-entry opportunity is terminated
+EOT-response and INT-yield, each gold event is matched to at most one committed
-if another non-wearer participant starts speaking before the wearer does.
+prediction and is scored by recall and signed detection latency. A prediction
-Predictions after this competing onset are therefore not matched to the
+inside a task-specific hard-negative region is counted as a false positive.
-previous opportunity. This prevents an other-to-other handover from being
+EOT hard negatives include mid-turn continuations and competing
-treated as an open floor for the wearer.
+other-to-other floor transfers; INT hard negatives include backchannels and
+overlaps in which the wearer retains the floor. Regions excluded during
+adjudication do not contribute to either the positive or negative denominator.
 
-Operating thresholds are selected on the development split and frozen for test
+We report the recall--false-positive trade-off and a common operating point at
-evaluation. Our primary behavioral metric is false wearer entries per minute at
+matched recall $R=0.8$. The primary behavioral quantities are therefore EOT
-a matched wearer-entry recall of $0.8$; dense mAP and wearer-yield performance
+false-positive rate, INT false-positive rate, and signed latency at the frozen
-are reported as complementary measures.
+operating point. We additionally report performance on the other-to-other
+subset to test whether local visual evidence specifically reduces erroneous
+wearer turn-taking during multiparty floor transfers. \todo{Freeze the FPR
+denominators and report signed median latency separately for EOT and INT.}
 
 
 \section{Results and Analysis}
@@ -268,25 +307,25 @@ frozen before test evaluation. \todo{Freeze all baseline implementations and
 verify identical targets, causal validity, and evaluation.}
 
 \begin{table}[t]
-\caption{Confirmatory EasyCom results. Higher mAP and yield recall are better;
+\caption{Confirmatory EasyCom results. EOT and INT false-positive rates are
-lower false entries and onset latency are better. FP/min is measured at
+measured at matched recall $R=0.8$. E/I Lat. gives signed median EOT/INT
-matched valid-onset recall $R=0.8$.}
+latency in seconds; values closer to zero are better.}
 \label{tab:main}
 \centering
 \small
 \resizebox{\columnwidth}{!}{%
 \begin{tabular}{lccccc}
 \hline
-Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ & FP/min $\downarrow$
+Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
-& Onset lat. $\downarrow$ & Yield R $\uparrow$\\
+& EOT FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I Lat.\\
 \hline
-EgoSpeak reproduction       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+EgoSpeak reproduction       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
-Omni-3B frozen $G$          & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Omni-3B frozen $G$          & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
-Omni-3B matched PEFT        & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Omni-3B matched PEFT        & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
-$G+$local audio             & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+$G+$local audio             & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
-$G+$matched temporal local  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+$G+$matched temporal local  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
-$G+$causal local AV (ours)  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+$G+$causal local AV (ours)  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
-Omni-7B scaling control     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Omni-7B scaling control     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
 \hline
 \end{tabular}}
 \end{table}
@@ -299,14 +338,13 @@ The gain is largest at \todo{short prediction horizons / transition-adjacent
 frames / specify actual observation}, indicating that local audiovisual
 evidence primarily improves \todo{the interpretation supported by the data}.
 
-\textbf{Floor-entry behavior.}
+\textbf{Committed behavior.}
-At matched valid-onset recall $R=0.8$, the proposed adapter reduces
+At matched EOT-response recall $R=0.8$, the proposed adapter reduces EOT FPR
-multi-other false entries from \todo{$X$} to \todo{$Y$} FP/min
+from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR
-(\todo{$Z\%$ relative reduction}).
+from \todo{$A$} to \todo{$B$}. Its signed median EOT/INT latencies are
-This reduction is obtained while \todo{maintaining / improving} onset latency
+\todo{$+0.XX/+0.XX$~s}. These recall-matched error rates and signed latencies
-and yield recall.
+test whether fewer erroneous commits arise from better floor decisions rather
-Thus, the improvement cannot be explained solely by a more conservative
+than indefinite waiting.
-decision threshold.
 
 \textbf{Comparison with adaptation baselines.}
 Matched PEFT yields \todo{result}, whereas adding recent local audio alone
@@ -351,36 +389,32 @@ evaluate the controls in Table~\ref{tab:ablation}.
 
 \begin{table}[t]
 \caption{Mechanism controls for the selected Omni-3B local adapter. All
-variants use the same prediction head and comparable temporal capacity.}
+variants use the same prediction head and comparable temporal capacity; FPRs
+are measured at matched recall $R=0.8$.}
 \label{tab:ablation}
 \centering
 \small
 \resizebox{\columnwidth}{!}{%
-\begin{tabular}{lccc}
+\begin{tabular}{lcccc}
 \hline
-Local condition & Mean mAP $\uparrow$ & FP/min@$R=.8$ $\downarrow$
+Local condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$
-& Yield R $\uparrow$\\
+& O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\
 \hline
-Correct causal local AV       & \todo{--} & \todo{--} & \todo{--}\\
+Remove vision           & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-Remove local vision           & \todo{--} & \todo{--} & \todo{--}\\
+Time-shift vision       & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-Time-shift local vision       & \todo{--} & \todo{--} & \todo{--}\\
+Full-frame vision       & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\
+Correct local AV        & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-Same-rate/capacity control    & \todo{--} & \todo{--} & \todo{--}\\
 \hline
 \end{tabular}}
 \end{table}
 
-Removing local vision changes \todo{metric} by \todo{$X$}, while temporally
+Removing local vision changes \todo{EOT/O$\rightarrow$O/INT FPR by $X/Y/Z$},
-shifting the visual stream by \todo{$\Delta t$} changes it by \todo{$Y$}.
+while shifting the visual stream by \todo{$\Delta t$} produces \todo{result}.
-Replacing person-local visual evidence with a global/full-frame visual pool
+Replacing person-local evidence with full-frame vision produces \todo{result}.
-produces \todo{result}.
+If aligned local AV improves estimates of surrounding floor occupancy, its
-Together, these comparisons \todo{support / do not support} the hypothesis
+largest selective reduction should appear on the O$\rightarrow$O hard-negative
-that the benefit depends specifically on correctly timed local visual
+subset. Together, these comparisons \todo{support / do not support} that
-evidence.
+mechanism beyond generic visual context.
-
-The same-rate/capacity control achieves \todo{result}, showing that
-\todo{the gain cannot / can largely} be attributed to additional temporal
-parameters or higher-rate processing alone.
 
 \todo{Report paired intervals and the supported interpretation. If correct
 and shifted/removed vision do not separate, drop the audiovisual
@@ -391,7 +425,8 @@ not describe lower false entry as success.}
 \subsection{Scaling Control}
 
 Increasing the frozen Omni backbone from 3B to 7B yields
-\todo{$X$ points} in 0.2-s mAP and \todo{$Y$} FP/min at matched recall.
+\todo{$X$ points} in 0.2-s mAP and \todo{EOT/INT FPR of $Y/Z$} at matched
+recall.
 The proposed 3B adapter \todo{remains better than / approaches / is surpassed
 by} the 7B scaling control.
 
@@ -422,8 +457,8 @@ This study asks whether a general-purpose Omni representation can be adapted
 for causal floor forecasting of an invisible egocentric wearer. The candidate
 method separates long-context Omni understanding from short-window local floor
 evidence without assuming persistent speaker identity. \todo{State the final
-learned-method improvement, matched-recall false-entry result, onset/yield
+learned-method improvement, matched-recall EOT/INT false-positive rates,
-non-degradation, causal-control evidence, and main limitation. If no learned
+signed latency, causal-control evidence, and main limitation. If no learned
 variant passes these tests, reframe the submission as an information audit or
 do not make the method claim.}