From 45facc8bf4352608e51ed555206134aaeaf38d97 Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Wed, 16 Sep 2026 17:12:31 +0800 Subject: [PATCH] Add wide evaluation table and qualitative placeholder --- manuscript/main/main.tex | 120 +++++++++++++++++++++++++++------------ 1 file changed, 83 insertions(+), 37 deletions(-) diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index d6dc707..a2721ba 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -339,9 +339,12 @@ non-wearer release. It resolves positively when the wearer enters, or negatively when the same non-wearer resumes or a different non-wearer takes the next floor. From the first causal 5-Hz state that observes the release, the entry score is the maximum projected wearer probability before that competing -resolution, capped at 2.0~s. Positive entries without a reviewer-selected -release are reported as unanchored and excluded from this matched-opportunity -analysis; they remain available to the separate raw-onset analysis. +resolution, capped at 2.0~s. A selected-release positive is excluded if another +non-wearer onset occurs before or exactly at the wearer entry, or if wearer +activity already occurred after the selected release. Positive entries without +a reviewer-selected release are reported as unanchored and excluded from this +matched-opportunity analysis; they remain available to the separate raw-onset +analysis. An INT opportunity begins when a non-wearer starts while the wearer holds the floor. It resolves positively when the reviewed behavior is an INT-yield, and @@ -352,8 +355,9 @@ strictly at the competing non-wearer offset. All prediction values are read diagonally from one causal source row of the target-aligned forecast tensor. No input after the observed action outcome is used. For current INT-positive annotations, which do not select a competing onset explicitly, the -implementation uses the latest supplied non-wearer interval crossing the -wearer offset and surfaces cases with multiple eligible intervals for audit. +implementation evaluates only a unique supplied non-wearer interval crossing +the wearer offset; cases with multiple eligible intervals are excluded and +counted for audit rather than assigned an arbitrary onset. The headline entry hard negatives are same-speaker mid-turn pauses and other- to-other transfers. Wearer backchannels are not pooled into headline false- @@ -394,42 +398,57 @@ latency claim. \subsection{Main Comparison} -Table~\ref{tab:main} compares the proposed causal local audiovisual adapter -with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$, -matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched -temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling -control. All methods use the same split, causal inputs, targets, and evaluation -protocol. Event thresholds are selected on training-side calibration data and -frozen before test evaluation. \todo{Freeze all baseline implementations and -verify identical targets, causal validity, and evaluation.} +Table~\ref{tab:main} places the released EgoSpeak checkpoint beside the +currently available Omni screening runs and reserves a row for the final causal +local audiovisual model. Historical unmatched runs are identified explicitly +and cannot support confirmatory attribution. The final comparison will use the +same split, causal inputs, targets, training/selection protocol, and three +random seeds; event thresholds will be selected on training-side calibration +data and frozen before test evaluation. \todo{Freeze all baseline +implementations and verify identical targets, causal validity, and evaluation.} -\begin{table}[t] -\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use -the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP -is the corresponding opportunity AP. Event results will be populated only -after protocol-v2 reruns and human adjudication.} +\begin{table*}[t] +\caption{EasyCom Sessions~1--3 dense future-activity and raw-boundary +screening. All values are percentages. ``Avg.'' averages dense mAP over the ten +horizons; On/Off are mean raw wearer onset/offset AP and are timing diagnostics, +not semantic ENTRY/INT scores. $\dagger$ denotes historical seed-0 runs with an +unmatched training/selection protocol; $\ddagger$ denotes a three-seed screen. +Missing horizon curves and the final matched comparison will be filled after +the matched dense/boundary reruns.} \label{tab:main} \centering -\small -\resizebox{\columnwidth}{!}{% -\begin{tabular}{lccccc} +\scriptsize +\setlength{\tabcolsep}{2.6pt} +\resizebox{\textwidth}{!}{% +\begin{tabular}{@{}lccccccccccccc@{}} \hline -Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ -& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\ +& \multicolumn{10}{c}{Dense mAP at prediction horizon (s) $\uparrow$} +& & \multicolumn{2}{c}{Raw boundary AP $\uparrow$}\\ +\cline{2-11}\cline{13-14} +Method & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 +& Avg. & On & Off\\ \hline -EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & \todo{--/--}\\ -Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ -Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ -$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ -$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ -$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ -Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ +EgoSpeak MiniROADA A+V + & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 + & 61.19 & 24.49 & 38.02\\ +Omni $G{+}A^{\dagger}$ + & 81.72 & -- & -- & -- & -- & -- & -- & -- & -- & -- + & 65.11 & 25.96 & 45.25\\ +Omni $G{+}A{+}$pooled-$V^{\dagger}$ + & 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & -- + & 65.40 & 26.02 & 45.37\\ +Current-frame F1$^{\ddagger}$ + & $81.78{\pm}0.16$ & -- & -- & -- & -- & -- & -- & -- & -- & -- + & $64.95{\pm}0.09$ & -- & --\\ +$G+$causal local AV (ours) + & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- + & -- & -- & --\\ \hline \end{tabular}} -\end{table} +\end{table*} \textbf{Dense forecasting.} -The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw +The EgoSpeak baseline reaches 74.20\% 0.2-s mAP and 61.19\% mean mAP; its raw wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values remain pending a protocol-v2 rerun and human review; continuous wearer-score plots do not alone establish semantic floor-yield selectivity. @@ -459,6 +478,34 @@ beyond additional temporal capacity}. \todo{If this difference is not statistically supported, state instead that the local temporal branch accounts for most of the gain.} +\subsection{Qualitative Floor-Event Analysis} + +We will compare EgoSpeak with the final matched model on human-confirmed +examples selected by pre-declared event subtype rather than post-hoc score +extremes. Each case will align participant-resolved activity, minimal transcript +context, the opportunity anchor and competing-risk censor, and the two models' +causally emitted wearer forecasts. The comparison will include a genuine wearer +entry, an other-to-other false-entry opportunity, and an INT-yield paired with a +non-takeover overlap; a shared failure case will be retained when space permits. + +\begin{figure*}[t] +\centering +\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{ +\centering +\textbf{Qualitative comparison placeholder}\\[0.7em] +\begin{tabular}{ccc} +Case A: wearer entry & Case B: other$\rightarrow$other HN +& Case C: INT-yield / failed takeover\\ +participant timeline + transcript & opportunity anchor + censor +& EgoSpeak vs. proposed forecast\\ +\end{tabular} +}} +\caption{Planned qualitative floor-event comparison. Final panels will use +human-confirmed protocol-v2 cases and show the same causal source time for both +models; the figure is a layout placeholder and contains no result claim.} +\label{fig:qualitative_floor_events} +\end{figure*} + \subsection{Information Opportunity} Before designing the deployable adapter, we measured whether local floor @@ -569,7 +616,7 @@ future-media intervention test. Report within-backbone 5-Hz versus 1-Hz deltas plus cache/no-cache throughput and memory; do not interpret unmatched Qwen--MiniCPM raw scores as a pure backbone effect.} -\subsection{Limitations and Qualitative Analysis} +\subsection{Limitations} The results support \todo{precise supported claim}, but several limitations remain. @@ -579,10 +626,9 @@ tracking assumption}. Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor exchange, etc.} remains challenging. -\todo{Add one compact, pre-selected qualitative timeline only if space permits: -one multi-other false-entry improvement paired with one preserved real wearer -onset or one failure. Examples illustrate aggregate rates and never establish -the claim by themselves.} +\todo{Replace Fig.~\ref{fig:qualitative_floor_events} with compact, pre-selected +human-confirmed timelines. Examples illustrate aggregate rates and never +establish the claim by themselves.} \section{Conclusion} \label{sec:conclusion}