Add wide evaluation table and qualitative placeholder

This commit is contained in:
Xin Wang
2026-09-16 17:12:31 +08:00
parent d294bd6ede
commit 45facc8bf4
+83 -37
View File
@@ -339,9 +339,12 @@ non-wearer release. It resolves positively when the wearer enters, or
negatively when the same non-wearer resumes or a different non-wearer takes the
next floor. From the first causal 5-Hz state that observes the release, the
entry score is the maximum projected wearer probability before that competing
resolution, capped at 2.0~s. Positive entries without a reviewer-selected
release are reported as unanchored and excluded from this matched-opportunity
analysis; they remain available to the separate raw-onset analysis.
resolution, capped at 2.0~s. A selected-release positive is excluded if another
non-wearer onset occurs before or exactly at the wearer entry, or if wearer
activity already occurred after the selected release. Positive entries without
a reviewer-selected release are reported as unanchored and excluded from this
matched-opportunity analysis; they remain available to the separate raw-onset
analysis.
An INT opportunity begins when a non-wearer starts while the wearer holds the
floor. It resolves positively when the reviewed behavior is an INT-yield, and
@@ -352,8 +355,9 @@ strictly at the competing non-wearer offset. All prediction values are read
diagonally from one causal source row of the target-aligned forecast tensor.
No input after the observed action outcome is used. For current INT-positive
annotations, which do not select a competing onset explicitly, the
implementation uses the latest supplied non-wearer interval crossing the
wearer offset and surfaces cases with multiple eligible intervals for audit.
implementation evaluates only a unique supplied non-wearer interval crossing
the wearer offset; cases with multiple eligible intervals are excluded and
counted for audit rather than assigned an arbitrary onset.
The headline entry hard negatives are same-speaker mid-turn pauses and other-
to-other transfers. Wearer backchannels are not pooled into headline false-
@@ -394,42 +398,57 @@ latency claim.
\subsection{Main Comparison}
Table~\ref{tab:main} compares the proposed causal local audiovisual adapter
with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$,
matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched
temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling
control. All methods use the same split, causal inputs, targets, and evaluation
protocol. Event thresholds are selected on training-side calibration data and
frozen before test evaluation. \todo{Freeze all baseline implementations and
verify identical targets, causal validity, and evaluation.}
Table~\ref{tab:main} places the released EgoSpeak checkpoint beside the
currently available Omni screening runs and reserves a row for the final causal
local audiovisual model. Historical unmatched runs are identified explicitly
and cannot support confirmatory attribution. The final comparison will use the
same split, causal inputs, targets, training/selection protocol, and three
random seeds; event thresholds will be selected on training-side calibration
data and frozen before test evaluation. \todo{Freeze all baseline
implementations and verify identical targets, causal validity, and evaluation.}
\begin{table}[t]
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use
the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP
is the corresponding opportunity AP. Event results will be populated only
after protocol-v2 reruns and human adjudication.}
\begin{table*}[t]
\caption{EasyCom Sessions~1--3 dense future-activity and raw-boundary
screening. All values are percentages. ``Avg.'' averages dense mAP over the ten
horizons; On/Off are mean raw wearer onset/offset AP and are timing diagnostics,
not semantic ENTRY/INT scores. $\dagger$ denotes historical seed-0 runs with an
unmatched training/selection protocol; $\ddagger$ denotes a three-seed screen.
Missing horizon curves and the final matched comparison will be filled after
the matched dense/boundary reruns.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\scriptsize
\setlength{\tabcolsep}{2.6pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccc@{}}
\hline
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\
& \multicolumn{10}{c}{Dense mAP at prediction horizon (s) $\uparrow$}
& & \multicolumn{2}{c}{Raw boundary AP $\uparrow$}\\
\cline{2-11}\cline{13-14}
Method & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0
& Avg. & On & Off\\
\hline
EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
EgoSpeak MiniROADA A+V
& 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82
& 61.19 & 24.49 & 38.02\\
Omni $G{+}A^{\dagger}$
& 81.72 & -- & -- & -- & -- & -- & -- & -- & -- & --
& 65.11 & 25.96 & 45.25\\
Omni $G{+}A{+}$pooled-$V^{\dagger}$
& 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & --
& 65.40 & 26.02 & 45.37\\
Current-frame F1$^{\ddagger}$
& $81.78{\pm}0.16$ & -- & -- & -- & -- & -- & -- & -- & -- & --
& $64.95{\pm}0.09$ & -- & --\\
$G+$causal local AV (ours)
& -- & -- & -- & -- & -- & -- & -- & -- & -- & --
& -- & -- & --\\
\hline
\end{tabular}}
\end{table}
\end{table*}
\textbf{Dense forecasting.}
The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw
The EgoSpeak baseline reaches 74.20\% 0.2-s mAP and 61.19\% mean mAP; its raw
wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values
remain pending a protocol-v2 rerun and human review; continuous wearer-score
plots do not alone establish semantic floor-yield selectivity.
@@ -459,6 +478,34 @@ beyond additional temporal capacity}.
\todo{If this difference is not statistically supported, state instead that
the local temporal branch accounts for most of the gain.}
\subsection{Qualitative Floor-Event Analysis}
We will compare EgoSpeak with the final matched model on human-confirmed
examples selected by pre-declared event subtype rather than post-hoc score
extremes. Each case will align participant-resolved activity, minimal transcript
context, the opportunity anchor and competing-risk censor, and the two models'
causally emitted wearer forecasts. The comparison will include a genuine wearer
entry, an other-to-other false-entry opportunity, and an INT-yield paired with a
non-takeover overlap; a shared failure case will be retained when space permits.
\begin{figure*}[t]
\centering
\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{
\centering
\textbf{Qualitative comparison placeholder}\\[0.7em]
\begin{tabular}{ccc}
Case A: wearer entry & Case B: other$\rightarrow$other HN
& Case C: INT-yield / failed takeover\\
participant timeline + transcript & opportunity anchor + censor
& EgoSpeak vs. proposed forecast\\
\end{tabular}
}}
\caption{Planned qualitative floor-event comparison. Final panels will use
human-confirmed protocol-v2 cases and show the same causal source time for both
models; the figure is a layout placeholder and contains no result claim.}
\label{fig:qualitative_floor_events}
\end{figure*}
\subsection{Information Opportunity}
Before designing the deployable adapter, we measured whether local floor
@@ -569,7 +616,7 @@ future-media intervention test. Report within-backbone 5-Hz versus 1-Hz
deltas plus cache/no-cache throughput and memory; do not interpret unmatched
Qwen--MiniCPM raw scores as a pure backbone effect.}
\subsection{Limitations and Qualitative Analysis}
\subsection{Limitations}
The results support \todo{precise supported claim}, but several limitations
remain.
@@ -579,10 +626,9 @@ tracking assumption}.
Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor
exchange, etc.} remains challenging.
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
one multi-other false-entry improvement paired with one preserved real wearer
onset or one failure. Examples illustrate aggregate rates and never establish
the claim by themselves.}
\todo{Replace Fig.~\ref{fig:qualitative_floor_events} with compact, pre-selected
human-confirmed timelines. Examples illustrate aggregate rates and never
establish the claim by themselves.}
\section{Conclusion}
\label{sec:conclusion}