Add EgoSpeak dense event proxy evaluation

This commit is contained in:
Xin Wang
2026-09-15 13:35:35 +08:00
parent b3dc02b293
commit a8a09d8fc5
8 changed files with 826 additions and 24 deletions
+37 -22
View File
@@ -328,32 +328,47 @@ configuration, trainable parameter counts, and inference latency.}
We first retain dense future-activity evaluation for direct comparison with
EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.
Behavioral evaluation uses fixed-horizon event classification. For both
wearer-entry and INT-yield, we align the dense forecast to each adjudicated
item at each of the ten nominal horizons. A positive remains a single corrected
5-Hz behavior-anchor bin: its entry score is wearer activity probability and
its INT-yield score is one minus wearer activity probability. We do not expand
the positive timestamp into a tolerance window. A hard-negative opportunity is
one sample whose score is the maximum erroneous-action probability over its
labeled half-open region at the same fixed horizon. Thus, each positive event
or hard-negative opportunity contributes exactly one score per horizon.
For dense-only models, behavioral evaluation begins with an explicitly
activity-derived event proxy rather than treating wearer voice activity as a
semantic event detector. Following the prediction-region principle used by
VAP~\cite{ekstedt2022vap}, every adjudicated item contributes one score from a
fixed 0.6-s (three-frame) causal decision region. A positive uses the three
source frames immediately preceding its corrected behavior anchor. An
independently mined hard negative uses the first three source frames after its
opportunity begins; a raw wearer boundary relabeled as a hard negative instead
uses the same pre-boundary region as a positive. Items without a complete
three-frame region are excluded and counted. The longer interval supplied to
the annotator is evidence context only and is never a model-scoring window.
At each decision frame, we follow the target-aligned forecast tensor
diagonally to recover the wearer probabilities emitted at that same causal
source time. We average the 0.6--2.0-s horizons to obtain a sustained wearer-
activity score. This mean is the entry proxy, and one minus the mean is the
INT-yield proxy. Max-pooling the three decision-frame scores gives exactly one
score per positive event or hard-negative opportunity. This construction can
capture sustained activity versus a brief vocalization, but it cannot by
itself establish semantic floor intent; in particular, wearer backchannels and
within-turn pauses remain deliberately difficult negatives. A later linear
event probe, if used, is fit only on training-side annotations and compared at
matched capacity across representations, following the separated downstream-
probe logic of MuVAP~\cite{qi2026muvap}.
\textbf{False-positive scoring rule.}
A false positive is a hard-negative opportunity whose region-level score
A false positive is a hard-negative opportunity whose fixed-region proxy score
exceeds the frozen threshold, regardless of whether the wearer subsequently
exhibits an onset or offset. Region max-pooling means that multiple high-scoring
frames still count as one false-positive opportunity. Thus, false-positive
exhibits an onset or offset. Max-pooling means that multiple high-scoring
decision frames still count as one false-positive opportunity. Thus, false-positive
evaluation includes conversational situations in which the correct wearer
behavior is to remain silent or retain the floor.
For every horizon we report event AP and the recall--false-positive trade-off.
The headline behavioral operating point uses the first, nominal 0.2-s forecast
bin at matched recall $R=0.8$; thresholds are selected independently for entry
and INT from training-side out-of-fold predictions. We additionally report the
complete 0.2--2.0-s curves and the other-to-other hard-negative subset to test
We report threshold-free proxy AP for entry and INT. The headline behavioral
operating point additionally reports false-positive rate at matched recall
$R=0.8$; thresholds are selected independently for entry and INT from training-
side out-of-fold predictions. The other-to-other hard-negative subset tests
whether local visual evidence specifically reduces erroneous wearer action
scores during multiparty floor transfers. \todo{Freeze the hard-negative
region endpoints and verify identical event denominators across methods.}
scores during multiparty floor transfers. \todo{Freeze the 0.6-s decision
window and 0.6--2.0-s aggregation range using training-side data, run window
sensitivity analysis, and verify identical event denominators across methods.}
A deployment-style committed-event analysis with one-to-one matching and
signed latency is secondary and is omitted unless a threshold, persistence,
@@ -378,8 +393,8 @@ verify identical targets, causal validity, and evaluation.}
\begin{table}[t]
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use
the nominal 0.2-s forecast bin at matched recall $R=0.8$; E/I AP averages each
event family over all ten fixed horizons.}
the fixed-region activity-derived proxy at matched recall $R=0.8$; E/I AP is
the corresponding event-balanced proxy AP.}
\label{tab:main}
\centering
\small
@@ -411,7 +426,7 @@ evidence primarily improves \todo{the interpretation supported by the data}.
\textbf{Annotated behavior.}
At matched wearer-entry recall $R=0.8$, the proposed adapter reduces entry FPR
from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR
from \todo{$A$} to \todo{$B$}. Its ten-horizon mean entry/INT event APs are
from \todo{$A$} to \todo{$B$}. Its entry/INT activity-proxy APs are
\todo{$E/I$}. These recall-matched opportunity error rates test whether lower
scores on confusing floor situations are obtained without suppressing
legitimate entry or yield events.