Fix opportunity competing-risk evaluation
This commit is contained in:
@@ -10,6 +10,12 @@ head.
|
||||
|
||||
A positive must have a reviewer-selected preceding non-wearer release. Its
|
||||
opportunity starts at that release and resolves at the reviewed wearer entry.
|
||||
If a new non-wearer onset occurs before or exactly at that entry, the release
|
||||
has already been resolved by a competing event and the positive is excluded.
|
||||
The same applies when wearer activity already occurred between the selected
|
||||
release and the reviewed onset: the selected release is then a stale,
|
||||
post-outcome anchor. Both cases are counted separately in
|
||||
`review_diagnostics.skipped`.
|
||||
Core hard negatives start at the same type of non-wearer release and resolve
|
||||
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
||||
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
||||
@@ -27,10 +33,10 @@ the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
||||
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
||||
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
||||
|
||||
Current INT reviews do not select one competing onset. The loader therefore
|
||||
uses the latest supplied other interval that crosses the wearer offset within
|
||||
the annotation tolerance. It records how many intervals were eligible so
|
||||
multi-competitor cases can be audited before the protocol is frozen.
|
||||
Current INT reviews do not select one competing onset. A unique supplied other
|
||||
interval that crosses the wearer offset within the annotation tolerance is
|
||||
used. Cases with multiple eligible intervals are excluded and counted rather
|
||||
than being assigned a deterministic latest-onset anchor.
|
||||
|
||||
## Provisional EgoSpeak run
|
||||
|
||||
@@ -58,21 +64,24 @@ Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
||||
cap, positive anchor-selection policy, and multiple-competitor handling using
|
||||
training-side data only.
|
||||
|
||||
## Recorded Sessions 1--3 baseline
|
||||
## Superseded v1 Sessions 1--3 baseline
|
||||
|
||||
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard
|
||||
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard
|
||||
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid
|
||||
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed
|
||||
confirmatory results. Dense, raw-boundary, and opportunity metrics and their
|
||||
Figure 7 interpretation are frozen together in the
|
||||
The first threshold-free run used the superseded v1 loader and reported ENTRY
|
||||
AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
|
||||
AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
|
||||
81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
|
||||
must not be reported: v1 allowed a
|
||||
selected ENTRY release to cross an intervening other onset and selected the
|
||||
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
|
||||
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
|
||||
before updating the manuscript. The historical output is retained in the
|
||||
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
||||
|
||||
The expanded Omni screen finds no opportunity-ranking gain from the current
|
||||
The expanded v1 Omni screen found no opportunity-ranking gain from the current
|
||||
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
|
||||
41.561%/22.991%, below the historical pooled-V seed-0 values
|
||||
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
|
||||
selection protocols, this is a stable negative screen rather than a matched
|
||||
confirmatory comparison. Dense, boundary, per-seed, and provenance details are
|
||||
selection protocols and used the superseded opportunity loader, only its dense
|
||||
and raw-boundary findings remain current. Per-seed and provenance details are
|
||||
in the
|
||||
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
|
||||
|
||||
Reference in New Issue
Block a user