88 lines
4.4 KiB
Markdown
88 lines
4.4 KiB
Markdown
# EasyCom opportunity-conditioned ENTRY/YIELD evaluation
|
|
|
|
[`evaluate_egospeak_opportunity_events.py`](../evaluate_egospeak_opportunity_events.py)
|
|
uses LLM-assisted or human-reviewed semantic labels to define matched
|
|
conversational opportunities while scoring every model only from its original
|
|
EgoSpeak-compatible wearer probabilities. It does not train a semantic event
|
|
head.
|
|
|
|
## ENTRY
|
|
|
|
A positive must have a reviewer-selected preceding non-wearer release. Its
|
|
opportunity starts at that release and resolves at the reviewed wearer entry.
|
|
If a new non-wearer onset occurs before or exactly at that entry, the release
|
|
has already been resolved by a competing event and the positive is excluded.
|
|
The same applies when wearer activity already occurred between the selected
|
|
release and the reviewed onset: the selected release is then a stale,
|
|
post-outcome anchor. Both cases are counted separately in
|
|
`review_diagnostics.skipped`.
|
|
Core hard negatives start at the same type of non-wearer release and resolve
|
|
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
|
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
|
maximum projected wearer probability up to the competing resolution, capped at
|
|
2.0 seconds.
|
|
|
|
`ENTRY_WEARER_BACKCHANNEL` is excluded from headline ENTRY FPR. Predicting
|
|
wearer activity for a real backchannel is correct under EgoSpeak's dense target;
|
|
ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.
|
|
|
|
## YIELD
|
|
|
|
An opportunity starts at a non-wearer onset during wearer speech. The score is
|
|
the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
|
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
|
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
|
|
|
Current INT reviews do not select one competing onset. A unique supplied other
|
|
interval that crosses the wearer offset within the annotation tolerance is
|
|
used. Cases with multiple eligible intervals are excluded and counted rather
|
|
than being assigned a deterministic latest-onset anchor.
|
|
|
|
## Provisional EgoSpeak run
|
|
|
|
Threshold-free AP on the official test sessions can be run without fitting a
|
|
test threshold:
|
|
|
|
```bash
|
|
python3 evaluate_egospeak_opportunity_events.py \
|
|
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
|
|
--run egospeak=results/egospeak_miniroada_floor_eval \
|
|
--sessions Session_1 Session_2 Session_3 \
|
|
--allow-provisional-llm-reviews \
|
|
--print-event-examples 10 \
|
|
--output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json
|
|
```
|
|
|
|
The output JSON records label coverage and every exclusion reason. The metrics
|
|
CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors,
|
|
causal source cutoffs, censor times, horizons, and wearer probabilities for
|
|
timeline auditing.
|
|
|
|
Do not request matched-recall operating points on Sessions 1--3 without
|
|
disjoint Sessions 4--12 calibration reviews and out-of-fold predictions.
|
|
Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
|
cap, positive anchor-selection policy, and multiple-competitor handling using
|
|
training-side data only.
|
|
|
|
## Superseded v1 Sessions 1--3 baseline
|
|
|
|
The first threshold-free run used the superseded v1 loader and reported ENTRY
|
|
AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
|
|
AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
|
|
81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
|
|
must not be reported: v1 allowed a
|
|
selected ENTRY release to cross an intervening other onset and selected the
|
|
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
|
|
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
|
|
before updating the manuscript. The historical output is retained in the
|
|
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
|
|
|
The expanded v1 Omni screen found no opportunity-ranking gain from the current
|
|
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
|
|
41.561%/22.991%, below the historical pooled-V seed-0 values
|
|
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
|
|
selection protocols and used the superseded opportunity loader, only its dense
|
|
and raw-boundary findings remain current. Per-seed and provenance details are
|
|
in the
|
|
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
|