79 lines
3.8 KiB
Markdown
79 lines
3.8 KiB
Markdown
# EasyCom opportunity-conditioned ENTRY/YIELD evaluation
|
|
|
|
[`evaluate_egospeak_opportunity_events.py`](../evaluate_egospeak_opportunity_events.py)
|
|
uses LLM-assisted or human-reviewed semantic labels to define matched
|
|
conversational opportunities while scoring every model only from its original
|
|
EgoSpeak-compatible wearer probabilities. It does not train a semantic event
|
|
head.
|
|
|
|
## ENTRY
|
|
|
|
A positive must have a reviewer-selected preceding non-wearer release. Its
|
|
opportunity starts at that release and resolves at the reviewed wearer entry.
|
|
Core hard negatives start at the same type of non-wearer release and resolve
|
|
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
|
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
|
maximum projected wearer probability up to the competing resolution, capped at
|
|
2.0 seconds.
|
|
|
|
`ENTRY_WEARER_BACKCHANNEL` is excluded from headline ENTRY FPR. Predicting
|
|
wearer activity for a real backchannel is correct under EgoSpeak's dense target;
|
|
ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.
|
|
|
|
## YIELD
|
|
|
|
An opportunity starts at a non-wearer onset during wearer speech. The score is
|
|
the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
|
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
|
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
|
|
|
Current INT reviews do not select one competing onset. The loader therefore
|
|
uses the latest supplied other interval that crosses the wearer offset within
|
|
the annotation tolerance. It records how many intervals were eligible so
|
|
multi-competitor cases can be audited before the protocol is frozen.
|
|
|
|
## Provisional EgoSpeak run
|
|
|
|
Threshold-free AP on the official test sessions can be run without fitting a
|
|
test threshold:
|
|
|
|
```bash
|
|
python3 evaluate_egospeak_opportunity_events.py \
|
|
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
|
|
--run egospeak=results/egospeak_miniroada_floor_eval \
|
|
--sessions Session_1 Session_2 Session_3 \
|
|
--allow-provisional-llm-reviews \
|
|
--print-event-examples 10 \
|
|
--output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json
|
|
```
|
|
|
|
The output JSON records label coverage and every exclusion reason. The metrics
|
|
CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors,
|
|
causal source cutoffs, censor times, horizons, and wearer probabilities for
|
|
timeline auditing.
|
|
|
|
Do not request matched-recall operating points on Sessions 1--3 without
|
|
disjoint Sessions 4--12 calibration reviews and out-of-fold predictions.
|
|
Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
|
cap, positive anchor-selection policy, and multiple-competitor handling using
|
|
training-side data only.
|
|
|
|
## Recorded Sessions 1--3 baseline
|
|
|
|
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard
|
|
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard
|
|
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid
|
|
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed
|
|
confirmatory results. Dense, raw-boundary, and opportunity metrics and their
|
|
Figure 7 interpretation are frozen together in the
|
|
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
|
|
|
The expanded Omni screen finds no opportunity-ranking gain from the current
|
|
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
|
|
41.561%/22.991%, below the historical pooled-V seed-0 values
|
|
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
|
|
selection protocols, this is a stable negative screen rather than a matched
|
|
confirmatory comparison. Dense, boundary, per-seed, and provenance details are
|
|
in the
|
|
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
|