Add opportunity-conditioned EgoSpeak event evaluation

2026-09-16 02:27:44 +08:00
parent af4817004b
commit 89e685c830
7 changed files with 1272 additions and 34 deletions
@@ -0,0 +1,59 @@
+# EasyCom opportunity-conditioned ENTRY/YIELD evaluation
+
+[`evaluate_egospeak_opportunity_events.py`](../evaluate_egospeak_opportunity_events.py)
+uses LLM-assisted or human-reviewed semantic labels to define matched
+conversational opportunities while scoring every model only from its original
+EgoSpeak-compatible wearer probabilities. It does not train a semantic event
+head.
+
+## ENTRY
+
+A positive must have a reviewer-selected preceding non-wearer release. Its
+opportunity starts at that release and resolves at the reviewed wearer entry.
+Core hard negatives start at the same type of non-wearer release and resolve
+when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
+non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
+maximum projected wearer probability up to the competing resolution, capped at
+2.0 seconds.
+
+`ENTRY_WEARER_BACKCHANNEL` is excluded from headline ENTRY FPR. Predicting
+wearer activity for a real backchannel is correct under EgoSpeak's dense target;
+ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.
+
+## YIELD
+
+An opportunity starts at a non-wearer onset during wearer speech. The score is
+the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
+non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
+backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
+
+Current INT reviews do not select one competing onset. The loader therefore
+uses the latest supplied other interval that crosses the wearer offset within
+the annotation tolerance. It records how many intervals were eligible so
+multi-competitor cases can be audited before the protocol is frozen.
+
+## Provisional EgoSpeak run
+
+Threshold-free AP on the official test sessions can be run without fitting a
+test threshold:
+
+```bash
+python3 evaluate_egospeak_opportunity_events.py \
+  --reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
+  --run egospeak=results/egospeak_miniroada_floor_eval \
+  --sessions Session_1 Session_2 Session_3 \
+  --allow-provisional-llm-reviews \
+  --print-event-examples 10 \
+  --output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json
+```
+
+The output JSON records label coverage and every exclusion reason. The metrics
+CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors,
+causal source cutoffs, censor times, horizons, and wearer probabilities for
+timeline auditing.
+
+Do not request matched-recall operating points on Sessions 1--3 without
+disjoint Sessions 4--12 calibration reviews and out-of-fold predictions.
+Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
+cap, positive anchor-selection policy, and multiple-competitor handling using
+training-side data only.