2.8 KiB
EasyCom opportunity-conditioned ENTRY/YIELD evaluation
evaluate_egospeak_opportunity_events.py
uses LLM-assisted or human-reviewed semantic labels to define matched
conversational opportunities while scoring every model only from its original
EgoSpeak-compatible wearer probabilities. It does not train a semantic event
head.
ENTRY
A positive must have a reviewer-selected preceding non-wearer release. Its
opportunity starts at that release and resolves at the reviewed wearer entry.
Core hard negatives start at the same type of non-wearer release and resolve
when the same speaker resumes (ENTRY_OTHER_MIDTURN_PAUSE) or a different
non-wearer takes the next floor (ENTRY_OTHER_TO_OTHER). The score is the
maximum projected wearer probability up to the competing resolution, capped at
2.0 seconds.
ENTRY_WEARER_BACKCHANNEL is excluded from headline ENTRY FPR. Predicting
wearer activity for a real backchannel is correct under EgoSpeak's dense target;
ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.
YIELD
An opportunity starts at a non-wearer onset during wearer speech. The score is
the maximum 1 - p_wearer over 0.2--0.6 seconds, censored strictly at that
non-wearer's offset. Reviewed INT_YIELD items are positive; other
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
Current INT reviews do not select one competing onset. The loader therefore uses the latest supplied other interval that crosses the wearer offset within the annotation tolerance. It records how many intervals were eligible so multi-competitor cases can be audited before the protocol is frozen.
Provisional EgoSpeak run
Threshold-free AP on the official test sessions can be run without fitting a test threshold:
python3 evaluate_egospeak_opportunity_events.py \
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
--run egospeak=results/egospeak_miniroada_floor_eval \
--sessions Session_1 Session_2 Session_3 \
--allow-provisional-llm-reviews \
--print-event-examples 10 \
--output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json
The output JSON records label coverage and every exclusion reason. The metrics CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors, causal source cutoffs, censor times, horizons, and wearer probabilities for timeline auditing.
Do not request matched-recall operating points on Sessions 1--3 without disjoint Sessions 4--12 calibration reviews and out-of-fold predictions. Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD cap, positive anchor-selection policy, and multiple-competitor handling using training-side data only.