60 lines
2.8 KiB
Markdown
60 lines
2.8 KiB
Markdown
|
# EasyCom opportunity-conditioned ENTRY/YIELD evaluation
|
||
|
|
|
||
|
|
[`evaluate_egospeak_opportunity_events.py`](../evaluate_egospeak_opportunity_events.py)
|
||
|
|
uses LLM-assisted or human-reviewed semantic labels to define matched
|
||
|
|
conversational opportunities while scoring every model only from its original
|
||
|
|
EgoSpeak-compatible wearer probabilities. It does not train a semantic event
|
||
|
|
head.
|
||
|
|
|
||
|
|
## ENTRY
|
||
|
|
|
||
|
|
A positive must have a reviewer-selected preceding non-wearer release. Its
|
||
|
|
opportunity starts at that release and resolves at the reviewed wearer entry.
|
||
|
|
Core hard negatives start at the same type of non-wearer release and resolve
|
||
|
|
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
||
|
|
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
||
|
|
maximum projected wearer probability up to the competing resolution, capped at
|
||
|
|
2.0 seconds.
|
||
|
|
|
||
|
|
`ENTRY_WEARER_BACKCHANNEL` is excluded from headline ENTRY FPR. Predicting
|
||
|
|
wearer activity for a real backchannel is correct under EgoSpeak's dense target;
|
||
|
|
ENTRY-versus-backchannel remains a separate semantic-selectivity diagnostic.
|
||
|
|
|
||
|
|
## YIELD
|
||
|
|
|
||
|
|
An opportunity starts at a non-wearer onset during wearer speech. The score is
|
||
|
|
the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
||
|
|
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
||
|
|
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
||
|
|
|
||
|
|
Current INT reviews do not select one competing onset. The loader therefore
|
||
|
|
uses the latest supplied other interval that crosses the wearer offset within
|
||
|
|
the annotation tolerance. It records how many intervals were eligible so
|
||
|
|
multi-competitor cases can be audited before the protocol is frozen.
|
||
|
|
|
||
|
|
## Provisional EgoSpeak run
|
||
|
|
|
||
|
|
Threshold-free AP on the official test sessions can be run without fitting a
|
||
|
|
test threshold:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python3 evaluate_egospeak_opportunity_events.py \
|
||
|
|
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
|
||
|
|
--run egospeak=results/egospeak_miniroada_floor_eval \
|
||
|
|
--sessions Session_1 Session_2 Session_3 \
|
||
|
|
--allow-provisional-llm-reviews \
|
||
|
|
--print-event-examples 10 \
|
||
|
|
--output results/easycom_egospeak_opportunity_events/sessions1_3_provisional_summary.json
|
||
|
|
```
|
||
|
|
|
||
|
|
The output JSON records label coverage and every exclusion reason. The metrics
|
||
|
|
CSV contains threshold-free ENTRY/YIELD AP. The event CSV contains anchors,
|
||
|
|
causal source cutoffs, censor times, horizons, and wearer probabilities for
|
||
|
|
timeline auditing.
|
||
|
|
|
||
|
|
Do not request matched-recall operating points on Sessions 1--3 without
|
||
|
|
disjoint Sessions 4--12 calibration reviews and out-of-fold predictions.
|
||
|
|
Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
||
|
|
cap, positive anchor-selection policy, and multiple-competitor handling using
|
||
|
|
training-side data only.
|