icassp2027-experiment/results/easycom_egospeak_three_level_eval_2026-09-16/README.md

# EasyCom EgoSpeak three-level evaluation

> **Protocol erratum.** The dense and raw-boundary results in this record remain
> valid. The opportunity-conditioned ENTRY/INT values were produced by the
> superseded v1 loader, which could cross an intervening ENTRY competitor and
> selected the latest onset in ambiguous multi-competitor INT cases. Do not cite
> those opportunity values; rerun protocol v2 and create a new result record.

This record freezes the first Sessions 1--3 evaluation of the released
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
questions. The values below were produced on the remote experiment host; the
aggregate console output was copied into this record on 2026-09-16. Generated
prediction and per-event artifacts remain on that host and are not committed.

## Protocol and split

- Test split: EasyCom Sessions 1--3.
- Model: released EgoSpeak MiniROADA audio-visual checkpoint.
- Dense forecasting: framewise AP for background, wearer, and aggregate
  others at ten horizons from 0.2 to 2.0 seconds.
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
  `ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
  INT claims.
- Opportunity-conditioned behavior: historical v1 activity-derived
  score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
  two-pass LLM-assisted reviews and have not been human reviewed.
- No test-set threshold is fitted for any AP reported here. Session 4 is needed
  only for threshold-dependent operating points, which are intentionally not
  summarized as headline findings in this record.

## Findings

Dense forecasting reaches **74.124% mAP at 0.2 s** and **61.155% mean mAP**
over ten horizons. The wearer class alone reaches 63.750% AP at 0.2 s and
53.493% averaged over horizons. Restricting evaluation to causally valid rows
changes mean mAP by only +0.038 percentage points (61.193% versus 61.155%).

Raw wearer onset and offset anticipation obtain **24.491%** and **38.024%**
mean AP, respectively. These lower values do not contradict the dense result:
dense AP rewards correct future state over sustained regions, whereas a raw
boundary event is counted once and has many harder non-boundary comparisons.

The superseded v1 run reported ENTRY AP of **52.487%** for 163
positives and 335 hard negatives, versus 32.731% positive prevalence
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
and 109 hard negatives, essentially equal to its 20.438% prevalence. These
values are retained only for provenance and must be replaced by a v2 rerun. The
original pattern was consistent with the output contract: wearer probability
can rank many entry opportunities, but `1-p(wearer)` does not encode whether an
interruption should cause a yield. The ENTRY and INT loaders additionally
exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit
before any final paper claim.

## Relation to EgoSpeak Figure 7

Figure 7 of EgoSpeak plots a continuous target/wearer-speaking confidence over
ground-truth speaking regions. It therefore corresponds most directly to the
dense **wearer AP**, not to raw onset/offset AP or semantic ENTRY/YIELD AP. A
visually convincing dense curve can coexist with weak boundary or semantic
selectivity. The figure caption does not identify one forecast horizon, and it
uses the paper's Transformer variants rather than proving the aggregate result
of this MiniROADA run.

For our eventual qualitative figure, the continuous wearer curve can be kept,
but it should also show the causal opportunity anchor, event boundary,
competing-risk censor, and the single score assigned to each positive or hard
negative.

## Interpretation limits

- The three metric layers answer different questions and their similar numeric
  values must not be equated.
- Aggregate `others` activity cannot identify a particular speaker, addressee,
  semantic EOT, or interruption intent.
- The opportunity result is provisional until label/exclusion audit and human
  review; it is useful for baseline diagnosis, not a final confirmatory claim.
- This is one released checkpoint, not a multi-seed uncertainty estimate.

Machine-readable curves and counts are in [`summary.json`](summary.json) and
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
protocol and reproduction command are documented in
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).