77 lines
4.0 KiB
Markdown
77 lines
4.0 KiB
Markdown
|
# EasyCom EgoSpeak three-level evaluation
|
||
|
|
|
||
|
|
This record freezes the first Sessions 1--3 evaluation of the released
|
||
|
|
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
|
||
|
|
questions. The values below were produced on the remote experiment host; the
|
||
|
|
aggregate console output was copied into this record on 2026-09-16. Generated
|
||
|
|
prediction and per-event artifacts remain on that host and are not committed.
|
||
|
|
|
||
|
|
## Protocol and split
|
||
|
|
|
||
|
|
- Test split: EasyCom Sessions 1--3.
|
||
|
|
- Model: released EgoSpeak MiniROADA audio-visual checkpoint.
|
||
|
|
- Dense forecasting: framewise AP for background, wearer, and aggregate
|
||
|
|
others at ten horizons from 0.2 to 2.0 seconds.
|
||
|
|
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
|
||
|
|
`ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
|
||
|
|
INT claims.
|
||
|
|
- Opportunity-conditioned behavior: one causally censored activity-derived
|
||
|
|
score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
|
||
|
|
two-pass LLM-assisted reviews and have not been human reviewed.
|
||
|
|
- No test-set threshold is fitted for any AP reported here. Session 4 is needed
|
||
|
|
only for threshold-dependent operating points, which are intentionally not
|
||
|
|
summarized as headline findings in this record.
|
||
|
|
|
||
|
|
## Findings
|
||
|
|
|
||
|
|
Dense forecasting reaches **74.124% mAP at 0.2 s** and **61.155% mean mAP**
|
||
|
|
over ten horizons. The wearer class alone reaches 63.750% AP at 0.2 s and
|
||
|
|
53.493% averaged over horizons. Restricting evaluation to causally valid rows
|
||
|
|
changes mean mAP by only +0.038 percentage points (61.193% versus 61.155%).
|
||
|
|
|
||
|
|
Raw wearer onset and offset anticipation obtain **24.491%** and **38.024%**
|
||
|
|
mean AP, respectively. These lower values do not contradict the dense result:
|
||
|
|
dense AP rewards correct future state over sustained regions, whereas a raw
|
||
|
|
boundary event is counted once and has many harder non-boundary comparisons.
|
||
|
|
|
||
|
|
On the provisional opportunity labels, ENTRY AP is **52.487%** for 163
|
||
|
|
positives and 335 hard negatives, versus 32.731% positive prevalence
|
||
|
|
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
|
||
|
|
and 109 hard negatives, essentially equal to its 20.438% prevalence. This is
|
||
|
|
consistent with the output contract: wearer probability can rank many entry
|
||
|
|
opportunities, but `1-p(wearer)` does not encode whether an interruption should
|
||
|
|
cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5
|
||
|
|
invalid/post-outcome cases; these exclusions require an audit before any final
|
||
|
|
paper claim.
|
||
|
|
|
||
|
|
## Relation to EgoSpeak Figure 7
|
||
|
|
|
||
|
|
Figure 7 of EgoSpeak plots a continuous target/wearer-speaking confidence over
|
||
|
|
ground-truth speaking regions. It therefore corresponds most directly to the
|
||
|
|
dense **wearer AP**, not to raw onset/offset AP or semantic ENTRY/YIELD AP. A
|
||
|
|
visually convincing dense curve can coexist with weak boundary or semantic
|
||
|
|
selectivity. The figure caption does not identify one forecast horizon, and it
|
||
|
|
uses the paper's Transformer variants rather than proving the aggregate result
|
||
|
|
of this MiniROADA run.
|
||
|
|
|
||
|
|
For our eventual qualitative figure, the continuous wearer curve can be kept,
|
||
|
|
but it should also show the causal opportunity anchor, event boundary,
|
||
|
|
competing-risk censor, and the single score assigned to each positive or hard
|
||
|
|
negative.
|
||
|
|
|
||
|
|
## Interpretation limits
|
||
|
|
|
||
|
|
- The three metric layers answer different questions and their similar numeric
|
||
|
|
values must not be equated.
|
||
|
|
- Aggregate `others` activity cannot identify a particular speaker, addressee,
|
||
|
|
semantic EOT, or interruption intent.
|
||
|
|
- The opportunity result is provisional until label/exclusion audit and human
|
||
|
|
review; it is useful for baseline diagnosis, not a final confirmatory claim.
|
||
|
|
- This is one released checkpoint, not a multi-seed uncertainty estimate.
|
||
|
|
|
||
|
|
Machine-readable curves and counts are in [`summary.json`](summary.json) and
|
||
|
|
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
|
||
|
|
protocol and reproduction command are documented in
|
||
|
|
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).
|
||
|
|
|