Fix opportunity competing-risk evaluation
This commit is contained in:
@@ -1,5 +1,11 @@
|
||||
# EasyCom EgoSpeak three-level evaluation
|
||||
|
||||
> **Protocol erratum.** The dense and raw-boundary results in this record remain
|
||||
> valid. The opportunity-conditioned ENTRY/INT values were produced by the
|
||||
> superseded v1 loader, which could cross an intervening ENTRY competitor and
|
||||
> selected the latest onset in ambiguous multi-competitor INT cases. Do not cite
|
||||
> those opportunity values; rerun protocol v2 and create a new result record.
|
||||
|
||||
This record freezes the first Sessions 1--3 evaluation of the released
|
||||
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
|
||||
questions. The values below were produced on the remote experiment host; the
|
||||
@@ -15,7 +21,7 @@ prediction and per-event artifacts remain on that host and are not committed.
|
||||
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
|
||||
`ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
|
||||
INT claims.
|
||||
- Opportunity-conditioned behavior: one causally censored activity-derived
|
||||
- Opportunity-conditioned behavior: historical v1 activity-derived
|
||||
score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
|
||||
two-pass LLM-assisted reviews and have not been human reviewed.
|
||||
- No test-set threshold is fitted for any AP reported here. Session 4 is needed
|
||||
@@ -34,15 +40,16 @@ mean AP, respectively. These lower values do not contradict the dense result:
|
||||
dense AP rewards correct future state over sustained regions, whereas a raw
|
||||
boundary event is counted once and has many harder non-boundary comparisons.
|
||||
|
||||
On the provisional opportunity labels, ENTRY AP is **52.487%** for 163
|
||||
The superseded v1 run reported ENTRY AP of **52.487%** for 163
|
||||
positives and 335 hard negatives, versus 32.731% positive prevalence
|
||||
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
|
||||
and 109 hard negatives, essentially equal to its 20.438% prevalence. This is
|
||||
consistent with the output contract: wearer probability can rank many entry
|
||||
opportunities, but `1-p(wearer)` does not encode whether an interruption should
|
||||
cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5
|
||||
invalid/post-outcome cases; these exclusions require an audit before any final
|
||||
paper claim.
|
||||
and 109 hard negatives, essentially equal to its 20.438% prevalence. These
|
||||
values are retained only for provenance and must be replaced by a v2 rerun. The
|
||||
original pattern was consistent with the output contract: wearer probability
|
||||
can rank many entry opportunities, but `1-p(wearer)` does not encode whether an
|
||||
interruption should cause a yield. The ENTRY and INT loaders additionally
|
||||
exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit
|
||||
before any final paper claim.
|
||||
|
||||
## Relation to EgoSpeak Figure 7
|
||||
|
||||
@@ -73,4 +80,3 @@ Machine-readable curves and counts are in [`summary.json`](summary.json) and
|
||||
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
|
||||
protocol and reproduction command are documented in
|
||||
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
{
|
||||
"recorded_at": "2026-09-16",
|
||||
"source": "user-provided remote experiment output",
|
||||
"opportunity_protocol_status": "superseded_v1_rerun_required",
|
||||
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
|
||||
"dataset": "EasyCom",
|
||||
"sessions": ["Session_1", "Session_2", "Session_3"],
|
||||
"model": "EgoSpeak MiniROADA audio-visual released checkpoint",
|
||||
@@ -57,4 +59,3 @@
|
||||
"threshold_free": true,
|
||||
"session4_required_for_reported_metrics": false
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user