Fix opportunity competing-risk evaluation
This commit is contained in:
@@ -1,5 +1,12 @@
|
||||
# EasyCom Omni floor-event screening
|
||||
|
||||
> **Protocol erratum.** Dense and raw-boundary metrics remain valid. All
|
||||
> opportunity-conditioned ENTRY/INT values below used the superseded v1 loader,
|
||||
> which could cross an intervening ENTRY competitor and selected the latest
|
||||
> onset in ambiguous multi-competitor INT cases. Retain them for provenance
|
||||
> only; do not use them for model selection or manuscript claims before a v2
|
||||
> rerun.
|
||||
|
||||
This record freezes the first comparison of EgoSpeak, Omni `G+A`, Omni
|
||||
`G+A+pooled-V`, and the three-seed current-frame F1 Frame-set Attention screen
|
||||
on EasyCom Sessions 1--3. Values were produced on the remote experiment host
|
||||
@@ -11,9 +18,8 @@ CSVs remain on that host and are not committed here.
|
||||
1. Dense future activity uses causal-valid framewise AP/mAP over ten horizons.
|
||||
2. Raw wearer endpoint and aggregate-other-boundary AP are threshold-free
|
||||
timing diagnostics; they are not semantic EOT/INT labels.
|
||||
3. Opportunity-conditioned ENTRY and INT/YIELD AP assigns one causally
|
||||
censored activity-derived score to each provisional LLM-assisted reviewed
|
||||
opportunity.
|
||||
3. Opportunity-conditioned ENTRY and INT/YIELD AP is a historical v1 screen
|
||||
that requires rerunning with the corrected v2 competing-risk protocol.
|
||||
|
||||
All opportunity runs use identical denominators: 163 ENTRY positives and 335
|
||||
hard negatives, with 81 invalid/post-outcome cases; and 28 INT positives and
|
||||
@@ -39,7 +45,7 @@ from EgoSpeak to G+A to G+A+pooled-V as follows: onset AP
|
||||
offset/yield-timing gain comes from G+A; pooled visual input adds almost
|
||||
nothing to raw transition timing.
|
||||
|
||||
## Interpretation
|
||||
## Historical v1 interpretation
|
||||
|
||||
- G+A strongly improves dense forecasting and generic wearer-offset/yield
|
||||
timing over EgoSpeak, but not reviewed semantic opportunities. In particular,
|
||||
@@ -77,4 +83,3 @@ nothing to raw transition timing.
|
||||
Exact aggregate values are stored in [`summary.json`](summary.json) and
|
||||
[`metrics.csv`](metrics.csv). The earlier EgoSpeak-only record is
|
||||
[`easycom_egospeak_three_level_eval_2026-09-16`](../easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
{
|
||||
"recorded_at": "2026-09-16",
|
||||
"source": "user-provided remote experiment output",
|
||||
"opportunity_protocol_status": "superseded_v1_rerun_required",
|
||||
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
|
||||
"dataset": "EasyCom",
|
||||
"test_sessions": ["Session_1", "Session_2", "Session_3"],
|
||||
"scale": "percent",
|
||||
@@ -82,4 +84,3 @@
|
||||
"threshold_free_metrics_only": true,
|
||||
"matched_pooled_control_available": false
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user