Add EgoSpeak dense event proxy evaluation
This commit is contained in:
+2
-1
@@ -34,7 +34,8 @@ tests.
|
||||
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
|
||||
| [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md) | Separating boundary observability, event-objective mismatch and addressee information with remote commands |
|
||||
| [`easycom_event_review_tool.md`](easycom_event_review_tool.md) | Reviewing automatically mined EasyCom floor-event candidates against local video and transcripts |
|
||||
| [`easycom_annotated_floor_evaluation.md`](easycom_annotated_floor_evaluation.md) | Scoring human-confirmed point events and hard-negative opportunity regions at fixed EgoSpeak horizons |
|
||||
| [`easycom_egospeak_event_proxy.md`](easycom_egospeak_event_proxy.md) | Scoring the EgoSpeak wearer trajectory over equal-width causal decision regions as an activity-derived ENTRY/INT proxy |
|
||||
| [`easycom_annotated_floor_evaluation.md`](easycom_annotated_floor_evaluation.md) | Reproducing the legacy asymmetric point-positive/region-negative fixed-horizon diagnostic |
|
||||
|
||||
## Design history and implementation options
|
||||
|
||||
|
||||
@@ -1,4 +1,10 @@
|
||||
# EasyCom human-annotated fixed-horizon floor-event evaluation
|
||||
# Legacy EasyCom fixed-horizon floor-event diagnostic
|
||||
|
||||
> **Status:** retained for reproducibility, not the current manuscript event
|
||||
> protocol. Its one-bin positives and variable-duration hard-negative regions
|
||||
> are asymmetric. Use the equal-width activity-derived protocol in
|
||||
> [`easycom_egospeak_event_proxy.md`](easycom_egospeak_event_proxy.md) for the
|
||||
> current EgoSpeak baseline.
|
||||
|
||||
This protocol evaluates saved EgoSpeak-layout dense predictions against
|
||||
human-confirmed `WEARER_ENTRY`, `INT_YIELD`, and hard-negative opportunities.
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# EgoSpeak dense-trajectory event proxy
|
||||
|
||||
`evaluate_egospeak_event_proxy.py` provides a deliberately limited bridge
|
||||
between EgoSpeak-compatible dense future-activity predictions and reviewed
|
||||
wearer ENTRY/INT opportunities. It does **not** claim that wearer voice
|
||||
activity is a semantic floor-event detector.
|
||||
|
||||
## Score
|
||||
|
||||
For every positive event, the default decision region contains the three 5-Hz
|
||||
source rows immediately before its corrected behavior anchor. An independently
|
||||
mined hard negative uses the first three rows of its labeled opportunity. A raw
|
||||
wearer onset/offset relabeled as a hard negative uses the same pre-boundary
|
||||
region as a positive. Regions shorter than 0.6 seconds are excluded and
|
||||
counted; longer annotation evidence is not scored.
|
||||
|
||||
At each source row, the script recovers the target-aligned wearer trajectory
|
||||
at 0.6, 0.8, ..., 2.0 seconds. ENTRY uses its mean and INT uses one minus its
|
||||
mean. Max-pooling over the three decision rows produces one score per reviewed
|
||||
opportunity. The output must be called an `activity-derived event proxy`.
|
||||
|
||||
## Threshold-free provisional run
|
||||
|
||||
This reports AP without fitting thresholds on Sessions 1--3:
|
||||
|
||||
```bash
|
||||
python3 evaluate_egospeak_event_proxy.py \
|
||||
--reviews reports/easycom_entry_int_sessions1_4_v16/easycom_eot_int_provisional_llm_reviews.jsonl \
|
||||
--run egospeak=results/egospeak_miniroada_floor_eval \
|
||||
--sessions Session_1 Session_2 Session_3 \
|
||||
--allow-provisional-llm-reviews \
|
||||
--print-event-examples 5 \
|
||||
--output results/easycom_egospeak_event_proxy/sessions1_3_provisional_summary.json
|
||||
```
|
||||
|
||||
The command writes JSON, a compact metrics CSV, and an event CSV containing the
|
||||
three decision-frame scores and the winning selected-horizon wearer trajectory.
|
||||
|
||||
## Recall-matched operating points
|
||||
|
||||
`--matched-recalls` on official test sessions additionally requires disjoint
|
||||
Sessions 4--12 reviews and predictions via `--calibration-reviews`,
|
||||
`--calibration-sessions`, and optionally `--calibration-run`. Do not use
|
||||
`--allow-in-sample-thresholds` for reported results.
|
||||
|
||||
The default 0.6-second decision window and 0.6--2.0-second trajectory range
|
||||
remain protocol choices to freeze using training-side data. Report shorter or
|
||||
longer windows only as predeclared sensitivity analyses.
|
||||
Reference in New Issue
Block a user