133 lines
6.3 KiB
Markdown
133 lines
6.3 KiB
Markdown
# EasyCom competing-risk floor-event evaluation
|
|
|
|
This runbook evaluates whether a candidate model reduces **false wearer-entry
|
|
triggers** and **false wearer-yield triggers** relative to a baseline. It reads
|
|
the saved `pred/` and `valid/` arrays; no GPU is required.
|
|
|
|
The protocol is implemented in `evaluate_floor_state_changes.py`. It is a
|
|
timing-derived behavior evaluation, not semantic EOT/INT, addressee, or
|
|
individual-speaker ground truth: EasyCom's `others` target is aggregate
|
|
activity.
|
|
|
|
Raw EgoSpeak MiniROADA predictions use a different source-indexed layout. Use
|
|
`prepare_egospeak_floor_eval.py` to align them and create an explicit valid
|
|
mask before including the GRU A+V baseline in the same comparison:
|
|
|
|
```bash
|
|
python3 prepare_egospeak_floor_eval.py \
|
|
--prediction-root /path/to/miniroad/raw/eval \
|
|
--target-root /path/to/easycom_target_frame \
|
|
--output-dir results/egospeak_miniroada_floor_eval \
|
|
--sessions Session_12 Session_3 Session_2 Session_1
|
|
```
|
|
|
|
The converter requires raw arrays shaped `[T-10,10,3]` and fails rather than
|
|
guessing if their shape, range, or target length differs. Do not transpose or
|
|
pad MiniROADA output by hand.
|
|
|
|
## Event attribution
|
|
|
|
At each horizon from 0.2 to 2.0 seconds, every eligible aggregate-other
|
|
boundary is counted once.
|
|
|
|
- **Response candidate:** `others` changes from active to inactive while the
|
|
wearer is inactive. A wearer onset is a response only if it occurs before
|
|
the next `others` onset.
|
|
- **Yield candidate:** `others` changes from inactive to active while the
|
|
wearer is active. A wearer offset is a yield only if it occurs before the
|
|
next `others` offset.
|
|
- The competing boundary itself is excluded. A simultaneous wearer transition
|
|
is therefore not assigned to the earlier candidate.
|
|
- An observed wearer outcome before the recording ends is retained. An
|
|
unresolved candidate without its full horizon is right-censored and omitted.
|
|
- Invalid future-score paths are omitted and counted separately.
|
|
|
|
For each event, predictions made from the same source time are max-pooled only
|
|
over the observable event window. This is a ranking score, not a calibrated
|
|
event probability.
|
|
|
|
The default `--decision-timing pre-boundary` uses the last row before the
|
|
aggregate-other transition and remains the primary anticipation protocol. The
|
|
optional `--decision-timing post-boundary` moves the decision source to the
|
|
first row after the transition. It is a separately identified reactive diagnostic,
|
|
not a replacement metric; see
|
|
[`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md).
|
|
|
|
## Confirmatory test command
|
|
|
|
Select each model's operating threshold on training-side predictions, then
|
|
apply that frozen threshold to official test Sessions 1--3. The run names in
|
|
`--calibration-run` must exactly match the names in `--run`.
|
|
|
|
```bash
|
|
python3 evaluate_floor_state_changes.py \
|
|
--target-root /path/to/easycom_target_frame \
|
|
--run baseline=/path/to/baseline/test_result \
|
|
--run candidate=/path/to/candidate/test_result \
|
|
--calibration-run baseline=/path/to/baseline/calibration_result \
|
|
--calibration-run candidate=/path/to/candidate/calibration_result \
|
|
--calibration-sessions Session_12 \
|
|
--sessions Session_3 Session_2 Session_1 \
|
|
--matched-recalls 0.7 0.8 0.9 \
|
|
--midturn-pause-max-seconds 2.0 \
|
|
--print-event-timelines 3 \
|
|
--output reports/easycom_floor_competing_risk/test_summary.json
|
|
```
|
|
|
|
If the calibration and test predictions already coexist under each `--run`
|
|
directory, omit `--calibration-run`. Official-test evaluation rejects missing
|
|
calibration and rejects calibration on Sessions 1--3. The
|
|
`--allow-in-sample-test-thresholds` option exists only for explicitly marked
|
|
exploratory audits and must not be used for a paper result.
|
|
|
|
Session 12 is shown as the current development convention. If the final model
|
|
has been trained on Session 12, prefer out-of-fold training-side predictions
|
|
or a pre-registered cross-fitting scheme; record exactly how calibration
|
|
predictions were produced.
|
|
|
|
## How to decide whether errors decreased
|
|
|
|
The command writes:
|
|
|
|
- `test_summary.json`: full protocol metadata, thresholds, counts, and metrics;
|
|
- `test_summary_events.csv`: one row per run and horizon for the primary event
|
|
evaluation;
|
|
- `test_summary_framewise.csv`: the older endpoint diagnostic, retained only
|
|
for secondary analysis.
|
|
|
|
At the headline recall (normally 0.8), compare the following event-CSV fields:
|
|
|
|
| Question | Primary field | Supporting checks |
|
|
|---|---|---|
|
|
| Did false entry decrease? | `no_response_trigger_rate_at_recall_0.8` | count, `response_recall_0.8`, response/no-response denominators |
|
|
| Did false entry during a short other-speaker restart decrease? | `short_resume_trigger_rate_at_recall_0.8` | trigger count and subgroup count |
|
|
| Did false yield/stop decrease? | `continue_false_stop_rate_at_recall_0.8` | count, `yield_recall_0.8`, yield/continue denominators |
|
|
|
|
Lower is better for all three error rates. A lower error rate is not evidence
|
|
of improvement if the candidate also obtains materially lower test response or
|
|
yield recall. The requested recall selects the threshold on calibration data;
|
|
`response_recall_0.8` and `yield_recall_0.8` report the recall actually achieved
|
|
on test and may differ from 0.8.
|
|
|
|
Before comparing models, confirm that their event denominators and invalid
|
|
window counts match. Then report both absolute percentage-point change and
|
|
relative error reduction, for example
|
|
`(baseline_rate - candidate_rate) / baseline_rate`. Event AP is a useful
|
|
threshold-free secondary result, but it does not replace the error-rate
|
|
comparison at a pre-registered operating point.
|
|
|
|
## Required manual audit before protocol freeze
|
|
|
|
Use `--print-event-timelines 3` first, then inspect a larger random sample with
|
|
an independent script or annotation view. For every printed timeline, verify:
|
|
|
|
1. the anchor row is the intended aggregate-other boundary;
|
|
2. no outcome at or after `C/excluded` is attributed to the earlier event;
|
|
3. the fixed-source horizon score and `valid` mask line up with physical time;
|
|
4. session/recording boundaries do not create transitions;
|
|
5. right-censored and invalid counts are plausible.
|
|
|
|
The current evaluator does not yet provide `third_party`, `addressed`, or
|
|
`group_or_unknown` strata, nor false entries per minute. Those require a
|
|
separately frozen addressee/scenario annotation contract and remain P0 TODOs.
|