icassp2027-experiment/docs/easycom_annotated_floor_evaluation.md

# EasyCom human-annotated fixed-horizon floor-event evaluation

This protocol evaluates saved EgoSpeak-layout dense predictions against
human-confirmed `WEARER_ENTRY`, `INT_YIELD`, and hard-negative opportunities.
It is separate from the timing-derived proxy in
[`evaluate_floor_state_changes.py`](../evaluate_floor_state_changes.py).

## Scoring contract

At each nominal 0.2--2.0 second horizon:

- a positive entry is one corrected 5 Hz behavior-anchor bin, scored by wearer
  activity probability;
- a positive INT-yield is one corrected 5 Hz behavior-anchor bin, scored by one
  minus wearer activity probability;
- a hard negative is one opportunity sample, scored by the maximum erroneous
  action probability over its annotated half-open region;
- a positive timestamp is never expanded into a tolerance window;
- an excluded or unreviewed candidate never enters scoring.

The prediction layout is `[target_time, class, future_horizon]`. Event frames
are rasterized from the original 20 Hz annotation with the same `frame // 4`
rule used by the EasyCom event derivation. A human `-0.2/0/+0.2` second boundary
correction shifts the resulting target bin; for hard negatives it shifts the
whole region.

The reported horizon is the nominal EgoSpeak forecast-bin horizon. Because an
acoustic boundary can occur anywhere inside a 200 ms target bin, do not
reinterpret it as an exact continuous-time lead without separately reporting
the quantization convention.

## Provisional LLM-only pipeline smoke

Before human adjudication is complete, unanimous two-pass LLM suggestions can
be materialized into the review-file schema for a pipeline smoke test:

```bash
python3 materialize_provisional_llm_reviews.py \
  --events reports/easycom_entry_int_session{1,2,3,4}_v14/easycom_eot_int_deepseek_official*.jsonl \
  --output reports/easycom_entry_int_sessions1_4_v14/easycom_eot_int_provisional_llm_reviews.jsonl \
  --acknowledge-provisional-llm-auto-accept
```

The converter skips pass disagreements, pending API failures, and candidates
without an LLM suggestion. It writes JSONL, CSV, and a summary JSON. Every
materialized row carries `review_source=PROVISIONAL_LLM_AUTO_ACCEPT` and
`human_reviewed=false`; the nested `human_status=REVIEWED` value exists only
for review-schema compatibility and must never be described as human review.
The discouraged `--include-nonunanimous` option is available only for debugging
the fallback path.

The evaluator rejects these rows by default. A provisional run requires an
explicit opt-in and marks its output protocol and semantic scope accordingly:

```bash
python3 evaluate_annotated_floor_events.py \
  --reviews reports/easycom_entry_int_sessions1_4_v14/easycom_eot_int_provisional_llm_reviews.jsonl \
  --run egospeak=results/egospeak_miniroada_floor_eval \
  --sessions Session_4 \
  --allow-provisional-llm-reviews \
  --print-event-examples 5 \
  --output results/easycom_egospeak_event_smoke/session4_summary.json
```

This output is suitable for alignment debugging and preliminary analysis, not
for a manuscript claim of human-adjudicated ground truth. Official test
Sessions 1--3 still require disjoint training-side calibration unless the
separate exploratory in-sample override is used.

## Development smoke run

Use only training-side sessions while the annotation and region rules are
being audited:

```bash
python3 evaluate_annotated_floor_events.py \
  --reviews reports/easycom_event_reviews/reviews.jsonl \
  --run egospeak=/path/to/egospeak/result \
  --sessions Session_12 \
  --matched-recalls 0.7 0.8 0.9 \
  --print-event-examples 3 \
  --output reports/easycom_annotated_floor_events/session12_summary.json
```

This selects thresholds on the same development events and labels the threshold
source as exploratory. It is not a confirmatory test command.

## Confirmatory test run

Thresholds must be selected independently for entry and INT at every horizon
from training-side out-of-fold predictions:

```bash
python3 evaluate_annotated_floor_events.py \
  --reviews reports/easycom_event_reviews/test_reviews.jsonl \
  --run baseline=/path/to/baseline/test_result \
  --run candidate=/path/to/candidate/test_result \
  --sessions Session_3 Session_2 Session_1 \
  --calibration-reviews reports/easycom_event_reviews/oof_train_reviews.jsonl \
  --calibration-run baseline=/path/to/baseline/oof_result \
  --calibration-run candidate=/path/to/candidate/oof_result \
  --calibration-sessions Session_4 Session_5 Session_6 Session_7 Session_8 Session_9 Session_10 Session_11 Session_12 \
  --matched-recalls 0.7 0.8 0.9 \
  --print-event-examples 3 \
  --output results/easycom_annotated_floor_events_YYYY-MM-DD/summary.json
```

The CLI rejects evaluation involving official Sessions 1--3 unless calibration
reviews and calibration sessions are supplied. The
`--allow-in-sample-thresholds` override is exploratory only.

Outputs:

- `summary.json`: protocol metadata, counts, ten horizons per family, AP and
  operating points;
- `summary_metrics.csv`: one row per run, family, horizon, and requested recall;
- `summary_events.csv`: one scored row per run, annotated opportunity, and
  horizon for alignment audits and paired analysis.

Before comparing models, verify identical positive, hard-negative, and invalid
counts. The headline manuscript operating point is the nominal 0.2 second bin
at matched recall 0.8; ten-horizon entry/INT AP and all horizon curves are
supporting results.

## Required audit before freezing

1. Print examples and verify the 20 Hz boundary, 5 Hz target row, horizon, and
   valid mask against the source timeline.
2. Confirm that every positive occupies one target row.
3. Confirm that each hard-negative region contributes one max-pooled score,
   not one negative per frame.
4. Audit continuous timeline segments without candidate hints to estimate
   candidate recall.
5. Freeze the annotation guide, region endpoints, boundary-correction policy,
   and out-of-fold calibration procedure before inspecting Sessions 1--3.

This evaluator does not emit committed actions or detection latency. Those
require a separately frozen online controller with threshold, persistence,
refractory period, matching interval, and competing-risk rules.