Add competing-risk evaluation and align ICASSP story

docs/easycom_competing_risk_floor_evaluation.md

@@ -0,0 +1,109 @@
+# EasyCom competing-risk floor-event evaluation
+
+This runbook evaluates whether a candidate model reduces **false wearer-entry
+triggers** and **false wearer-yield triggers** relative to a baseline. It reads
+the saved `pred/` and `valid/` arrays; no GPU is required.
+
+The protocol is implemented in `evaluate_floor_state_changes.py`. It is a
+timing-derived behavior evaluation, not semantic EOT/INT, addressee, or
+individual-speaker ground truth: EasyCom's `others` target is aggregate
+activity.
+
+## Event attribution
+
+At each horizon from 0.2 to 2.0 seconds, every eligible aggregate-other
+boundary is counted once.
+
+- **Response candidate:** `others` changes from active to inactive while the
+  wearer is inactive. A wearer onset is a response only if it occurs before
+  the next `others` onset.
+- **Yield candidate:** `others` changes from inactive to active while the
+  wearer is active. A wearer offset is a yield only if it occurs before the
+  next `others` offset.
+- The competing boundary itself is excluded. A simultaneous wearer transition
+  is therefore not assigned to the earlier candidate.
+- An observed wearer outcome before the recording ends is retained. An
+  unresolved candidate without its full horizon is right-censored and omitted.
+- Invalid future-score paths are omitted and counted separately.
+
+For each event, predictions made from the same source time are max-pooled only
+over the observable event window. This is a ranking score, not a calibrated
+event probability.
+
+## Confirmatory test command
+
+Select each model's operating threshold on training-side predictions, then
+apply that frozen threshold to official test Sessions 1--3. The run names in
+`--calibration-run` must exactly match the names in `--run`.
+
+```bash
+python3 evaluate_floor_state_changes.py \
+  --target-root /path/to/easycom_target_frame \
+  --run baseline=/path/to/baseline/test_result \
+  --run candidate=/path/to/candidate/test_result \
+  --calibration-run baseline=/path/to/baseline/calibration_result \
+  --calibration-run candidate=/path/to/candidate/calibration_result \
+  --calibration-sessions Session_12 \
+  --sessions Session_3 Session_2 Session_1 \
+  --matched-recalls 0.7 0.8 0.9 \
+  --midturn-pause-max-seconds 2.0 \
+  --print-event-timelines 3 \
+  --output reports/easycom_floor_competing_risk/test_summary.json
+```
+
+If the calibration and test predictions already coexist under each `--run`
+directory, omit `--calibration-run`. Official-test evaluation rejects missing
+calibration and rejects calibration on Sessions 1--3. The
+`--allow-in-sample-test-thresholds` option exists only for explicitly marked
+exploratory audits and must not be used for a paper result.
+
+Session 12 is shown as the current development convention. If the final model
+has been trained on Session 12, prefer out-of-fold training-side predictions
+or a pre-registered cross-fitting scheme; record exactly how calibration
+predictions were produced.
+
+## How to decide whether errors decreased
+
+The command writes:
+
+- `test_summary.json`: full protocol metadata, thresholds, counts, and metrics;
+- `test_summary_events.csv`: one row per run and horizon for the primary event
+  evaluation;
+- `test_summary_framewise.csv`: the older endpoint diagnostic, retained only
+  for secondary analysis.
+
+At the headline recall (normally 0.8), compare the following event-CSV fields:
+
+| Question | Primary field | Supporting checks |
+|---|---|---|
+| Did false entry decrease? | `no_response_trigger_rate_at_recall_0.8` | count, `response_recall_0.8`, response/no-response denominators |
+| Did false entry during a short other-speaker restart decrease? | `short_resume_trigger_rate_at_recall_0.8` | trigger count and subgroup count |
+| Did false yield/stop decrease? | `continue_false_stop_rate_at_recall_0.8` | count, `yield_recall_0.8`, yield/continue denominators |
+
+Lower is better for all three error rates. A lower error rate is not evidence
+of improvement if the candidate also obtains materially lower test response or
+yield recall. The requested recall selects the threshold on calibration data;
+`response_recall_0.8` and `yield_recall_0.8` report the recall actually achieved
+on test and may differ from 0.8.
+
+Before comparing models, confirm that their event denominators and invalid
+window counts match. Then report both absolute percentage-point change and
+relative error reduction, for example
+`(baseline_rate - candidate_rate) / baseline_rate`. Event AP is a useful
+threshold-free secondary result, but it does not replace the error-rate
+comparison at a pre-registered operating point.
+
+## Required manual audit before protocol freeze
+
+Use `--print-event-timelines 3` first, then inspect a larger random sample with
+an independent script or annotation view. For every printed timeline, verify:
+
+1. the anchor row is the intended aggregate-other boundary;
+2. no outcome at or after `C/excluded` is attributed to the earlier event;
+3. the fixed-source horizon score and `valid` mask line up with physical time;
+4. session/recording boundaries do not create transitions;
+5. right-censored and invalid counts are plausible.
+
+The current evaluator does not yet provide `third_party`, `addressed`, or
+`group_or_unknown` strata, nor false entries per minute. Those require a
+separately frozen addressee/scenario annotation contract and remain P0 TODOs.