Add manuscript-aligned EasyCom EOT INT review workflow

This commit is contained in:
Xin Wang
2026-09-14 16:29:25 +08:00
parent f4a7c01552
commit 7d9db4af58
12 changed files with 3112 additions and 0 deletions
+1
View File
@@ -33,6 +33,7 @@ tests.
| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
| [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md) | Separating boundary observability, event-objective mismatch and addressee information with remote commands |
| [`easycom_event_review_tool.md`](easycom_event_review_tool.md) | Reviewing automatically mined EasyCom floor-event candidates against local video and transcripts |
## Design history and implementation options
+97
View File
@@ -0,0 +1,97 @@
# EasyCom EOT/INT review workspace
`review_floor_events.py` is a local annotation workspace for reviewing the
manuscript-aligned regions produced by `prepare_easycom_eot_int_review.py`.
It can also open the older timing/transcript proxy reports from
`derive_easycom_interaction_events.py` and `audit_easycom_floor_events.py`, but
those older reports do not have the directionality or hard-negative region
contract required by the current manuscript. Raw EasyCom media stays read-only;
human decisions are stored separately as JSONL and CSV.
## Prepare manuscript-aligned candidates
Develop the annotation guide on training-side Sessions 4--12:
```bash
python3 prepare_easycom_eot_int_review.py \
--dataset-root /path/to/easycom \
--sessions Session_4 Session_5 Session_6 Session_7 Session_8 Session_9 Session_10 Session_11 Session_12 \
--output-dir reports/easycom_floor_event_candidates
```
For a one-event pilot, add `--max-candidates 1`. The generator follows the
manuscript direction exactly:
- EOT-response begins at an aggregate-other offset while the wearer is silent.
- INT-yield begins at an aggregate-other onset while the wearer is active.
- An observed wearer transition before the competing other transition is a
timing-positive candidate.
- A competing other transition, or a fully observed no-transition window, is a
timing hard-negative candidate.
- A source boundary without a complete outcome/window is marked for exclusion.
Automatic labels remain candidates until independent human semantic review.
## Optional DeepSeek suggestion
DeepSeek suggestions are triage metadata and never count as a human label. The
script sends only the candidate transcript excerpt and timing fields. Inspect
the exact request without network access first:
```bash
python3 adjudicate_easycom_eot_int_candidates.py \
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
--model deepseek-v4-flash \
--max-events 1 \
--dry-run
```
After setting `DEEPSEEK_API_KEY`, make the authorized request:
```bash
python3 adjudicate_easycom_eot_int_candidates.py \
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
--model deepseek-v4-flash \
--max-events 1 \
--allow-external-transcript-upload
```
Point the review server at the DeepSeek-enriched JSONL to show its suggestion
beside the video. Keep suggestions hidden from an initial independent human
pass if anchoring bias is a concern.
## Start the local workspace
```bash
python3 review_floor_events.py \
--dataset-root /path/to/easycom \
--events reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
--output reports/easycom_event_reviews/reviewer_1.jsonl
```
Open `http://127.0.0.1:8765`. The page resolves the source MP4 and transcript,
seeks to a six-second event context, and supports session/split/task/status
filtering.
Each save atomically rewrites both JSONL and CSV. The review explicitly records
`positive`, `hard_negative`, `exclude`, or `uncertain`; the hard-negative
subtype; boundary validity; onset function; conversational routing; offset
outcome; three 200 ms anchor/region corrections; confidence; and reviewer ID.
## Freeze discipline
Develop the annotation guide and resolve category disagreements on training
Sessions 4--12. Freeze the guide before reviewing official test Sessions 1--3.
Do not use test reviews to change candidate windows, label categories, or the
R=.8 threshold calibration rule.
For paper-quality labels, use two independent reviewers with separate
`--output` files and adjudicate disagreements in a third output. A non-empty
reviewer ID is required on every saved row. DeepSeek is not one of the two
reviewers.
This workspace produces audited labels. It does not yet convert them into the
final committed-event matching metrics; that integration follows only after
the schema and real-data pilot are frozen.