wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revamp EasyCom floor event review pipeline

Browse Source
This commit is contained in:
Xin Wang
2026-09-14 18:43:58 +08:00
parent 7d9db4af58
commit 4f88806d56
11 changed files with 1081 additions and 965 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/easycom_event_review_tool.md
+70 -51
View File
@@ -1,16 +1,19 @@
# EasyCom EOT/INT review workspace
# EasyCom wearer-centric EOT/INT pre-label and review pipeline
`review_floor_events.py` is a local annotation workspace for reviewing the
manuscript-aligned regions produced by `prepare_easycom_eot_int_review.py`.
It can also open the older timing/transcript proxy reports from
`derive_easycom_interaction_events.py` and `audit_easycom_floor_events.py`, but
those older reports do not have the directionality or hard-negative region
contract required by the current manuscript. Raw EasyCom media stays read-only;
human decisions are stored separately as JSONL and CSV.
The workflow has three explicit stages. Speech activity timestamps are
read-only throughout; reviewed events never modify the dense 5 Hz activity
ground truth.
## Prepare manuscript-aligned candidates
```text
EasyCom participant speech intervals
-> rule candidates + structured timing evidence
-> DeepSeek semantic pre-label (optional but recommended)
-> human Accept / Relabel / Exclude / Second review
```
Develop the annotation guide on training-side Sessions 4--12:
## 1. Generate candidates
Develop the rules on training-side Sessions 4--12:
```bash
python3 prepare_easycom_eot_int_review.py \
@@ -19,24 +22,39 @@ python3 prepare_easycom_eot_int_review.py \
--output-dir reports/easycom_floor_event_candidates
```
For a one-event pilot, add `--max-candidates 1`. The generator follows the
manuscript direction exactly:
The generator emits two independent sources:
- EOT-response begins at an aggregate-other offset while the wearer is silent.
- INT-yield begins at an aggregate-other onset while the wearer is active.
- An observed wearer transition before the competing other transition is a
timing-positive candidate.
- A competing other transition, or a fully observed no-transition window, is a
timing hard-negative candidate.
- A source boundary without a complete outcome/window is marked for exclusion.
- every wearer `0->1` (`WEARER_ONSET`, `ENTRY_CANDIDATE`) and `1->0`
(`WEARER_OFFSET`, `YIELD_CANDIDATE`) transition;
- hard-negative regions that need no wearer transition:
`SAME_SPEAKER_PAUSE`, `OTHER_TO_OTHER`, `OTHER_BACKCHANNEL`, and
`NON_TAKEOVER_OVERLAP`.
Automatic labels remain candidates until independent human semantic review.
Wearer transitions are initially `unsure`; timing alone is not a semantic
positive label. Hard-negative scans are rule hints and still require AI and
human review. The JSONL retains candidate source, original event time,
participant-level activity, raw durations/gaps/overlap evidence, transcript,
and `Target_of_Speech` when present. `Target_of_Speech` is annotation assistance
only and cannot be a deployable model input.
## Optional DeepSeek suggestion
All heuristic durations are configurable rather than frozen paper rules:
DeepSeek suggestions are triage metadata and never count as a human label. The
script sends only the candidate transcript excerpt and timing fields. Inspect
the exact request without network access first:
```text
--relation-window-seconds
--same-speaker-pause-max-seconds
--other-handover-max-seconds
--short-vocalization-max-seconds
--continuation-min-seconds
```
Use `--max-candidates 1` for a one-event smoke test. `summary.json` records the
exact configuration and candidate counts. `easycom_session_timelines.json`
stores the read-only participant speech lanes used by the full-session web
timeline; it is not sent to DeepSeek.
## 2. Add DeepSeek pre-labels
First inspect exactly what would be uploaded:
```bash
python3 adjudicate_easycom_eot_int_candidates.py \
@@ -47,51 +65,52 @@ python3 adjudicate_easycom_eot_int_candidates.py \
--dry-run
```
After setting `DEEPSEEK_API_KEY`, make the authorized request:
After setting `DEEPSEEK_API_KEY` and obtaining authorization to upload the
candidate excerpts:
```bash
python3 adjudicate_easycom_eot_int_candidates.py \
--input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
--output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
--model deepseek-v4-flash \
--max-events 1 \
--allow-external-transcript-upload
```
Point the review server at the DeepSeek-enriched JSONL to show its suggestion
beside the video. Keep suggestions hidden from an initial independent human
pass if anchoring bias is a concern.
DeepSeek sees candidate source, structured timing, per-speaker activity, and
transcript together. It returns one of `POSITIVE`, `HARD_NEGATIVE`,
`NOT_APPLICABLE`, `EXCLUDE`, or `UNSURE`; positives additionally receive
`EOT_RESPONSE` or `INT_YIELD`, and hard negatives receive a named subtype. It
cannot modify timestamps. The output preserves both the original rule candidate
and the complete AI decision/provenance.
## Start the local workspace
## 3. Human review
Run the web workspace on the AI-enriched JSONL:
```bash
python3 review_floor_events.py \
--dataset-root /path/to/easycom \
--events reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
--events reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
--output reports/easycom_event_reviews/reviewer_1.jsonl
```
Open `http://127.0.0.1:8765`. The page resolves the source MP4 and transcript,
seeks to a six-second event context, and supports session/split/task/status
filtering.
Open `http://127.0.0.1:8765`. The page shows a session candidate overview,
participant activity near the event, video/audio, transcript, structured timing,
and the AI verdict/confidence/reason. The common path is one operation:
Each save atomically rewrites both JSONL and CSV. The review explicitly records
`positive`, `hard_negative`, `exclude`, or `uncertain`; the hard-negative
subtype; boundary validity; onset function; conversational routing; offset
outcome; three 200 ms anchor/region corrections; confidence; and reviewer ID.
- **Accept** keeps the AI verdict/type;
- **Relabel** expands verdict and subtype controls;
- **Exclude** requires a concrete data-quality reason;
- **Unsure / Second review** preserves the event for adjudication.
Boundary correction is hidden under an optional disclosure and is limited to
one 200 ms bin. The original timestamp is always retained. Each save atomically
rewrites JSONL and CSV while preserving the rule candidate, AI decision, human
decision, relabel flag, exclusion state, reviewer, and provenance.
## Freeze discipline
Develop the annotation guide and resolve category disagreements on training
Sessions 4--12. Freeze the guide before reviewing official test Sessions 1--3.
Do not use test reviews to change candidate windows, label categories, or the
R=.8 threshold calibration rule.
For paper-quality labels, use two independent reviewers with separate
`--output` files and adjudicate disagreements in a third output. A non-empty
reviewer ID is required on every saved row. DeepSeek is not one of the two
reviewers.
This workspace produces audited labels. It does not yet convert them into the
final committed-event matching metrics; that integration follows only after
the schema and real-data pilot are frozen.
Develop the guide and resolve disagreements only on Sessions 4--12. Freeze
categories and thresholds before reviewing official test Sessions 1--3. Use two
independent human review files plus a separate adjudication output for paper-
quality labels. DeepSeek is pre-label metadata, not an independent reviewer.