icassp2027-experiment/docs/easycom_event_review_tool.md

# EasyCom wearer-entry/INT pre-label and review pipeline

The workflow has three explicit stages. Speech activity timestamps are
read-only throughout; reviewed events never modify the dense 5 Hz activity
ground truth.

```text
EasyCom participant speech intervals
  -> floor-episode grouping + rule candidates + structured timing evidence
  -> DeepSeek semantic pre-label (optional but recommended)
  -> human Accept / Relabel / Exclude
```

## 1. Generate candidates

Develop the rules on training-side Sessions 4--12:

```bash
python3 prepare_easycom_eot_int_review.py \
  --dataset-root /path/to/easycom \
  --sessions Session_4 Session_5 Session_6 Session_7 Session_8 Session_9 Session_10 Session_11 Session_12 \
  --output-dir reports/easycom_floor_event_candidates
```

The generator emits two independent files:

- `easycom_eot_int_candidates.jsonl` (legacy-compatible filename) contains every
  floor-level wearer `0->1` behavior anchor (`WEARER_ONSET`,
  `WEARER_ENTRY_CANDIDATE`), only episode-ending wearer
  `1->0` boundaries where a new non-wearer onset satisfies
  `wearer_floor_start < other_onset < wearer_offset` and the same activity
  crosses the offset (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
  regions that need no wearer transition:
  `SAME_SPEAKER_PAUSE`, `OTHER_TO_OTHER`, `OTHER_BACKCHANNEL`, and
  `NON_TAKEOVER_OVERLAP`;
- `easycom_eot_int_controls.jsonl` contains short within-turn raw onsets
  (`WEARER_RESUME`) and wearer offsets that fail
  the strict INT ordering/crossing prerequisite (`WEARER_OFFSET_CONTROL`). They
  are retained for audits, but are not sent to DeepSeek or placed in the
  primary human-review queue.

A raw wearer onset is merged into its preceding floor episode only when its
preceding silence is at most 0.8 seconds and no plausible sustained non-wearer
takeover occurs in that gap. Short feedback-like lexical or nonverbal activity
is recorded but does not by itself split the episode. A non-feedback other
contribution that occupies at least `continuation_min_seconds` of the gap or
continues across the resumed onset conservatively preserves a new wearer-entry
candidate. Every decision is stored in `temporal_evidence`, including the raw
gap, feedback IDs, competing IDs, threshold, and `floor_episode_merged` flag.

Entry construction retains every floor-level wearer onset, then searches backward
for preceding non-wearer releases as optional context. A release is an admissible option only when the
wearer stays silent and no new non-wearer onset or already-active other floor
invalidates the interval before the wearer onset. Silence length alone does not
invalidate an option. Each wearer onset produces exactly one primary row. A
missing release does not demote an entry to a control: an entry may occur from
silence or while another speaker is active. `entry_context_timing` records
`AFTER_OTHER_RELEASE`, `OTHER_ACTIVE`, or `FROM_SILENCE_OR_UNRESOLVED` as
metadata. When present, `opportunity_options` contains the only release anchors
that DeepSeek or a human may optionally select; neither may invent a timestamp.

Floor-level wearer onsets and strictly qualified offsets carry the rule-stage value
`candidate`; timing alone is not a semantic positive label. Deterministic rows
outside the primary queue carry `control`, never a semantic verdict. The INT prerequisite is
`wearer_floor_start < other.start < wearer_offset < other.end`, with a
configurable one-frame (50 ms) tolerance only at the offset boundary.
Pre-existing other activity and raw offsets inside a merged wearer episode stay
in the controls file. Hard-negative scans are rule hints and still require AI
and human review. The JSONL retains candidate source, original event time,
participant-level activity, raw durations/gaps/overlap evidence, transcript,
and `Target_of_Speech` when present. `Target_of_Speech` is annotation assistance
only and cannot be a deployable model input.

All heuristic durations are configurable rather than frozen paper rules:

```text
--relation-window-seconds
--same-speaker-pause-max-seconds
--other-handover-max-seconds
--short-vocalization-max-seconds
--continuation-min-seconds
--int-offset-tolerance-seconds
--wearer-resume-merge-max-seconds
```

Use `--max-candidates 1` for a one-event smoke test. `summary.json` records the
exact configuration and candidate counts. `easycom_session_timelines.json`
stores the read-only participant speech lanes used by the full-session web
timeline; it is not sent to DeepSeek.

## 2. Add DeepSeek pre-labels

Prompt v11 first reconstructs the exact participant-resolved chronology and
then uses transcript content and `Target_of_Speech` to determine conversational
function. It treats the EasyCom transcript marker `[L]` as laughter-only evidence,
not as a lexical floor take. Wearer laughter during another participant's
retained turn is directed toward `ENTRY_WEARER_BACKCHANNEL`; non-wearer laughter
overlapping a retained wearer turn is directed toward `INT_OTHER_BACKCHANNEL`.
Ambiguous laughter falls back to `EXCLUDE`, and `[L]` alone is neither evidence
of a floor take nor evidence of a visible smile. Preceding release options are
optional entry metadata, whereas strict INT ordering remains a deterministic
retrieval prerequisite. A positive `WEARER_ENTRY` may optionally select one
supplied release; entries from silence or during other activity use `null`.

`needs_human_review` is not requested from DeepSeek: every pre-label enters the
human queue by protocol, so the adjudicator assigns that boolean locally. This
prevents a string-valued model field from invalidating and retrying an otherwise
valid batch.

First inspect exactly what would be uploaded:

```bash
python3 adjudicate_easycom_eot_int_candidates.py \
  --input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
  --output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
  --model deepseek-v4-flash \
  --max-events 1 \
  --dry-run
```

After setting `DEEPSEEK_API_KEY` and obtaining authorization to upload the
candidate excerpts:

```bash
python3 adjudicate_easycom_eot_int_candidates.py \
  --input reports/easycom_floor_event_candidates/easycom_eot_int_candidates.jsonl \
  --output reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
  --model deepseek-v4-flash \
  --allow-external-transcript-upload
```

DeepSeek sees candidate source, structured timing, per-speaker activity, and
transcript together. It returns one of `POSITIVE`, `HARD_NEGATIVE`, or
`EXCLUDE`; positives additionally receive
`WEARER_ENTRY` or `INT_YIELD`, and hard negatives receive a named subtype. It
cannot modify timestamps. The output preserves the original rule candidate,
the selected preceding-release UID when applicable, and the complete AI
decision/provenance. `EXCLUDE` is the sole non-scoring outcome. It covers
out-of-scope non-events (including a normal wearer completion that merely
coincides with feedback), insufficient evidence, unresolvable data quality, or
disagreement between the two model passes.

Hard negatives use behavior-specific subtypes. ENTRY distinguishes a same-other
mid-turn pause, a different-other transition, and a wearer backchannel that
does not take the floor. INT first distinguishes an other backchannel, then a
non-backchannel overlap after which the wearer demonstrably holds or resumes
the floor, and finally another interpretable overlap that fails to become a
sustained takeover. `OTHER_HARD_NEGATIVE` is reserved for valid false-positive
opportunities outside these named behaviors; it is never an uncertainty
fallback. A short but complete answer to an addressed question remains a
positive wearer entry; duration alone and later resumption by another speaker
do not turn it into a wearer backchannel.

During live adjudication, the CLI displays a dependency-free progress bar over
all candidate/pass work, including the current pass, batch, percentage, and
elapsed time. Redirected logs emit periodic progress lines instead of carriage
return animation.

As defense in depth, the adjudicator also detects and skips non-crossing
`WEARER_OFFSET` rows from older v2 candidate files before making any API call.

## 3. Human review

Run the web workspace on the AI-enriched JSONL:

```bash
python3 review_floor_events.py \
  --dataset-root /path/to/easycom \
  --events reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
  --output reports/easycom_event_reviews/reviews.jsonl
```

Open `http://127.0.0.1:8765`. The page shows a session candidate overview,
participant activity near the event, video/audio, transcript, structured timing,
and the AI verdict/confidence/reason. A decision guide first states the behavioral
question, then presents the exact AI verdict/subtype with a plain-language
meaning; the Accept control repeats that exact label so its effect is explicit.
The common path is one operation:

- **Accept** keeps the AI verdict/type and optional preceding-release metadata;
- **Relabel** expands verdict and subtype controls; a positive entry may
  optionally select one of the supplied release anchors;
- **Exclude** requires a concrete out-of-scope or data-quality reason.

The current primary-review UI deliberately has no second-review action. Older
saved `second_review` records remain readable for backward compatibility.

Boundary correction is hidden under an optional disclosure and is limited to
one 200 ms bin. The original timestamp is always retained. Each save atomically
rewrites JSONL and CSV while preserving the rule candidate, AI decision, human
decision, relabel flag, exclusion state, confidence, and provenance. The current
UI does not ask for a reviewer name; the export retains an empty `reviewer` field
only for compatibility with older review files.

The primary review queue also filters legacy non-crossing `WEARER_OFFSET` rows
and explicit `WEARER_RESUME` controls. To audit the controls intentionally,
pass `--events .../easycom_eot_int_controls.jsonl --include-controls` and use a
separate review output file. The old `--include-offset-controls` spelling
remains an alias.

Protocol wording for the paper appendix:

> Raw wearer speech onsets are first grouped into floor-level speech episodes
> by merging short within-turn resumptions that are not separated by a
> competing floor takeover. Every resulting floor-level 0→1 transition is
> proposed as a wearer-entry candidate; a preceding non-wearer release is
> retained as context rather than required for positive status.

> The original speech intervals are preserved unchanged for dense forecasting
> and auditability; this grouping is used only for behavioral event annotation.

## Freeze discipline

Develop the guide and resolve disagreements only on Sessions 4--12. Freeze
categories and thresholds before reviewing official test Sessions 1--3. Use two
independent human review files plus a separate adjudication output for paper-
quality labels. DeepSeek is pre-label metadata, not an independent reviewer.