wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update llm assisted annotation

Browse Source
This commit is contained in:
Xin Wang
2026-09-14 23:04:21 +08:00
parent 068b7ab9e6
commit 06b5e6c385
12 changed files with 1348 additions and 238 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/easycom_event_review_tool.md
+91 -15
View File
@@ -6,9 +6,9 @@ ground truth.
```text
EasyCom participant speech intervals
-> rule candidates + structured timing evidence
-> floor-episode grouping + rule candidates + structured timing evidence
-> DeepSeek semantic pre-label (optional but recommended)
-> human Accept / Relabel / Exclude / Second review
-> human Accept / Relabel / Exclude
```
## 1. Generate candidates
@@ -22,17 +22,46 @@ python3 prepare_easycom_eot_int_review.py \
--output-dir reports/easycom_floor_event_candidates
```
The generator emits two independent sources:
The generator emits two independent files:
- every wearer `0->1` (`WEARER_ONSET`, `ENTRY_CANDIDATE`) and `1->0`
(`WEARER_OFFSET`, `YIELD_CANDIDATE`) transition;
- hard-negative regions that need no wearer transition:
- `easycom_eot_int_candidates.jsonl` contains EOT opportunities that pair a
non-wearer release with a later floor-level wearer `0->1` behavior anchor
(`WEARER_ONSET`, `EOT_OPPORTUNITY_CANDIDATE`), only episode-ending wearer
`1->0` boundaries where a new non-wearer onset satisfies
`wearer_floor_start < other_onset < wearer_offset` and the same activity
crosses the offset (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
regions that need no wearer transition:
`SAME_SPEAKER_PAUSE`, `OTHER_TO_OTHER`, `OTHER_BACKCHANNEL`, and
`NON_TAKEOVER_OVERLAP`.
`NON_TAKEOVER_OVERLAP`;
- `easycom_eot_int_controls.jsonl` contains short within-turn raw onsets
(`WEARER_RESUME`), floor-level wearer entries with no surviving preceding
release opportunity (`WEARER_ONSET_CONTROL`), and wearer offsets that fail
the strict INT ordering/crossing prerequisite (`WEARER_OFFSET_CONTROL`). They
are retained for audits, but are not sent to DeepSeek or placed in the
primary human-review queue.
Wearer transitions are initially `unsure`; timing alone is not a semantic
positive label. Hard-negative scans are rule hints and still require AI and
human review. The JSONL retains candidate source, original event time,
A raw wearer onset is merged into its preceding floor episode only when its
preceding silence is at most 0.8 seconds and no plausible sustained non-wearer
takeover occurs in that gap. Short feedback-like lexical or nonverbal activity
is recorded but does not by itself split the episode. A non-feedback other
contribution that occupies at least `continuation_min_seconds` of the gap or
continues across the resumed onset conservatively preserves a new wearer-entry
candidate. Every decision is stored in `temporal_evidence`, including the raw
gap, feedback IDs, competing IDs, threshold, and `floor_episode_merged` flag.
After each non-wearer release, EOT construction scans forward until a
floor-level wearer onset or a competing non-wearer onset. Silence length alone
does not invalidate the opportunity. Each primary EOT candidate stores both
`opportunity_anchor_seconds` and `behavior_anchor_seconds`, plus their response
delay.
Paired wearer onsets and strictly qualified offsets are initially `unsure`;
timing alone is not a semantic positive label. The INT prerequisite is
`wearer_floor_start < other.start < wearer_offset < other.end`, with a
configurable one-frame (50 ms) tolerance only at the offset boundary.
Pre-existing other activity and raw offsets inside a merged wearer episode stay
in the controls file. Hard-negative scans are rule hints and still require AI
and human review. The JSONL retains candidate source, original event time,
participant-level activity, raw durations/gaps/overlap evidence, transcript,
and `Target_of_Speech` when present. `Target_of_Speech` is annotation assistance
only and cannot be a deployable model input.
@@ -45,6 +74,8 @@ All heuristic durations are configurable rather than frozen paper rules:
--other-handover-max-seconds
--short-vocalization-max-seconds
--continuation-min-seconds
--int-offset-tolerance-seconds
--wearer-resume-merge-max-seconds
```
Use `--max-candidates 1` for a one-event smoke test. `summary.json` records the
@@ -54,6 +85,20 @@ timeline; it is not sent to DeepSeek.
## 2. Add DeepSeek pre-labels
Prompt v6 treats the EasyCom transcript marker `[L]` as laughter-only evidence,
not as a lexical floor take. Wearer laughter during another participant's
retained turn is directed toward `EOT_WEARER_BACKCHANNEL`; non-wearer laughter
overlapping a retained wearer turn is directed toward `INT_OTHER_BACKCHANNEL`.
Ambiguous laughter remains `UNSURE`, and `[L]` alone is neither an exclusion
reason nor evidence of a visible smile. The prompt also treats the two EOT
anchors and strict INT ordering as deterministic prerequisites rather than
asking the model to reconstruct them.
`needs_human_review` is not requested from DeepSeek: every pre-label enters the
human queue by protocol, so the adjudicator assigns that boolean locally. This
prevents a string-valued model field from invalidating and retrying an otherwise
valid batch.
First inspect exactly what would be uploaded:
```bash
@@ -83,6 +128,14 @@ transcript together. It returns one of `POSITIVE`, `HARD_NEGATIVE`,
cannot modify timestamps. The output preserves both the original rule candidate
and the complete AI decision/provenance.
During live adjudication, the CLI displays a dependency-free progress bar over
all candidate/pass work, including the current pass, batch, percentage, and
elapsed time. Redirected logs emit periodic progress lines instead of carriage
return animation.
As defense in depth, the adjudicator also detects and skips non-crossing
`WEARER_OFFSET` rows from older v2 candidate files before making any API call.
## 3. Human review
Run the web workspace on the AI-enriched JSONL:
@@ -91,22 +144,45 @@ Run the web workspace on the AI-enriched JSONL:
python3 review_floor_events.py \
--dataset-root /path/to/easycom \
--events reports/easycom_floor_event_candidates/easycom_eot_int_deepseek.jsonl \
--output reports/easycom_event_reviews/reviewer_1.jsonl
--output reports/easycom_event_reviews/reviews.jsonl
```
Open `http://127.0.0.1:8765`. The page shows a session candidate overview,
participant activity near the event, video/audio, transcript, structured timing,
and the AI verdict/confidence/reason. The common path is one operation:
and the AI verdict/confidence/reason. A decision guide first states the behavioral
question, then presents the exact AI verdict/subtype with a plain-language
meaning; the Accept control repeats that exact label so its effect is explicit.
The common path is one operation:
- **Accept** keeps the AI verdict/type;
- **Relabel** expands verdict and subtype controls;
- **Exclude** requires a concrete data-quality reason;
- **Unsure / Second review** preserves the event for adjudication.
- **Exclude** requires a concrete data-quality reason.
The current primary-review UI deliberately has no second-review action. Older
saved `second_review` records remain readable for backward compatibility.
Boundary correction is hidden under an optional disclosure and is limited to
one 200 ms bin. The original timestamp is always retained. Each save atomically
rewrites JSONL and CSV while preserving the rule candidate, AI decision, human
decision, relabel flag, exclusion state, reviewer, and provenance.
decision, relabel flag, exclusion state, confidence, and provenance. The current
UI does not ask for a reviewer name; the export retains an empty `reviewer` field
only for compatibility with older review files.
The primary review queue also filters legacy non-crossing `WEARER_OFFSET` rows
and explicit `WEARER_RESUME` controls. To audit the controls intentionally,
pass `--events .../easycom_eot_int_controls.jsonl --include-controls` and use a
separate review output file. The old `--include-offset-controls` spelling
remains an alias.
Protocol wording for the paper appendix:
> Raw wearer speech onsets are first grouped into floor-level speech episodes
> by merging short within-turn resumptions that are not separated by a
> competing floor takeover. Only the resulting floor-level 0→1 transitions are
> proposed as EOT-response candidates.
> The original speech intervals are preserved unchanged for dense forecasting
> and auditability; this grouping is used only for behavioral event annotation.
## Freeze discipline