Delegate wearer boundary semantics to review
This commit is contained in:
@@ -6,7 +6,7 @@ ground truth.
|
||||
|
||||
```text
|
||||
EasyCom participant speech intervals
|
||||
-> floor-episode grouping + rule candidates + structured timing evidence
|
||||
-> all wearer boundaries + independent rule regions + structured factual evidence
|
||||
-> DeepSeek semantic pre-label (optional but recommended)
|
||||
-> human Accept / Relabel / Exclude
|
||||
```
|
||||
@@ -25,47 +25,37 @@ python3 prepare_easycom_eot_int_review.py \
|
||||
The generator emits two independent files:
|
||||
|
||||
- `easycom_eot_int_candidates.jsonl` (legacy-compatible filename) contains every
|
||||
floor-level wearer `0->1` behavior anchor (`WEARER_ONSET`,
|
||||
`WEARER_ENTRY_CANDIDATE`), only episode-ending wearer
|
||||
`1->0` boundaries where a new non-wearer onset satisfies
|
||||
`wearer_floor_start < other_onset < wearer_offset` and the same activity
|
||||
crosses the offset (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
|
||||
raw wearer `0->1` boundary (`WEARER_ONSET`, `WEARER_ENTRY_CANDIDATE`), every
|
||||
raw wearer `1->0` boundary (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
|
||||
regions that need no wearer transition:
|
||||
`SAME_SPEAKER_PAUSE`, `OTHER_TO_OTHER`, `OTHER_BACKCHANNEL`, and
|
||||
`NON_TAKEOVER_OVERLAP`;
|
||||
- `easycom_eot_int_controls.jsonl` contains short within-turn raw onsets
|
||||
(`WEARER_RESUME`) and wearer offsets that fail
|
||||
the strict INT ordering/crossing prerequisite (`WEARER_OFFSET_CONTROL`). They
|
||||
are retained for audits, but are not sent to DeepSeek or placed in the
|
||||
primary human-review queue.
|
||||
- `easycom_eot_int_controls.jsonl` is retained for file-format compatibility
|
||||
and legacy inputs. The current boundary protocol does not deterministically
|
||||
demote wearer onsets or offsets into this file.
|
||||
|
||||
A raw wearer onset is merged into its preceding floor episode only when its
|
||||
preceding silence is at most 0.8 seconds and no plausible sustained non-wearer
|
||||
takeover occurs in that gap. Short feedback-like lexical or nonverbal activity
|
||||
is recorded but does not by itself split the episode. A non-feedback other
|
||||
contribution that occupies at least `continuation_min_seconds` of the gap or
|
||||
continues across the resumed onset conservatively preserves a new wearer-entry
|
||||
candidate. Every decision is stored in `temporal_evidence`, including the raw
|
||||
gap, feedback IDs, competing IDs, threshold, and `floor_episode_merged` flag.
|
||||
No short-gap merge or takeover heuristic controls wearer-boundary admission.
|
||||
For each raw onset, `temporal_evidence` stores the previous wearer offset,
|
||||
silence duration, every other-speaker interval intersecting that gap, interval
|
||||
coverage, whether it continues across the onset, transcript, and
|
||||
`Target_of_Speech`. DeepSeek and the human reviewer decide whether the onset is
|
||||
a genuine entry, within-turn resumption, backchannel, or excluded non-event.
|
||||
|
||||
Entry construction retains every floor-level wearer onset, then searches backward
|
||||
for preceding non-wearer releases as optional context. A release is an admissible option only when the
|
||||
wearer stays silent and no new non-wearer onset or already-active other floor
|
||||
invalidates the interval before the wearer onset. Silence length alone does not
|
||||
invalidate an option. Each wearer onset produces exactly one primary row. A
|
||||
missing release does not demote an entry to a control: an entry may occur from
|
||||
silence or while another speaker is active. `entry_context_timing` records
|
||||
`AFTER_OTHER_RELEASE`, `OTHER_ACTIVE`, or `FROM_SILENCE_OR_UNRESOLVED` as
|
||||
metadata. When present, `opportunity_options` contains the only release anchors
|
||||
that DeepSeek or a human may optionally select; neither may invent a timestamp.
|
||||
Entry construction searches backward for nearby non-wearer releases as optional
|
||||
context. Every option carries factual flags for intervening wearer activity,
|
||||
intervening other onsets, and activity at the wearer onset; the generator does
|
||||
not declare one a valid conversational cause. Each wearer onset produces exactly
|
||||
one primary row. When present, `opportunity_options` contains the only release
|
||||
anchors that DeepSeek or a human may optionally select; neither may invent a
|
||||
timestamp.
|
||||
|
||||
Floor-level wearer onsets and strictly qualified offsets carry the rule-stage value
|
||||
`candidate`; timing alone is not a semantic positive label. Deterministic rows
|
||||
outside the primary queue carry `control`, never a semantic verdict. The INT prerequisite is
|
||||
`wearer_floor_start < other.start < wearer_offset < other.end`, with a
|
||||
configurable one-frame (50 ms) tolerance only at the offset boundary.
|
||||
Pre-existing other activity and raw offsets inside a merged wearer episode stay
|
||||
in the controls file. Hard-negative scans are rule hints and still require AI
|
||||
All wearer onsets and offsets carry the rule-stage value `candidate`; timing
|
||||
alone is not a semantic positive label. For each offset, the JSON records the
|
||||
wearer activity-run start, nearby other intervals, pre-existing and new-other
|
||||
flags, activity immediately before/after the boundary, strict and configurable
|
||||
50-ms-tolerant crossing, overlap duration, and subsequent speech onsets.
|
||||
`wearer_start < other.start < wearer_offset < other.end` is evidence rather than
|
||||
an INT prerequisite. Hard-negative scans are rule hints and still require AI
|
||||
and human review. The JSONL retains candidate source, original event time,
|
||||
participant-level activity, raw durations/gaps/overlap evidence, transcript,
|
||||
and `Target_of_Speech` when present. `Target_of_Speech` is annotation assistance
|
||||
@@ -80,7 +70,6 @@ All heuristic durations are configurable rather than frozen paper rules:
|
||||
--short-vocalization-max-seconds
|
||||
--continuation-min-seconds
|
||||
--int-offset-tolerance-seconds
|
||||
--wearer-resume-merge-max-seconds
|
||||
```
|
||||
|
||||
Use `--max-candidates 1` for a one-event smoke test. `summary.json` records the
|
||||
@@ -90,7 +79,7 @@ timeline; it is not sent to DeepSeek.
|
||||
|
||||
## 2. Add DeepSeek pre-labels
|
||||
|
||||
Prompt v11 first reconstructs the exact participant-resolved chronology and
|
||||
Prompt v14 first reconstructs the exact participant-resolved chronology and
|
||||
then uses transcript content and `Target_of_Speech` to determine conversational
|
||||
function. It treats the EasyCom transcript marker `[L]` as laughter-only evidence,
|
||||
not as a lexical floor take. Wearer laughter during another participant's
|
||||
@@ -98,8 +87,8 @@ retained turn is directed toward `ENTRY_WEARER_BACKCHANNEL`; non-wearer laughter
|
||||
overlapping a retained wearer turn is directed toward `INT_OTHER_BACKCHANNEL`.
|
||||
Ambiguous laughter falls back to `EXCLUDE`, and `[L]` alone is neither evidence
|
||||
of a floor take nor evidence of a visible smile. Preceding release options are
|
||||
optional entry metadata, whereas strict INT ordering remains a deterministic
|
||||
retrieval prerequisite. A positive `WEARER_ENTRY` may optionally select one
|
||||
optional entry metadata. No entry/within-turn or INT/takeover semantic decision
|
||||
is made before the prompt. A positive `WEARER_ENTRY` may optionally select one
|
||||
supplied release; entries from silence or during other activity use `null`.
|
||||
|
||||
`needs_human_review` is not requested from DeepSeek: every pre-label enters the
|
||||
@@ -135,10 +124,14 @@ transcript together. It returns one of `POSITIVE`, `HARD_NEGATIVE`, or
|
||||
`WEARER_ENTRY` or `INT_YIELD`, and hard negatives receive a named subtype. It
|
||||
cannot modify timestamps. The output preserves the original rule candidate,
|
||||
the selected preceding-release UID when applicable, and the complete AI
|
||||
decision/provenance. `EXCLUDE` is the sole non-scoring outcome. It covers
|
||||
out-of-scope non-events (including a normal wearer completion that merely
|
||||
coincides with feedback), insufficient evidence, unresolvable data quality, or
|
||||
disagreement between the two model passes.
|
||||
decision/provenance. `EXCLUDE` is the sole non-scoring outcome, but it is never
|
||||
an untyped bucket. Every AI and human exclusion stores one closed-vocabulary
|
||||
`exclude_reason` and the derived `exclude_reason_group`. `OUT_OF_SCOPE_*` means
|
||||
the annotations are valid but the behavior is outside the scored target;
|
||||
`INSUFFICIENT_EVIDENCE` means the supplied valid material cannot resolve
|
||||
conversational function; and `DATA_*` identifies unusable annotation,
|
||||
recording, context, artifact, or timestamp quality. Model-pass disagreement is
|
||||
recorded as `INSUFFICIENT_EVIDENCE`, never as data corruption.
|
||||
|
||||
Hard negatives use behavior-specific subtypes. ENTRY distinguishes a same-other
|
||||
mid-turn pause, a different-other transition, and a wearer backchannel that
|
||||
@@ -156,8 +149,9 @@ all candidate/pass work, including the current pass, batch, percentage, and
|
||||
elapsed time. Redirected logs emit periodic progress lines instead of carriage
|
||||
return animation.
|
||||
|
||||
As defense in depth, the adjudicator also detects and skips non-crossing
|
||||
`WEARER_OFFSET` rows from older v2 candidate files before making any API call.
|
||||
Every current `WEARER_ONSET` and `WEARER_OFFSET` row is sent for semantic
|
||||
adjudication regardless of its timing pattern. Explicit control rows from older
|
||||
protocol files remain excluded unless controls are intentionally reviewed.
|
||||
|
||||
## 3. Human review
|
||||
|
||||
@@ -192,22 +186,18 @@ decision, relabel flag, exclusion state, confidence, and provenance. The current
|
||||
UI does not ask for a reviewer name; the export retains an empty `reviewer` field
|
||||
only for compatibility with older review files.
|
||||
|
||||
The primary review queue also filters legacy non-crossing `WEARER_OFFSET` rows
|
||||
and explicit `WEARER_RESUME` controls. To audit the controls intentionally,
|
||||
The primary review queue filters only explicit legacy control rows. To audit
|
||||
those controls intentionally,
|
||||
pass `--events .../easycom_eot_int_controls.jsonl --include-controls` and use a
|
||||
separate review output file. The old `--include-offset-controls` spelling
|
||||
remains an alias.
|
||||
|
||||
Protocol wording for the paper appendix:
|
||||
|
||||
> Raw wearer speech onsets are first grouped into floor-level speech episodes
|
||||
> by merging short within-turn resumptions that are not separated by a
|
||||
> competing floor takeover. Every resulting floor-level 0→1 transition is
|
||||
> proposed as a wearer-entry candidate; a preceding non-wearer release is
|
||||
> retained as context rather than required for positive status.
|
||||
|
||||
> The original speech intervals are preserved unchanged for dense forecasting
|
||||
> and auditability; this grouping is used only for behavioral event annotation.
|
||||
> Every observed wearer speech onset and offset is retained as a semantic review
|
||||
> candidate. Deterministic construction records only participant-resolved timing,
|
||||
> transcript, and target-of-speech facts; it does not infer within-turn
|
||||
> resumption, floor entry, interruption, or takeover.
|
||||
|
||||
## Freeze discipline
|
||||
|
||||
|
||||
Reference in New Issue
Block a user