Delegate wearer boundary semantics to review
This commit is contained in:
+1
-1
@@ -304,7 +304,7 @@ Do not reorganize them physically without compatibility wrappers.
|
||||
| `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies |
|
||||
| `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication |
|
||||
| `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates |
|
||||
| `prepare_easycom_eot_int_review.py` | Group short non-competed wearer resumptions into floor episodes; retain every resulting wearer onset as an entry candidate with optional preceding-release metadata, require `wearer_floor_start < other_onset < wearer_offset` plus boundary crossing for INT, preserve ineligible raw boundaries as controls, and scan independent hard negatives; semantic labels remain AI/human review outputs |
|
||||
| `prepare_easycom_eot_int_review.py` | Retain every raw wearer onset and offset as an unresolved semantic candidate; attach nearby participant boundaries, activity relations, transcripts, and target-of-speech facts without entry/takeover gating; also scan independent hard-negative regions; semantic labels remain AI/human review outputs |
|
||||
| `adjudicate_easycom_eot_int_candidates.py` | Add optional DeepSeek transcript/timing suggestions; never human ground truth |
|
||||
| `review_floor_events.py` | Serve the local video/transcript workspace and persist human event reviews |
|
||||
| `prepare_easycom_oracle_features.py` | Build explicit past-only oracle person-activity sidecars and analysis-only addressee controls |
|
||||
|
||||
@@ -23,10 +23,11 @@ from typing import Any, Sequence, TextIO
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
from easycom_event_taxonomy import EXCLUDE_REASONS, exclude_reason_group
|
||||
from prepare_easycom_eot_int_review import requires_semantic_adjudication
|
||||
|
||||
|
||||
PROMPT_VERSION = "easycom-wearer-entry-int-prelabel-v12"
|
||||
PROMPT_VERSION = "easycom-wearer-boundary-semantic-prelabel-v14"
|
||||
DEFAULT_MODEL = "deepseek-v4-flash"
|
||||
DEFAULT_BASE_URL = "https://api.deepseek.com"
|
||||
VERDICTS = {"POSITIVE", "HARD_NEGATIVE", "EXCLUDE"}
|
||||
@@ -40,8 +41,6 @@ HARD_NEGATIVE_TYPES = {
|
||||
"INT_WEARER_HOLDS_FLOOR",
|
||||
"OTHER_HARD_NEGATIVE",
|
||||
}
|
||||
|
||||
|
||||
def _progress_line(
|
||||
completed: int, total: int, *, label: str, elapsed_seconds: float, width: int = 26
|
||||
) -> str:
|
||||
@@ -91,10 +90,10 @@ SYSTEM_PROMPT = """You conservatively pre-label candidate regions for a scientif
|
||||
Evidence rules:
|
||||
1. Reconstruct the event chronologically before assigning semantics. Use exact start/end timestamps to identify (a) the proposed release or competing onset, (b) the wearer behavior anchor, (c) which same participant continues across the anchor, and (d) who produces the next substantive turn. Then use transcript content and Target_of_Speech, when supplied, to decide conversational function. Timing has priority over speculative transcript interpretation, but duration alone never determines floor function.
|
||||
2. Never alter or propose an event timestamp. Do not infer prosody, gaze, intention, or acoustic cues that were not supplied.
|
||||
3. A raw wearer onset is not automatically a floor entry; a wearer offset is not automatically a yield; an other onset is not automatically a takeover. A pause is not automatically a release, overlap is not automatically interruption, and words such as "yeah" are not automatically backchannels. Determine what the utterance does in context, not only how long it lasts.
|
||||
3. Every supplied WEARER_ONSET and WEARER_OFFSET is a raw observed wearer activity boundary. Candidate generation has made no floor-entry, within-turn-resumption, interruption, or takeover decision. A raw wearer onset is not automatically a floor entry; a wearer offset is not automatically a yield; an other onset is not automatically a takeover. A pause is not automatically a release, overlap is not automatically interruption, and words such as "yeah" are not automatically backchannels. Determine what the utterance does in context, not only how long it lasts.
|
||||
4. EXCLUDE is the sole non-scoring outcome. Use it both for insufficient/unresolvable evidence and for clear out-of-scope non-events that should not enter evaluation, including a normal wearer completion that merely coincides with laughter or feedback. Never force such a region into HARD_NEGATIVE for coverage.
|
||||
5. Every supplied WEARER_ONSET is an observed floor-level onset after deterministic within-turn pause merging. Decide whether the wearer genuinely enters or re-enters the conversational floor. A POSITIVE WEARER_ENTRY does not require a preceding non-wearer release: it may occur after a release, from silence, or while another speaker is active. temporal_evidence.opportunity_options and entry_context_timing are context metadata only. If one supplied release clearly prompted the entry, selected_opportunity_uid may contain its uid; otherwise return null. Never invent a trigger or timestamp. A direct, semantically complete answer can be a genuine entry even when short. Do not call it a backchannel merely because it lasts less than one second or because another participant speaks later. For example, "Nope." beginning 0.15 s after an addressed question ends is normally WEARER_ENTRY, not ENTRY_WEARER_BACKCHANNEL. Later conversation does not retroactively cancel a completed entry.
|
||||
6. Every supplied WEARER_OFFSET has already passed both deterministic INT prerequisites: a new non-wearer onset occurs strictly inside the current wearer floor episode (t_w < t_o < t_y), and that same non-wearer interval crosses the wearer yield boundary within the recorded annotation tolerance. This is only a retrieval gate. Verify that the same incoming non-wearer actually takes and sustains the next floor after the wearer offset. A different participant's later substantive turn is not evidence that the incoming speaker took over. Do not label INT_WEARER_HOLDS_FLOOR merely because the wearer speaks again later: if the competing other establishes an intervening substantive floor and the wearer has an intervening silent interval, the later wearer onset is a new WEARER_ENTRY and does not invalidate the earlier INT_YIELD. If the wearer reaches a semantically complete natural ending while the overlapping contribution is only [L] or feedback, and that contributor does not take the next substantive floor, return EXCLUDE: out-of-scope normal completion, not INT_YIELD and not a scored hard negative. Pre-existing other activity, post-completion other onset, within-turn wearer pauses, and completion-to-silence are controls outside this queue.
|
||||
5. For every WEARER_ONSET, decide whether the wearer genuinely enters or re-enters the conversational floor. First inspect the previous wearer offset and silence gap, gap_other_intervals, nearby other activity, transcript, and Target_of_Speech. A short within-turn pause followed by continuation of the same wearer contribution is not a new entry; use EXCLUDE unless the region is a valid named false-positive opportunity. A POSITIVE WEARER_ENTRY does not require a preceding non-wearer release: it may occur after a release, from silence, or while another speaker is active. temporal_evidence.opportunity_options contains nearby preceding other offsets with factual relation flags, not pre-approved causes. If one supplied release clearly prompted the entry, selected_opportunity_uid may contain its uid; otherwise return null. Never invent a trigger or timestamp. A direct, semantically complete answer can be a genuine entry even when short. Do not call it a backchannel merely because it lasts less than one second or because another participant speaks later. For example, "Nope." beginning 0.15 s after an addressed question ends is normally WEARER_ENTRY, not ENTRY_WEARER_BACKCHANNEL. Later conversation does not retroactively cancel a completed entry.
|
||||
6. For every WEARER_OFFSET, answer the central question: did the wearer stop because a non-wearer genuinely took over the conversational floor? Reconstruct the wearer activity-run start and offset, every nearby other onset and offset, which others are active immediately before and after the offset, whether each other is pre-existing or starts during wearer speech, whether it crosses the offset, overlap duration, who speaks next, transcript content, and Target_of_Speech. All temporal booleans are descriptive evidence, not prerequisite labels. A valid INT_YIELD may involve a newly entering or pre-existing competitor if the chronology and semantics establish genuine takeover and wearer relinquishment. Conversely, crossing the offset or satisfying t_w < t_o < t_y never proves takeover by itself. Verify that the relevant non-wearer actually takes and sustains the next floor. A different participant's later substantive turn is not evidence that the proposed competitor took over. Do not label INT_WEARER_HOLDS_FLOOR merely because the wearer speaks again later: if a competitor establishes an intervening substantive floor and the wearer has an intervening silent interval, the later wearer onset is a new WEARER_ENTRY and does not invalidate the earlier INT_YIELD. Natural completion, completion-to-silence, a within-turn wearer pause, or a post-completion other onset that could not have caused the stop is normally EXCLUDE unless the region is a valid named hard-negative opportunity. If the wearer reaches a semantically complete natural ending while overlapping activity is only [L] or feedback and that contributor does not take the next substantive floor, return EXCLUDE.
|
||||
7. Transcript event markers are evidence, not lexical turns. In EasyCom, a transcription consisting of [L] denotes laughter-only vocalization. Do not treat [L] as a sustained floor entry or interruption takeover merely because its activity interval creates an onset, offset, or overlap candidate. Specifically:
|
||||
- wearer [L] during a non-wearer's retained turn is normally HARD_NEGATIVE / ENTRY_WEARER_BACKCHANNEL, not WEARER_ENTRY;
|
||||
- non-wearer [L] overlapping a wearer's retained turn is normally HARD_NEGATIVE / INT_OTHER_BACKCHANNEL, not INT_YIELD;
|
||||
@@ -106,6 +105,19 @@ Top-level verdicts:
|
||||
- HARD_NEGATIVE: a deliberately retained, valid false-positive opportunity matching one of the behavioral counterexamples below; it enters evaluation as a negative.
|
||||
- EXCLUDE: a non-scoring region. This includes clear out-of-scope/non-target opportunities such as natural wearer completion coincident with feedback, as well as insufficient evidence, annotation corruption, a recording boundary, missing context, a non-speech artifact, timestamp corruption, or another unresolvable defect.
|
||||
|
||||
EXCLUDE requires exactly one exclude_reason from this closed taxonomy:
|
||||
- OUT_OF_SCOPE_NORMAL_COMPLETION: the wearer naturally completes; no interruption-yield target occurs.
|
||||
- OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION: a raw boundary is only a pause/resumption within the same wearer contribution.
|
||||
- OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY: another contribution begins only after wearer completion and cannot have caused the boundary.
|
||||
- OUT_OF_SCOPE_NON_EVENT: another clear, interpretable non-target boundary not covered above.
|
||||
- INSUFFICIENT_EVIDENCE: the supplied valid evidence cannot resolve conversational function.
|
||||
- DATA_ANNOTATION_ERROR: participant, speech, transcript, or target annotation is evidently erroneous or contradictory.
|
||||
- DATA_RECORDING_BOUNDARY: the recording/clip boundary prevents reliable interpretation.
|
||||
- DATA_MISSING_CONTEXT: required context is absent from the supplied annotation window or source.
|
||||
- DATA_NON_SPEECH_ARTIFACT: the boundary is caused by an artifact rather than an annotated conversational vocalization.
|
||||
- DATA_TIMESTAMP_CORRUPTION: timestamps are corrupted or irreconcilably misaligned.
|
||||
The OUT_OF_SCOPE_* reasons describe valid data outside the evaluation target. DATA_* reasons describe bad or unusable data. Never substitute one family for the other.
|
||||
|
||||
POSITIVE event_type must be exactly WEARER_ENTRY or INT_YIELD. WEARER_ENTRY means the wearer genuinely enters or re-enters the floor with a contextually complete response or substantive contribution; no prior release is required and the contribution need not be long. INT_YIELD means a non-wearer genuinely takes over during wearer speech and the wearer relinquishes the floor. A later WEARER_ENTRY after an intervening other floor is a new event and must not retroactively invalidate the preceding INT_YIELD.
|
||||
|
||||
HARD_NEGATIVE subtypes are behavioral counterexamples, not data-quality labels:
|
||||
@@ -124,12 +136,12 @@ Subtype precedence when descriptions overlap:
|
||||
4. Use OTHER_HARD_NEGATIVE only after the named subtypes have been ruled out. Never use it merely because confidence is low.
|
||||
|
||||
Return exactly one JSON object with this shape and no markdown:
|
||||
{"items":[{"uid":"...","verdict":"EXCLUDE","event_type":null,"subtype":null,"selected_opportunity_uid":null,"confidence":0.25,"reason":"brief evidence-based sentence"}]}
|
||||
{"items":[{"uid":"...","verdict":"EXCLUDE","event_type":null,"subtype":null,"exclude_reason":"INSUFFICIENT_EVIDENCE","selected_opportunity_uid":null,"confidence":0.25,"reason":"brief evidence-based sentence"}]}
|
||||
Field combinations are strict:
|
||||
- POSITIVE / WEARER_ENTRY: subtype=null; selected_opportunity_uid is one supplied opportunity uid or null.
|
||||
- POSITIVE / INT_YIELD: subtype=null; selected_opportunity_uid=null.
|
||||
- HARD_NEGATIVE: event_type=null; selected_opportunity_uid=null; subtype is one valid subtype from the candidate's ENTRY or INT family.
|
||||
- EXCLUDE: event_type=null; subtype=null; selected_opportunity_uid=null.
|
||||
- POSITIVE / WEARER_ENTRY: subtype=null; exclude_reason=null; selected_opportunity_uid is one supplied opportunity uid or null.
|
||||
- POSITIVE / INT_YIELD: subtype=null; exclude_reason=null; selected_opportunity_uid=null.
|
||||
- HARD_NEGATIVE: event_type=null; exclude_reason=null; selected_opportunity_uid=null; subtype is one valid subtype from the candidate's ENTRY or INT family.
|
||||
- EXCLUDE: event_type=null; subtype=null; selected_opportunity_uid=null; exclude_reason is exactly one taxonomy value above.
|
||||
Return every supplied uid exactly once. Copy uid byte-for-byte and never add a backslash before an underscore. confidence must be a number from 0 to 1."""
|
||||
|
||||
|
||||
@@ -245,6 +257,7 @@ def validate_decisions(
|
||||
verdict = item.get("verdict")
|
||||
event_type = item.get("event_type")
|
||||
subtype = item.get("subtype")
|
||||
exclude_reason = item.get("exclude_reason")
|
||||
selected_opportunity_uid = item.get("selected_opportunity_uid")
|
||||
confidence = item.get("confidence")
|
||||
reason = item.get("reason")
|
||||
@@ -260,6 +273,10 @@ def validate_decisions(
|
||||
raise ValueError(f"HARD_NEGATIVE requires a valid subtype for {uid}")
|
||||
if verdict != "HARD_NEGATIVE" and subtype is not None:
|
||||
raise ValueError(f"Only HARD_NEGATIVE may have subtype for {uid}")
|
||||
if verdict == "EXCLUDE" and exclude_reason not in EXCLUDE_REASONS:
|
||||
raise ValueError(f"EXCLUDE requires a valid exclude_reason for {uid}")
|
||||
if verdict != "EXCLUDE" and exclude_reason is not None:
|
||||
raise ValueError(f"Only EXCLUDE may have exclude_reason for {uid}")
|
||||
family = None if expected_families is None else expected_families.get(str(uid))
|
||||
if verdict == "POSITIVE" and family in {"ENTRY", "INT"}:
|
||||
required_event_type = "WEARER_ENTRY" if family == "ENTRY" else "INT_YIELD"
|
||||
@@ -287,6 +304,8 @@ def validate_decisions(
|
||||
"verdict": str(verdict),
|
||||
"event_type": event_type,
|
||||
"subtype": subtype,
|
||||
"exclude_reason": exclude_reason,
|
||||
"exclude_reason_group": exclude_reason_group(exclude_reason),
|
||||
"selected_opportunity_uid": selected_opportunity_uid,
|
||||
"confidence": float(confidence),
|
||||
"reason": reason.strip(),
|
||||
@@ -437,12 +456,12 @@ def add_suggestions(
|
||||
for source in rows:
|
||||
decisions = by_uid[source["uid"]]
|
||||
labels = {
|
||||
(d["verdict"], d["event_type"], d["subtype"])
|
||||
(d["verdict"], d["event_type"], d["subtype"], d["exclude_reason"])
|
||||
for d in decisions
|
||||
}
|
||||
unanimous = len(labels) == 1
|
||||
if unanimous:
|
||||
verdict, event_type, subtype = next(iter(labels))
|
||||
verdict, event_type, subtype, exclude_reason = next(iter(labels))
|
||||
selected_options = {d["selected_opportunity_uid"] for d in decisions}
|
||||
selected_opportunity_uid = (
|
||||
next(iter(selected_options)) if len(selected_options) == 1 else None
|
||||
@@ -451,7 +470,9 @@ def add_suggestions(
|
||||
reason = " | ".join(item["reason"] for item in decisions)
|
||||
needs_human_review = any(item["needs_human_review"] for item in decisions)
|
||||
else:
|
||||
verdict, event_type, subtype, selected_opportunity_uid = "EXCLUDE", None, None, None
|
||||
verdict, event_type, subtype, exclude_reason, selected_opportunity_uid = (
|
||||
"EXCLUDE", None, None, "INSUFFICIENT_EVIDENCE", None
|
||||
)
|
||||
confidence = min(float(item["confidence"]) for item in decisions)
|
||||
reason = "The independent DeepSeek passes disagreed."
|
||||
needs_human_review = True
|
||||
@@ -474,6 +495,8 @@ def add_suggestions(
|
||||
"verdict": verdict,
|
||||
"event_type": event_type,
|
||||
"subtype": subtype,
|
||||
"exclude_reason": exclude_reason,
|
||||
"exclude_reason_group": exclude_reason_group(exclude_reason),
|
||||
"selected_opportunity_uid": selected_opportunity_uid,
|
||||
"selected_opportunity_seconds": selected_opportunity_seconds,
|
||||
"valid_prediction_interval_seconds": (
|
||||
@@ -490,6 +513,8 @@ def add_suggestions(
|
||||
ai_verdict=verdict,
|
||||
ai_event_type=event_type,
|
||||
ai_subtype=subtype,
|
||||
ai_exclude_reason=exclude_reason,
|
||||
ai_exclude_reason_group=exclude_reason_group(exclude_reason),
|
||||
ai_selected_opportunity_uid=selected_opportunity_uid,
|
||||
ai_selected_opportunity_seconds=selected_opportunity_seconds,
|
||||
ai_confidence=confidence,
|
||||
|
||||
@@ -6,7 +6,7 @@ ground truth.
|
||||
|
||||
```text
|
||||
EasyCom participant speech intervals
|
||||
-> floor-episode grouping + rule candidates + structured timing evidence
|
||||
-> all wearer boundaries + independent rule regions + structured factual evidence
|
||||
-> DeepSeek semantic pre-label (optional but recommended)
|
||||
-> human Accept / Relabel / Exclude
|
||||
```
|
||||
@@ -25,47 +25,37 @@ python3 prepare_easycom_eot_int_review.py \
|
||||
The generator emits two independent files:
|
||||
|
||||
- `easycom_eot_int_candidates.jsonl` (legacy-compatible filename) contains every
|
||||
floor-level wearer `0->1` behavior anchor (`WEARER_ONSET`,
|
||||
`WEARER_ENTRY_CANDIDATE`), only episode-ending wearer
|
||||
`1->0` boundaries where a new non-wearer onset satisfies
|
||||
`wearer_floor_start < other_onset < wearer_offset` and the same activity
|
||||
crosses the offset (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
|
||||
raw wearer `0->1` boundary (`WEARER_ONSET`, `WEARER_ENTRY_CANDIDATE`), every
|
||||
raw wearer `1->0` boundary (`WEARER_OFFSET`, `YIELD_CANDIDATE`), and hard-negative
|
||||
regions that need no wearer transition:
|
||||
`SAME_SPEAKER_PAUSE`, `OTHER_TO_OTHER`, `OTHER_BACKCHANNEL`, and
|
||||
`NON_TAKEOVER_OVERLAP`;
|
||||
- `easycom_eot_int_controls.jsonl` contains short within-turn raw onsets
|
||||
(`WEARER_RESUME`) and wearer offsets that fail
|
||||
the strict INT ordering/crossing prerequisite (`WEARER_OFFSET_CONTROL`). They
|
||||
are retained for audits, but are not sent to DeepSeek or placed in the
|
||||
primary human-review queue.
|
||||
- `easycom_eot_int_controls.jsonl` is retained for file-format compatibility
|
||||
and legacy inputs. The current boundary protocol does not deterministically
|
||||
demote wearer onsets or offsets into this file.
|
||||
|
||||
A raw wearer onset is merged into its preceding floor episode only when its
|
||||
preceding silence is at most 0.8 seconds and no plausible sustained non-wearer
|
||||
takeover occurs in that gap. Short feedback-like lexical or nonverbal activity
|
||||
is recorded but does not by itself split the episode. A non-feedback other
|
||||
contribution that occupies at least `continuation_min_seconds` of the gap or
|
||||
continues across the resumed onset conservatively preserves a new wearer-entry
|
||||
candidate. Every decision is stored in `temporal_evidence`, including the raw
|
||||
gap, feedback IDs, competing IDs, threshold, and `floor_episode_merged` flag.
|
||||
No short-gap merge or takeover heuristic controls wearer-boundary admission.
|
||||
For each raw onset, `temporal_evidence` stores the previous wearer offset,
|
||||
silence duration, every other-speaker interval intersecting that gap, interval
|
||||
coverage, whether it continues across the onset, transcript, and
|
||||
`Target_of_Speech`. DeepSeek and the human reviewer decide whether the onset is
|
||||
a genuine entry, within-turn resumption, backchannel, or excluded non-event.
|
||||
|
||||
Entry construction retains every floor-level wearer onset, then searches backward
|
||||
for preceding non-wearer releases as optional context. A release is an admissible option only when the
|
||||
wearer stays silent and no new non-wearer onset or already-active other floor
|
||||
invalidates the interval before the wearer onset. Silence length alone does not
|
||||
invalidate an option. Each wearer onset produces exactly one primary row. A
|
||||
missing release does not demote an entry to a control: an entry may occur from
|
||||
silence or while another speaker is active. `entry_context_timing` records
|
||||
`AFTER_OTHER_RELEASE`, `OTHER_ACTIVE`, or `FROM_SILENCE_OR_UNRESOLVED` as
|
||||
metadata. When present, `opportunity_options` contains the only release anchors
|
||||
that DeepSeek or a human may optionally select; neither may invent a timestamp.
|
||||
Entry construction searches backward for nearby non-wearer releases as optional
|
||||
context. Every option carries factual flags for intervening wearer activity,
|
||||
intervening other onsets, and activity at the wearer onset; the generator does
|
||||
not declare one a valid conversational cause. Each wearer onset produces exactly
|
||||
one primary row. When present, `opportunity_options` contains the only release
|
||||
anchors that DeepSeek or a human may optionally select; neither may invent a
|
||||
timestamp.
|
||||
|
||||
Floor-level wearer onsets and strictly qualified offsets carry the rule-stage value
|
||||
`candidate`; timing alone is not a semantic positive label. Deterministic rows
|
||||
outside the primary queue carry `control`, never a semantic verdict. The INT prerequisite is
|
||||
`wearer_floor_start < other.start < wearer_offset < other.end`, with a
|
||||
configurable one-frame (50 ms) tolerance only at the offset boundary.
|
||||
Pre-existing other activity and raw offsets inside a merged wearer episode stay
|
||||
in the controls file. Hard-negative scans are rule hints and still require AI
|
||||
All wearer onsets and offsets carry the rule-stage value `candidate`; timing
|
||||
alone is not a semantic positive label. For each offset, the JSON records the
|
||||
wearer activity-run start, nearby other intervals, pre-existing and new-other
|
||||
flags, activity immediately before/after the boundary, strict and configurable
|
||||
50-ms-tolerant crossing, overlap duration, and subsequent speech onsets.
|
||||
`wearer_start < other.start < wearer_offset < other.end` is evidence rather than
|
||||
an INT prerequisite. Hard-negative scans are rule hints and still require AI
|
||||
and human review. The JSONL retains candidate source, original event time,
|
||||
participant-level activity, raw durations/gaps/overlap evidence, transcript,
|
||||
and `Target_of_Speech` when present. `Target_of_Speech` is annotation assistance
|
||||
@@ -80,7 +70,6 @@ All heuristic durations are configurable rather than frozen paper rules:
|
||||
--short-vocalization-max-seconds
|
||||
--continuation-min-seconds
|
||||
--int-offset-tolerance-seconds
|
||||
--wearer-resume-merge-max-seconds
|
||||
```
|
||||
|
||||
Use `--max-candidates 1` for a one-event smoke test. `summary.json` records the
|
||||
@@ -90,7 +79,7 @@ timeline; it is not sent to DeepSeek.
|
||||
|
||||
## 2. Add DeepSeek pre-labels
|
||||
|
||||
Prompt v11 first reconstructs the exact participant-resolved chronology and
|
||||
Prompt v14 first reconstructs the exact participant-resolved chronology and
|
||||
then uses transcript content and `Target_of_Speech` to determine conversational
|
||||
function. It treats the EasyCom transcript marker `[L]` as laughter-only evidence,
|
||||
not as a lexical floor take. Wearer laughter during another participant's
|
||||
@@ -98,8 +87,8 @@ retained turn is directed toward `ENTRY_WEARER_BACKCHANNEL`; non-wearer laughter
|
||||
overlapping a retained wearer turn is directed toward `INT_OTHER_BACKCHANNEL`.
|
||||
Ambiguous laughter falls back to `EXCLUDE`, and `[L]` alone is neither evidence
|
||||
of a floor take nor evidence of a visible smile. Preceding release options are
|
||||
optional entry metadata, whereas strict INT ordering remains a deterministic
|
||||
retrieval prerequisite. A positive `WEARER_ENTRY` may optionally select one
|
||||
optional entry metadata. No entry/within-turn or INT/takeover semantic decision
|
||||
is made before the prompt. A positive `WEARER_ENTRY` may optionally select one
|
||||
supplied release; entries from silence or during other activity use `null`.
|
||||
|
||||
`needs_human_review` is not requested from DeepSeek: every pre-label enters the
|
||||
@@ -135,10 +124,14 @@ transcript together. It returns one of `POSITIVE`, `HARD_NEGATIVE`, or
|
||||
`WEARER_ENTRY` or `INT_YIELD`, and hard negatives receive a named subtype. It
|
||||
cannot modify timestamps. The output preserves the original rule candidate,
|
||||
the selected preceding-release UID when applicable, and the complete AI
|
||||
decision/provenance. `EXCLUDE` is the sole non-scoring outcome. It covers
|
||||
out-of-scope non-events (including a normal wearer completion that merely
|
||||
coincides with feedback), insufficient evidence, unresolvable data quality, or
|
||||
disagreement between the two model passes.
|
||||
decision/provenance. `EXCLUDE` is the sole non-scoring outcome, but it is never
|
||||
an untyped bucket. Every AI and human exclusion stores one closed-vocabulary
|
||||
`exclude_reason` and the derived `exclude_reason_group`. `OUT_OF_SCOPE_*` means
|
||||
the annotations are valid but the behavior is outside the scored target;
|
||||
`INSUFFICIENT_EVIDENCE` means the supplied valid material cannot resolve
|
||||
conversational function; and `DATA_*` identifies unusable annotation,
|
||||
recording, context, artifact, or timestamp quality. Model-pass disagreement is
|
||||
recorded as `INSUFFICIENT_EVIDENCE`, never as data corruption.
|
||||
|
||||
Hard negatives use behavior-specific subtypes. ENTRY distinguishes a same-other
|
||||
mid-turn pause, a different-other transition, and a wearer backchannel that
|
||||
@@ -156,8 +149,9 @@ all candidate/pass work, including the current pass, batch, percentage, and
|
||||
elapsed time. Redirected logs emit periodic progress lines instead of carriage
|
||||
return animation.
|
||||
|
||||
As defense in depth, the adjudicator also detects and skips non-crossing
|
||||
`WEARER_OFFSET` rows from older v2 candidate files before making any API call.
|
||||
Every current `WEARER_ONSET` and `WEARER_OFFSET` row is sent for semantic
|
||||
adjudication regardless of its timing pattern. Explicit control rows from older
|
||||
protocol files remain excluded unless controls are intentionally reviewed.
|
||||
|
||||
## 3. Human review
|
||||
|
||||
@@ -192,22 +186,18 @@ decision, relabel flag, exclusion state, confidence, and provenance. The current
|
||||
UI does not ask for a reviewer name; the export retains an empty `reviewer` field
|
||||
only for compatibility with older review files.
|
||||
|
||||
The primary review queue also filters legacy non-crossing `WEARER_OFFSET` rows
|
||||
and explicit `WEARER_RESUME` controls. To audit the controls intentionally,
|
||||
The primary review queue filters only explicit legacy control rows. To audit
|
||||
those controls intentionally,
|
||||
pass `--events .../easycom_eot_int_controls.jsonl --include-controls` and use a
|
||||
separate review output file. The old `--include-offset-controls` spelling
|
||||
remains an alias.
|
||||
|
||||
Protocol wording for the paper appendix:
|
||||
|
||||
> Raw wearer speech onsets are first grouped into floor-level speech episodes
|
||||
> by merging short within-turn resumptions that are not separated by a
|
||||
> competing floor takeover. Every resulting floor-level 0→1 transition is
|
||||
> proposed as a wearer-entry candidate; a preceding non-wearer release is
|
||||
> retained as context rather than required for positive status.
|
||||
|
||||
> The original speech intervals are preserved unchanged for dense forecasting
|
||||
> and auditability; this grouping is used only for behavioral event annotation.
|
||||
> Every observed wearer speech onset and offset is retained as a semantic review
|
||||
> candidate. Deterministic construction records only participant-resolved timing,
|
||||
> transcript, and target-of-speech facts; it does not infer within-turn
|
||||
> resumption, floor entry, interruption, or takeover.
|
||||
|
||||
## Freeze discipline
|
||||
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Closed vocabularies shared by EasyCom event pre-label and review tools."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
EXCLUDE_REASONS = {
|
||||
"OUT_OF_SCOPE_NORMAL_COMPLETION",
|
||||
"OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION",
|
||||
"OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY",
|
||||
"OUT_OF_SCOPE_NON_EVENT",
|
||||
"INSUFFICIENT_EVIDENCE",
|
||||
"DATA_ANNOTATION_ERROR",
|
||||
"DATA_RECORDING_BOUNDARY",
|
||||
"DATA_MISSING_CONTEXT",
|
||||
"DATA_NON_SPEECH_ARTIFACT",
|
||||
"DATA_TIMESTAMP_CORRUPTION",
|
||||
}
|
||||
|
||||
# Accepted only when loading or editing old review records. New DeepSeek and UI
|
||||
# outputs use the prefixed taxonomy above so scope and data-quality counts are
|
||||
# never silently combined.
|
||||
LEGACY_EXCLUDE_REASONS = {
|
||||
"ANNOTATION_ERROR",
|
||||
"RECORDING_BOUNDARY",
|
||||
"MISSING_CONTEXT",
|
||||
"NON_SPEECH_ARTIFACT",
|
||||
"TIMESTAMP_CORRUPTION",
|
||||
"UNRESOLVABLE",
|
||||
}
|
||||
|
||||
|
||||
def exclude_reason_group(reason: str | None) -> str | None:
|
||||
"""Map a leaf reason to a stable aggregation group."""
|
||||
if reason is None:
|
||||
return None
|
||||
if reason.startswith("OUT_OF_SCOPE_"):
|
||||
return "OUT_OF_SCOPE"
|
||||
if reason == "INSUFFICIENT_EVIDENCE":
|
||||
return "INSUFFICIENT_EVIDENCE"
|
||||
if reason.startswith("DATA_"):
|
||||
return "DATA_QUALITY"
|
||||
if reason in LEGACY_EXCLUDE_REASONS:
|
||||
return "LEGACY_UNSPECIFIED"
|
||||
return "UNKNOWN"
|
||||
@@ -57,13 +57,23 @@ const decisionMeanings = {
|
||||
INT_WEARER_HOLDS_FLOOR: "Other 不是单纯 backchannel,且 wearer 在重叠后明确继续或恢复原 turn。",
|
||||
OTHER_HARD_NEGATIVE: "这是可解释的困难负例,但不符合任何命名子类型;不能用于替代 Exclude。",
|
||||
EXCLUDE: "该区域是无需参评的非目标事件,或存在证据/数据问题,不进入评分。",
|
||||
OUT_OF_SCOPE_NORMAL_COMPLETION: "有效语境中的自然结束,不是 interruption yield。",
|
||||
OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION: "这是同一 wearer turn 内的暂停或继续,不是新的 floor event。",
|
||||
OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY: "Other 在 wearer 已结束后才开始,不能构成导致停止的 takeover。",
|
||||
OUT_OF_SCOPE_NON_EVENT: "数据有效且可解释,但该边界不属于目标行为。",
|
||||
INSUFFICIENT_EVIDENCE: "数据未显示明确损坏,但现有证据不足以可靠确定 conversational function。",
|
||||
DATA_ANNOTATION_ERROR: "参与者、语音、转写或 target 标注明显错误或互相矛盾。",
|
||||
DATA_RECORDING_BOUNDARY: "录制或 clip 边界使该事件无法可靠解释。",
|
||||
DATA_MISSING_CONTEXT: "判断所需上下文未被提供。",
|
||||
DATA_NON_SPEECH_ARTIFACT: "候选由非语音伪影造成。",
|
||||
DATA_TIMESTAMP_CORRUPTION: "时间戳损坏或无法对齐。",
|
||||
};
|
||||
|
||||
const currentAiVerdicts = new Set(["POSITIVE", "HARD_NEGATIVE", "EXCLUDE"]);
|
||||
|
||||
function suggestionView(suggestion) {
|
||||
const verdict = suggestion.verdict || suggestion.protocol_disposition || "—";
|
||||
const label = suggestion.event_type || suggestion.subtype || suggestion.hard_negative_type || verdict;
|
||||
const label = suggestion.event_type || suggestion.subtype || suggestion.exclude_reason || suggestion.hard_negative_type || verdict;
|
||||
const rawConfidence = suggestion.confidence ?? suggestion.final_confidence;
|
||||
const numericConfidence = Number(rawConfidence);
|
||||
const confidence = Number.isFinite(numericConfidence)
|
||||
@@ -170,7 +180,7 @@ function applyFilters({ keepCurrent = true } = {}) {
|
||||
(elements.splitFilter.value === "all" || item.split === elements.splitFilter.value) &&
|
||||
(elements.familyFilter.value === "all" || item.family === elements.familyFilter.value) &&
|
||||
(status === "all" || (status === "reviewed" ? reviewed : !reviewed)) &&
|
||||
(!search || `${item.uid} ${item.candidate_source || ""} ${item.rule_candidate_type || ""} ${item.ai_verdict || ""} ${item.ai_event_type || ""} ${item.ai_subtype || ""} ${item.session}`.toLowerCase().includes(search))
|
||||
(!search || `${item.uid} ${item.candidate_source || ""} ${item.rule_candidate_type || ""} ${item.ai_verdict || ""} ${item.ai_event_type || ""} ${item.ai_subtype || ""} ${item.ai_exclude_reason || ""} ${item.session}`.toLowerCase().includes(search))
|
||||
);
|
||||
});
|
||||
renderQueue();
|
||||
@@ -206,7 +216,7 @@ function renderQueue() {
|
||||
const label = document.createElement("div");
|
||||
label.className = "queue-label";
|
||||
label.textContent = item.ai_verdict
|
||||
? `AI ${item.ai_verdict} · ${item.ai_event_type || item.ai_subtype || "—"}`
|
||||
? `AI ${item.ai_verdict} · ${item.ai_event_type || item.ai_subtype || item.ai_exclude_reason || "—"}`
|
||||
: `${item.rule_candidate_type || item.automatic_label} · 待 AI 粗标`;
|
||||
button.append(line, meta, label);
|
||||
button.addEventListener("click", () => selectCandidate(item.candidate_id));
|
||||
@@ -292,7 +302,8 @@ function renderDetail(detail) {
|
||||
const familyPrefix = detail.family === "entry" ? "ENTRY_" : detail.family === "int" ? "INT_" : null;
|
||||
const canAccept = currentAiVerdicts.has(view.verdict)
|
||||
&& (view.verdict !== "POSITIVE" || !familyEvent || suggestion.event_type === familyEvent)
|
||||
&& (view.verdict !== "HARD_NEGATIVE" || !familyPrefix || suggestion.subtype === "OTHER_HARD_NEGATIVE" || String(suggestion.subtype || "").startsWith(familyPrefix));
|
||||
&& (view.verdict !== "HARD_NEGATIVE" || !familyPrefix || suggestion.subtype === "OTHER_HARD_NEGATIVE" || String(suggestion.subtype || "").startsWith(familyPrefix))
|
||||
&& (view.verdict !== "EXCLUDE" || Boolean(suggestion.exclude_reason));
|
||||
elements.acceptRadio.disabled = !canAccept;
|
||||
elements.acceptTitle.textContent = canAccept
|
||||
? `接受 ${view.verdict.replaceAll("_", " ")}`
|
||||
@@ -302,7 +313,9 @@ function renderDetail(detail) {
|
||||
const anchorText = selectedOption ? ` · trigger ${formatTime(selectedOption.anchor_seconds)} P${selectedOption.participant_id}` : "";
|
||||
elements.acceptSummary.textContent = canAccept
|
||||
? `${view.label}${anchorText} · ${view.meaning}`
|
||||
: "该标签已移除,请改标为正例/硬负例,或选择 Exclude。";
|
||||
: (view.verdict === "EXCLUDE"
|
||||
? "旧 AI EXCLUDE 没有 reason taxonomy,请选择 Exclude 并补充原因。"
|
||||
: "该标签已移除,请改标为正例/硬负例,或选择 Exclude。");
|
||||
} else {
|
||||
elements.acceptRadio.disabled = true;
|
||||
elements.acceptTitle.textContent = "没有 AI 结论";
|
||||
|
||||
@@ -88,7 +88,7 @@
|
||||
<label id="subtypeField" hidden><span>Hard-negative subtype</span><select name="human_subtype"><option value="">请选择</option><option>ENTRY_OTHER_MIDTURN_PAUSE</option><option>ENTRY_OTHER_TO_OTHER</option><option>ENTRY_WEARER_BACKCHANNEL</option><option>INT_OTHER_BACKCHANNEL</option><option>INT_NON_TAKEOVER_OVERLAP</option><option>INT_WEARER_HOLDS_FLOOR</option><option>OTHER_HARD_NEGATIVE</option></select></label>
|
||||
</section>
|
||||
|
||||
<section id="excludeFields" class="conditional-fields" hidden><label><span>排除原因</span><select name="exclude_reason"><option value="">请选择</option><option value="OUT_OF_SCOPE_NORMAL_COMPLETION">OUT_OF_SCOPE_NORMAL_COMPLETION — wearer 正常说完</option><option value="OUT_OF_SCOPE_NON_EVENT">OUT_OF_SCOPE_NON_EVENT — 其他无需参评事件</option><option>ANNOTATION_ERROR</option><option>RECORDING_BOUNDARY</option><option>MISSING_CONTEXT</option><option>NON_SPEECH_ARTIFACT</option><option>TIMESTAMP_CORRUPTION</option><option>UNRESOLVABLE</option></select></label></section>
|
||||
<section id="excludeFields" class="conditional-fields" hidden><label><span>排除原因(分类统计)</span><select name="exclude_reason"><option value="">请选择</option><optgroup label="OUT OF SCOPE — 数据有效但不是目标行为"><option value="OUT_OF_SCOPE_NORMAL_COMPLETION">正常说完,没有 INT yield</option><option value="OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION">同一 wearer turn 内暂停/续说</option><option value="OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY">Other 在 wearer 结束后才开始</option><option value="OUT_OF_SCOPE_NON_EVENT">其他明确的非目标事件</option></optgroup><optgroup label="INSUFFICIENT EVIDENCE — 数据未坏但语义无法确定"><option value="INSUFFICIENT_EVIDENCE">现有证据不足以可靠判断</option></optgroup><optgroup label="DATA QUALITY — 数据或边界不可用"><option value="DATA_ANNOTATION_ERROR">标注明显错误或矛盾</option><option value="DATA_RECORDING_BOUNDARY">录制/clip 边界截断语境</option><option value="DATA_MISSING_CONTEXT">所需上下文缺失</option><option value="DATA_NON_SPEECH_ARTIFACT">非语音伪影</option><option value="DATA_TIMESTAMP_CORRUPTION">时间戳损坏或错位</option></optgroup></select></label></section>
|
||||
|
||||
<details class="boundary-details"><summary>仅在原 annotation 明显错误时校正边界</summary><label><span>可选校正(原时间始终保留)</span><select name="boundary_correction_seconds"><option value="">不校正</option><option value="-0.2">−0.2 s</option><option value="0.2">+0.2 s</option></select></label></details>
|
||||
|
||||
|
||||
+46
-44
@@ -233,14 +233,16 @@ participant timeline is retained to derive committed interaction events for
|
||||
behavioral evaluation.
|
||||
|
||||
\textbf{Behavioral event annotation.}
|
||||
We use floor-level wearer transitions as high-recall anchors for positive-event
|
||||
annotation. Each wearer onset is examined retrospectively for a preceding
|
||||
non-wearer floor release as contextual metadata and labeled as a wearer-entry
|
||||
whenever the subsequent wearer speech constitutes a genuine floor take,
|
||||
including re-entry while another speaker is active. Similarly, each wearer offset
|
||||
is examined for a new competing non-wearer onset during the current wearer turn
|
||||
and labeled as INT-yield only when the competing speaker genuinely takes over
|
||||
the floor. We use permissive temporal retrieval and let an LLM jointly inspect
|
||||
We use every observed wearer activity transition as a high-recall semantic
|
||||
candidate. For each wearer onset, nearby preceding non-wearer releases, the
|
||||
intervening silence gap, and concurrent activity are retained as contextual
|
||||
facts; the onset is labeled as wearer entry only when the subsequent wearer
|
||||
speech constitutes a genuine floor take, including re-entry while another
|
||||
speaker is active. For each wearer offset, nearby non-wearer onsets and offsets,
|
||||
pre-existing and newly active speakers, overlap durations, and subsequent
|
||||
speakers are retained without a timing-based takeover gate; the offset is
|
||||
labeled as INT-yield only when a non-wearer genuinely takes over and the wearer
|
||||
relinquishes the floor. We let an LLM jointly inspect
|
||||
participant-resolved timing and transcripts to provide a semantic pre-label,
|
||||
followed by mandatory human adjudication. Separately, we mine regions with no
|
||||
wearer response that nevertheless constitute false-positive opportunities:
|
||||
@@ -272,7 +274,9 @@ HN: other backchannel & \todo{--} & \todo{--} \\
|
||||
HN: non-takeover overlap & \todo{--} & \todo{--} \\
|
||||
HN: wearer holds floor & \todo{--} & \todo{--} \\
|
||||
\hline
|
||||
Excluded / fallback & \todo{--} & \todo{--} \\
|
||||
Excluded: valid but out of scope & \todo{--} & \todo{--} \\
|
||||
Excluded: insufficient evidence & \todo{--} & \todo{--} \\
|
||||
Excluded: data quality & \todo{--} & \todo{--} \\
|
||||
\textbf{Scoring-eligible total} & \todo{--} & \todo{--} \\
|
||||
\hline
|
||||
\end{tabular}
|
||||
@@ -557,42 +561,38 @@ because, by definition, many contain no wearer transition.
|
||||
|
||||
\subsection{Positive-event mining}
|
||||
|
||||
\textbf{Floor-level transitions.}
|
||||
Short wearer-silence gaps are conditionally merged before positive mining. In
|
||||
the current audit configuration, a raw onset after at most 0.8~s of silence is
|
||||
treated as a within-turn resumption unless the gap contains sustained competing
|
||||
non-wearer activity. Short feedback-like activity does not by itself split the
|
||||
wearer episode. This grouping changes only the behavioral-event layer, not the
|
||||
source intervals or dense targets.
|
||||
\textbf{Wearer activity transitions.}
|
||||
We preserve every raw wearer $0\!\rightarrow\!1$ and $1\!\rightarrow\!0$
|
||||
boundary as a semantic review candidate. Candidate construction does not merge
|
||||
short silence gaps, infer within-turn resumption, or decide whether overlapping
|
||||
speech constitutes takeover. It only records participant-resolved temporal and
|
||||
transcript facts. The source intervals and dense targets remain unchanged.
|
||||
|
||||
\textbf{Wearer entry.}
|
||||
Every floor-level wearer onset at $t_b$ is proposed as a wearer-entry candidate.
|
||||
We additionally search backward for plausible non-wearer floor releases. A
|
||||
release at $t_r$ is retained as a contextual option
|
||||
only when the interval from $t_r$ to $t_b$ contains neither intervening wearer
|
||||
speech nor a competing non-wearer onset. Each wearer onset contributes at most
|
||||
one review item: $t_b$ is its immutable \emph{behavior anchor}, and the item
|
||||
stores every admissible \emph{opportunity-anchor} option and response delay.
|
||||
Silence duration alone does not reject an option. A preceding release is not
|
||||
required for a positive entry: the wearer may enter after a release, from
|
||||
silence, or while another speaker remains active. When a release clearly
|
||||
prompts the entry, the LLM or human may select one supplied option; neither may
|
||||
invent or move a trigger timestamp. The release relation is metadata for
|
||||
response-oriented analyses, not part of the positive entry label.
|
||||
Every raw wearer onset at $t_b$ is proposed as one wearer-entry review item.
|
||||
We search backward within the retrieval window for non-wearer offsets and store
|
||||
each one with its response delay and factual indicators of intervening wearer
|
||||
speech, intervening other onsets, and concurrent activity. These are possible
|
||||
context anchors rather than deterministically admissible causes. The LLM and
|
||||
human reviewer decide whether the boundary is a genuine new entry, a
|
||||
within-turn resumption, a wearer backchannel, or an excluded non-event. A
|
||||
preceding release is not required for a positive entry: the wearer may enter
|
||||
after a release, from silence, or while another speaker remains active. When a
|
||||
release clearly prompts the entry, the LLM or human may select one supplied
|
||||
option; neither may invent or move a timestamp.
|
||||
|
||||
\textbf{INT-yield.}
|
||||
Starting from an episode-ending wearer offset $t_y$, we search backward inside
|
||||
the current wearer episode for a new competing non-wearer onset $t_o$. With
|
||||
$t_w$ denoting the wearer-episode start, an INT candidate must satisfy
|
||||
\begin{equation}
|
||||
t_w < t_o < t_y < t_o^{\mathrm{off}}.
|
||||
\label{eq:int_candidate}
|
||||
\end{equation}
|
||||
The current audit allows a one-frame (50-ms) tolerance only at the offset
|
||||
boundary. Pre-existing other activity, post-completion onsets, offsets inside a
|
||||
merged wearer episode, and completion-to-silence remain controls. Timing only
|
||||
retrieves a possible interruption; the LLM and human annotator decide whether
|
||||
the competitor genuinely takes over and the wearer yields.
|
||||
Every raw wearer offset $t_y$ is proposed as an INT-yield review item. The
|
||||
generator records the current wearer activity-run start, all nearby non-wearer
|
||||
onsets and offsets, activity immediately before and after $t_y$, whether an
|
||||
other speaker is pre-existing or begins during wearer speech, strict and
|
||||
50-ms-tolerant boundary crossing, overlap duration, the next speech onset,
|
||||
transcripts, and target-of-speech metadata. Relations such as
|
||||
$t_w<t_o<t_y<t_o^{\mathrm{off}}$ are descriptive evidence only: they neither
|
||||
admit nor reject a candidate and do not establish takeover. The LLM and human
|
||||
annotator answer whether the wearer stopped because a non-wearer genuinely took
|
||||
over the floor. Natural completion, post-completion activity, and within-turn
|
||||
pauses are therefore resolved semantically rather than pre-filtered.
|
||||
|
||||
\subsection{False-positive opportunity mining}
|
||||
|
||||
@@ -624,9 +624,11 @@ relabel it, or exclude an
|
||||
unscorable region. Only human-confirmed \textsc{Wearer-Entry},
|
||||
\textsc{INT-Yield}, and hard-negative opportunities enter evaluation;
|
||||
\textsc{Exclude} candidates do not. Thus \textsc{Exclude} is the sole
|
||||
non-scoring outcome for out-of-scope non-events (e.g., natural wearer completion
|
||||
that merely coincides with feedback), insufficient evidence, unresolvable data
|
||||
quality, or pre-label disagreement.
|
||||
non-scoring verdict, but every exclusion retains a closed-vocabulary reason and
|
||||
derived aggregation group. We report valid-but-out-of-scope behavior (e.g.,
|
||||
natural wearer completion coincident with feedback), insufficient semantic
|
||||
evidence, and data-quality failures separately; model-pass disagreement belongs
|
||||
to insufficient evidence and is never counted as corrupt data.
|
||||
Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
|
||||
and future observations are never supplied to the evaluated forecasting model.
|
||||
|
||||
|
||||
+236
-215
@@ -2,13 +2,12 @@
|
||||
"""Generate wearer-centric EasyCom entry/INT candidates for AI and human review.
|
||||
|
||||
This stage never changes EasyCom speech boundaries and never claims semantic
|
||||
ground truth. It groups short wearer resumptions into floor-level episodes,
|
||||
starts entry candidates from every observed floor-level wearer onset and
|
||||
retains preceding non-wearer release options as context, requires a new
|
||||
non-wearer onset inside the wearer floor episode for INT candidates, and keeps
|
||||
ineligible raw boundaries as deterministic controls. All timing
|
||||
thresholds are CLI parameters, and raw evidence is retained for DeepSeek
|
||||
pre-labeling and human correction.
|
||||
ground truth. Every observed wearer onset and offset is retained as a semantic
|
||||
candidate. Deterministic code only records participant-resolved timing,
|
||||
transcripts, and target-of-speech facts; it never decides floor entry,
|
||||
within-turn resumption, or takeover semantics. All timing windows are CLI
|
||||
parameters, and raw evidence is retained for DeepSeek pre-labeling and human
|
||||
correction.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -28,8 +27,8 @@ from derive_easycom_interaction_events import (
|
||||
from easycom_manifest import ALL_SESSIONS, TEST_SESSIONS
|
||||
|
||||
|
||||
SCHEMA_VERSION = 5
|
||||
PROTOCOL_VERSION = "easycom-wearer-entry-int-candidates-v7"
|
||||
SCHEMA_VERSION = 6
|
||||
PROTOCOL_VERSION = "easycom-wearer-entry-int-candidates-v8"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -41,7 +40,6 @@ class CandidateConfig:
|
||||
short_vocalization_max_seconds: float = 1.0
|
||||
continuation_min_seconds: float = 0.4
|
||||
int_offset_tolerance_seconds: float = 0.05
|
||||
wearer_resume_merge_max_seconds: float = 0.8
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -88,26 +86,13 @@ class ProtocolCandidate:
|
||||
def requires_semantic_adjudication(row: dict[str, Any]) -> bool:
|
||||
"""Return whether a stored row belongs in the primary AI/human queue.
|
||||
|
||||
Explicit legacy control sources are authoritative. For old v2 candidate
|
||||
files, reconstruct the same-speaker crossing requirement from the stored
|
||||
activity IDs. Missing legacy evidence remains reviewable rather than being
|
||||
silently discarded.
|
||||
Explicit legacy control sources are authoritative. Current WEARER_OFFSET
|
||||
rows are always semantic candidates: crossing/order fields are evidence,
|
||||
never admission gates.
|
||||
"""
|
||||
source = row.get("candidate_source")
|
||||
if source in {"WEARER_ONSET_CONTROL", "WEARER_OFFSET_CONTROL", "WEARER_RESUME"}:
|
||||
return False
|
||||
if source != "WEARER_OFFSET":
|
||||
return True
|
||||
evidence = row.get("temporal_evidence")
|
||||
if not isinstance(evidence, dict):
|
||||
return True
|
||||
explicit = evidence.get("other_crosses_wearer_offset")
|
||||
if isinstance(explicit, bool):
|
||||
return explicit
|
||||
before = evidence.get("active_other_ids_before_offset")
|
||||
after = evidence.get("active_other_ids_after_offset")
|
||||
if isinstance(before, list) and isinstance(after, list):
|
||||
return bool(set(before) & set(after))
|
||||
return True
|
||||
|
||||
|
||||
@@ -157,7 +142,7 @@ def _other_ids_crossing_offset(
|
||||
) -> tuple[int, ...]:
|
||||
"""Return other speakers whose annotated interval crosses a wearer offset.
|
||||
|
||||
With zero tolerance this encodes the strict prerequisite
|
||||
With zero tolerance this records the strict temporal relation
|
||||
``other.start < wearer_offset < other.end``. A one-frame tolerance accepts
|
||||
an annotation boundary at the same 20 Hz frame, but not an other onset one
|
||||
or more frames after the wearer offset.
|
||||
@@ -170,26 +155,19 @@ def _other_ids_crossing_offset(
|
||||
}))
|
||||
|
||||
|
||||
def _wearer_gap_competition(
|
||||
def _wearer_gap_evidence(
|
||||
rows: Sequence[Utterance], *, gap_start_frame: int, onset_frame: int,
|
||||
wearer_id: int, short_frames: int, continuation_frames: int,
|
||||
wearer_id: int,
|
||||
) -> dict[str, Any]:
|
||||
"""Summarize plausible competing floor activity inside a short wearer gap.
|
||||
|
||||
Feedback-like short lexical/nonverbal activity is retained as evidence but
|
||||
does not by itself split the wearer floor episode. Any other contribution
|
||||
that occupies the gap for ``continuation_frames`` or continues across the
|
||||
resumed wearer onset conservatively preserves a new ENTRY candidate.
|
||||
"""
|
||||
"""Return descriptive other-speaker facts within a wearer-silence gap."""
|
||||
gap_rows = [
|
||||
row for row in rows
|
||||
if row.participant_id != wearer_id
|
||||
and row.dense_start_frame < onset_frame
|
||||
and row.dense_end_frame_exclusive > gap_start_frame
|
||||
]
|
||||
feedback_ids: set[int] = set()
|
||||
competing_ids: set[int] = set()
|
||||
max_gap_coverage = 0
|
||||
intervals: list[dict[str, Any]] = []
|
||||
for row in gap_rows:
|
||||
coverage = max(
|
||||
0,
|
||||
@@ -197,20 +175,20 @@ def _wearer_gap_competition(
|
||||
- max(row.dense_start_frame, gap_start_frame),
|
||||
)
|
||||
max_gap_coverage = max(max_gap_coverage, coverage)
|
||||
feedback_like = (
|
||||
row.dense_end_frame_exclusive - row.dense_start_frame <= short_frames
|
||||
and (is_lexical_feedback(row.transcription) or is_pure_nonverbal(row.transcription))
|
||||
)
|
||||
if feedback_like:
|
||||
feedback_ids.add(row.participant_id)
|
||||
elif coverage >= continuation_frames or row.dense_end_frame_exclusive > onset_frame:
|
||||
competing_ids.add(row.participant_id)
|
||||
intervals.append({
|
||||
"uid": row.uid,
|
||||
"participant_id": row.participant_id,
|
||||
"onset_seconds": row.dense_start_seconds,
|
||||
"offset_seconds": row.dense_end_seconds,
|
||||
"gap_overlap_seconds": coverage / SOURCE_FPS,
|
||||
"continues_across_wearer_onset": row.dense_end_frame_exclusive > onset_frame,
|
||||
"transcription": row.transcription,
|
||||
"target_of_speech": list(row.target_of_speech),
|
||||
})
|
||||
return {
|
||||
"gap_other_participant_ids": sorted({row.participant_id for row in gap_rows}),
|
||||
"feedback_like_other_ids_during_gap": sorted(feedback_ids),
|
||||
"sustained_competing_other_ids_during_gap": sorted(competing_ids),
|
||||
"gap_other_intervals": intervals,
|
||||
"max_other_gap_coverage_seconds": max_gap_coverage / SOURCE_FPS,
|
||||
"sustained_other_takeover_during_gap": bool(competing_ids),
|
||||
}
|
||||
|
||||
|
||||
@@ -327,7 +305,6 @@ def derive_protocol_candidates(utterances: Sequence[Utterance], clips: Sequence[
|
||||
short_frames = round(cfg.short_vocalization_max_seconds * SOURCE_FPS)
|
||||
continuation_frames = round(cfg.continuation_min_seconds * SOURCE_FPS)
|
||||
int_offset_tolerance_frames = round(cfg.int_offset_tolerance_seconds * SOURCE_FPS)
|
||||
wearer_resume_merge_frames = round(cfg.wearer_resume_merge_max_seconds * SOURCE_FPS)
|
||||
rows_by_segment: dict[int, list[Utterance]] = defaultdict(list)
|
||||
clips_by_segment: dict[int, list[Clip]] = defaultdict(list)
|
||||
for row in utterances:
|
||||
@@ -352,10 +329,8 @@ def derive_protocol_candidates(utterances: Sequence[Utterance], clips: Sequence[
|
||||
segment_end=segment_end, wearer_id=wearer_id)
|
||||
other_rows = [row for row in rows if row.participant_id != wearer_id]
|
||||
|
||||
# A: classify raw wearer onsets before proposing any behavioral event.
|
||||
# Short uncompeted gaps are internal to a floor episode. All other raw
|
||||
# onsets begin a new floor episode and are all entry candidates. A
|
||||
# preceding non-wearer release is optional context, not a gate.
|
||||
# A: retain every raw wearer onset and offset. Gap and nearby-speaker
|
||||
# properties are evidence only; they never suppress a semantic item.
|
||||
onset_records: dict[int, dict[str, Any]] = {}
|
||||
raw_offsets: list[int] = []
|
||||
for index in range(1, len(wearer)):
|
||||
@@ -365,25 +340,16 @@ def derive_protocol_candidates(utterances: Sequence[Utterance], clips: Sequence[
|
||||
previous_wearer_index = index - gap_frames - 1
|
||||
has_previous_wearer = previous_wearer_index >= 0 and bool(wearer[previous_wearer_index])
|
||||
previous_wearer_offset = segment_start + index - gap_frames
|
||||
competition = _wearer_gap_competition(
|
||||
gap_evidence = _wearer_gap_evidence(
|
||||
rows,
|
||||
gap_start_frame=previous_wearer_offset,
|
||||
onset_frame=frame,
|
||||
wearer_id=wearer_id,
|
||||
short_frames=short_frames,
|
||||
continuation_frames=continuation_frames,
|
||||
) if has_previous_wearer else {
|
||||
"gap_other_participant_ids": [],
|
||||
"feedback_like_other_ids_during_gap": [],
|
||||
"sustained_competing_other_ids_during_gap": [],
|
||||
"gap_other_intervals": [],
|
||||
"max_other_gap_coverage_seconds": 0.0,
|
||||
"sustained_other_takeover_during_gap": False,
|
||||
}
|
||||
within_turn_resume = (
|
||||
has_previous_wearer
|
||||
and gap_frames <= wearer_resume_merge_frames
|
||||
and not competition["sustained_other_takeover_during_gap"]
|
||||
)
|
||||
evidence = _relation_evidence(rows, frame=frame, wearer_id=wearer_id,
|
||||
window_frames=relation_frames)
|
||||
evidence.update(
|
||||
@@ -392,166 +358,112 @@ def derive_protocol_candidates(utterances: Sequence[Utterance], clips: Sequence[
|
||||
previous_wearer_offset_seconds=(
|
||||
previous_wearer_offset / SOURCE_FPS if has_previous_wearer else None
|
||||
),
|
||||
wearer_resume_merge_max_seconds=cfg.wearer_resume_merge_max_seconds,
|
||||
floor_episode_merged=within_turn_resume,
|
||||
original_timestamp_seconds=frame / SOURCE_FPS,
|
||||
**competition,
|
||||
**gap_evidence,
|
||||
)
|
||||
onset_records[frame] = {
|
||||
"within_turn_resume": within_turn_resume,
|
||||
"evidence": evidence,
|
||||
}
|
||||
onset_records[frame] = {"evidence": evidence}
|
||||
elif wearer[index - 1] and not wearer[index]:
|
||||
raw_offsets.append(frame)
|
||||
|
||||
resume_onsets = {
|
||||
frame for frame, record in onset_records.items() if record["within_turn_resume"]
|
||||
}
|
||||
floor_onsets = sorted(set(onset_records) - resume_onsets)
|
||||
|
||||
for frame in sorted(resume_onsets):
|
||||
record = onset_records[frame]
|
||||
add(_candidate(
|
||||
session=session, task="wearer_entry_control", source="WEARER_RESUME",
|
||||
rule_type="WITHIN_TURN_RESUME_CONTROL", anchor_type="wearer_onset", frame=frame,
|
||||
region_start=frame, region_end=min(segment_end, frame + relation_frames),
|
||||
label="control", subtype=None,
|
||||
previous_ids=_active_other_ids(rows, frame - 1, wearer_id=wearer_id),
|
||||
incoming_ids=_active_other_ids(rows, frame, wearer_id=wearer_id),
|
||||
evidence=record["evidence"], clips=segment_clips, rows=rows,
|
||||
wearer_id=wearer_id, context_frames=context_frames,
|
||||
reason="short raw wearer pause without sustained competing other takeover; merged into the existing floor episode",
|
||||
), ("WEARER_RESUME", frame, ()))
|
||||
|
||||
# A1: every observed floor-level wearer onset is an entry candidate.
|
||||
# Search backward for release anchors only to preserve response context;
|
||||
# no preceding release is required for a positive WEARER_ENTRY.
|
||||
for onset_frame in floor_onsets:
|
||||
# A1: every observed raw wearer onset is an unresolved entry candidate.
|
||||
# Every nearby preceding other offset is supplied with factual relation
|
||||
# flags; no release is declared admissible or causal by this stage.
|
||||
for onset_frame in sorted(onset_records):
|
||||
release_options: list[tuple[Utterance, int]] = []
|
||||
for release in other_rows:
|
||||
opportunity_frame = release.dense_end_frame_exclusive
|
||||
if opportunity_frame > onset_frame:
|
||||
if opportunity_frame > onset_frame or onset_frame - opportunity_frame > relation_frames:
|
||||
continue
|
||||
wearer_silent = (
|
||||
opportunity_frame == onset_frame
|
||||
or _all_state(
|
||||
wearer,
|
||||
opportunity_frame - segment_start,
|
||||
onset_frame - segment_start,
|
||||
False,
|
||||
)
|
||||
)
|
||||
competing_onset = any(
|
||||
opportunity_frame <= row.dense_start_frame < onset_frame
|
||||
for row in other_rows
|
||||
)
|
||||
other_already_holds = bool(
|
||||
_active_other_ids(rows, opportunity_frame, wearer_id=wearer_id)
|
||||
)
|
||||
if wearer_silent and not competing_onset and not other_already_holds:
|
||||
release_options.append((release, opportunity_frame))
|
||||
|
||||
release_options.sort(
|
||||
key=lambda item: (item[1], item[0].participant_id, item[0].uid)
|
||||
)
|
||||
if release_options:
|
||||
_, latest_opportunity_frame = release_options[-1]
|
||||
earliest_opportunity_frame = release_options[0][1]
|
||||
option_rows = [{
|
||||
earliest_opportunity_frame = (
|
||||
release_options[0][1] if release_options else max(segment_start, onset_frame - relation_frames)
|
||||
)
|
||||
option_rows = []
|
||||
for release, release_frame in release_options:
|
||||
interval_left = release_frame - segment_start
|
||||
interval_right = onset_frame - segment_start
|
||||
option_rows.append({
|
||||
"uid": release.uid,
|
||||
"anchor_type": "other_offset",
|
||||
"anchor_frame": frame,
|
||||
"anchor_seconds": frame / SOURCE_FPS,
|
||||
"anchor_frame": release_frame,
|
||||
"anchor_seconds": release_frame / SOURCE_FPS,
|
||||
"participant_id": release.participant_id,
|
||||
"transcription": release.transcription,
|
||||
"target_of_speech": list(release.target_of_speech),
|
||||
"response_delay_seconds": (onset_frame - frame) / SOURCE_FPS,
|
||||
} for release, frame in release_options]
|
||||
"response_delay_seconds": (onset_frame - release_frame) / SOURCE_FPS,
|
||||
"wearer_active_between_release_and_onset": any(
|
||||
bool(wearer[i]) for i in range(max(0, interval_left), min(len(wearer), interval_right))
|
||||
),
|
||||
"other_onset_between_release_and_onset": any(
|
||||
release_frame <= row.dense_start_frame < onset_frame for row in other_rows
|
||||
),
|
||||
"other_active_at_wearer_onset": bool(
|
||||
_active_other_ids(rows, onset_frame, wearer_id=wearer_id)
|
||||
),
|
||||
})
|
||||
evidence = dict(onset_records[onset_frame]["evidence"])
|
||||
evidence.update(
|
||||
opportunity_options=option_rows,
|
||||
opportunity_anchor_type=None,
|
||||
opportunity_anchor_frame=None,
|
||||
opportunity_anchor_seconds=None,
|
||||
behavior_anchor_type="wearer_floor_onset",
|
||||
behavior_anchor_type="wearer_raw_onset",
|
||||
behavior_anchor_frame=onset_frame,
|
||||
behavior_anchor_seconds=onset_frame / SOURCE_FPS,
|
||||
minimum_response_delay_seconds=(onset_frame - latest_opportunity_frame) / SOURCE_FPS,
|
||||
maximum_response_delay_seconds=(onset_frame - earliest_opportunity_frame) / SOURCE_FPS,
|
||||
competing_other_onsets_before_behavior=[],
|
||||
preceding_release_paired=True,
|
||||
preceding_release_options_present=bool(option_rows),
|
||||
minimum_response_delay_seconds=(
|
||||
None if not option_rows
|
||||
else min(option["response_delay_seconds"] for option in option_rows)
|
||||
),
|
||||
maximum_response_delay_seconds=(
|
||||
None if not option_rows
|
||||
else max(option["response_delay_seconds"] for option in option_rows)
|
||||
),
|
||||
entry_context_timing=(
|
||||
"OTHER_ACTIVE"
|
||||
if _active_other_ids(rows, onset_frame, wearer_id=wearer_id)
|
||||
else "AFTER_OTHER_RELEASE"
|
||||
else "NO_OTHER_ACTIVE_AT_ONSET"
|
||||
),
|
||||
)
|
||||
previous_ids = tuple(sorted({row.participant_id for row, _ in release_options}))
|
||||
add(_candidate(
|
||||
session=session, task="wearer_entry", source="WEARER_ONSET",
|
||||
rule_type="WEARER_ENTRY_CANDIDATE", anchor_type="wearer_onset",
|
||||
frame=onset_frame, region_start=earliest_opportunity_frame,
|
||||
rule_type="WEARER_ENTRY_CANDIDATE", anchor_type="wearer_onset", frame=onset_frame,
|
||||
region_start=earliest_opportunity_frame,
|
||||
region_end=min(segment_end, onset_frame + relation_frames),
|
||||
label="candidate", subtype=None, previous_ids=previous_ids,
|
||||
label="candidate", subtype=None,
|
||||
previous_ids=previous_ids,
|
||||
incoming_ids=_active_other_ids(rows, onset_frame, wearer_id=wearer_id),
|
||||
evidence=evidence, clips=segment_clips, rows=rows, wearer_id=wearer_id,
|
||||
context_frames=context_frames,
|
||||
reason="floor-level wearer onset; preceding release options retained as context",
|
||||
), ("WEARER_ONSET", onset_frame, previous_ids))
|
||||
continue
|
||||
|
||||
frame = onset_frame
|
||||
evidence = dict(onset_records[frame]["evidence"])
|
||||
evidence.update(
|
||||
opportunity_options=[],
|
||||
opportunity_anchor_type=None,
|
||||
opportunity_anchor_frame=None,
|
||||
opportunity_anchor_seconds=None,
|
||||
behavior_anchor_type="wearer_floor_onset",
|
||||
behavior_anchor_frame=frame,
|
||||
behavior_anchor_seconds=frame / SOURCE_FPS,
|
||||
preceding_release_paired=False,
|
||||
entry_context_timing=(
|
||||
"OTHER_ACTIVE"
|
||||
if _active_other_ids(rows, frame, wearer_id=wearer_id)
|
||||
else "FROM_SILENCE_OR_UNRESOLVED"
|
||||
reason=(
|
||||
"raw wearer onset retained without floor-entry or within-turn-resumption gating; "
|
||||
"nearby release relations are descriptive evidence"
|
||||
),
|
||||
)
|
||||
previous_ids = _active_other_ids(rows, frame - 1, wearer_id=wearer_id)
|
||||
add(_candidate(
|
||||
session=session, task="wearer_entry", source="WEARER_ONSET",
|
||||
rule_type="WEARER_ENTRY_CANDIDATE", anchor_type="wearer_onset", frame=frame,
|
||||
region_start=max(segment_start, frame - relation_frames),
|
||||
region_end=min(segment_end, frame + relation_frames),
|
||||
label="candidate", subtype=None,
|
||||
previous_ids=previous_ids,
|
||||
incoming_ids=_active_other_ids(rows, frame, wearer_id=wearer_id),
|
||||
evidence=evidence, clips=segment_clips, rows=rows, wearer_id=wearer_id,
|
||||
context_frames=context_frames,
|
||||
reason="floor-level wearer onset without a surviving preceding release option",
|
||||
), ("WEARER_ONSET", frame, previous_ids))
|
||||
), ("WEARER_ONSET", onset_frame, previous_ids))
|
||||
|
||||
# A2: a raw wearer offset is an INT candidate only at the end of a
|
||||
# floor-level wearer episode and only if a new non-wearer onset obeys
|
||||
# t_w < t_o < t_y and that same interval crosses the yield boundary.
|
||||
# All other offsets remain auditable controls.
|
||||
ordered_raw_onsets = sorted(onset_records)
|
||||
current_floor_start = segment_start if wearer and wearer[0] else None
|
||||
floor_start_by_offset: dict[int, tuple[int | None, bool]] = {}
|
||||
# A2: every raw wearer offset is an unresolved INT candidate. The code
|
||||
# records temporal relationships but does not infer takeover semantics.
|
||||
# In particular, new/pre-existing/post-offset other activity and
|
||||
# boundary crossing are evidence for DeepSeek and the human reviewer,
|
||||
# never deterministic admission gates.
|
||||
current_activity_run_start = segment_start if wearer and wearer[0] else None
|
||||
activity_run_start_by_offset: dict[int, int | None] = {}
|
||||
for index in range(1, len(wearer)):
|
||||
frame = segment_start + index
|
||||
if not wearer[index - 1] and wearer[index] and frame not in resume_onsets:
|
||||
current_floor_start = frame
|
||||
if not wearer[index - 1] and wearer[index]:
|
||||
current_activity_run_start = frame
|
||||
elif wearer[index - 1] and not wearer[index]:
|
||||
next_onset = next((value for value in ordered_raw_onsets if value > frame), None)
|
||||
internal_pause = next_onset in resume_onsets
|
||||
floor_start_by_offset[frame] = (current_floor_start, internal_pause)
|
||||
if not internal_pause:
|
||||
current_floor_start = None
|
||||
activity_run_start_by_offset[frame] = current_activity_run_start
|
||||
current_activity_run_start = None
|
||||
|
||||
for frame in raw_offsets:
|
||||
index = frame - segment_start
|
||||
floor_start, internal_pause = floor_start_by_offset[frame]
|
||||
activity_run_start = activity_run_start_by_offset[frame]
|
||||
strict_crossing_ids = _other_ids_crossing_offset(
|
||||
other_rows, frame=frame, wearer_id=wearer_id, tolerance_frames=0
|
||||
)
|
||||
@@ -559,71 +471,178 @@ def derive_protocol_candidates(utterances: Sequence[Utterance], clips: Sequence[
|
||||
other_rows, frame=frame, wearer_id=wearer_id,
|
||||
tolerance_frames=int_offset_tolerance_frames,
|
||||
)
|
||||
eligible_rows = [
|
||||
new_other_rows = [
|
||||
row for row in other_rows
|
||||
if floor_start is not None
|
||||
and floor_start < row.dense_start_frame < frame
|
||||
and row.dense_end_frame_exclusive > frame - int_offset_tolerance_frames
|
||||
if activity_run_start is not None
|
||||
and activity_run_start < row.dense_start_frame < frame
|
||||
]
|
||||
eligible_ids = tuple(sorted({row.participant_id for row in eligible_rows}))
|
||||
preexisting_ids = tuple(sorted({
|
||||
row.participant_id for row in other_rows
|
||||
if floor_start is not None
|
||||
and row.dense_start_frame <= floor_start
|
||||
new_crossing_rows = [
|
||||
row for row in new_other_rows
|
||||
if row.dense_end_frame_exclusive > frame - int_offset_tolerance_frames
|
||||
]
|
||||
new_other_ids = tuple(sorted({row.participant_id for row in new_other_rows}))
|
||||
new_crossing_ids = tuple(sorted({row.participant_id for row in new_crossing_rows}))
|
||||
preexisting_rows = [
|
||||
row for row in other_rows
|
||||
if activity_run_start is not None
|
||||
and row.dense_start_frame <= activity_run_start
|
||||
and row.dense_end_frame_exclusive > activity_run_start
|
||||
]
|
||||
preexisting_ids = tuple(sorted({row.participant_id for row in preexisting_rows}))
|
||||
nearby_other_rows = [
|
||||
row for row in other_rows
|
||||
if row.dense_end_frame_exclusive > frame - relation_frames
|
||||
and row.dense_start_frame < frame + relation_frames
|
||||
]
|
||||
nearby_other_intervals = [{
|
||||
"uid": row.uid,
|
||||
"participant_id": row.participant_id,
|
||||
"onset_seconds": row.dense_start_seconds,
|
||||
"offset_seconds": row.dense_end_seconds,
|
||||
"onset_relative_to_wearer_offset_seconds": (
|
||||
row.dense_start_frame - frame
|
||||
) / SOURCE_FPS,
|
||||
"offset_relative_to_wearer_offset_seconds": (
|
||||
row.dense_end_frame_exclusive - frame
|
||||
) / SOURCE_FPS,
|
||||
"starts_during_wearer_activity_run": bool(
|
||||
activity_run_start is not None
|
||||
and activity_run_start < row.dense_start_frame < frame
|
||||
),
|
||||
"preexisting_at_wearer_activity_run_start": bool(
|
||||
activity_run_start is not None
|
||||
and row.dense_start_frame <= activity_run_start
|
||||
and row.dense_end_frame_exclusive > activity_run_start
|
||||
),
|
||||
"active_immediately_before_offset": bool(
|
||||
row.dense_start_frame <= frame - 1 < row.dense_end_frame_exclusive
|
||||
),
|
||||
"active_immediately_after_offset": bool(
|
||||
row.dense_start_frame <= frame < row.dense_end_frame_exclusive
|
||||
),
|
||||
"strictly_crosses_offset": bool(
|
||||
row.dense_start_frame < frame < row.dense_end_frame_exclusive
|
||||
),
|
||||
"crosses_offset_with_tolerance": bool(
|
||||
row.dense_start_frame < frame + int_offset_tolerance_frames
|
||||
and row.dense_end_frame_exclusive > frame - int_offset_tolerance_frames
|
||||
}))
|
||||
is_int_candidate = bool(eligible_ids) and not internal_pause
|
||||
),
|
||||
"overlap_with_wearer_before_offset_seconds": (
|
||||
0.0 if activity_run_start is None else max(
|
||||
0, min(row.dense_end_frame_exclusive, frame)
|
||||
- max(row.dense_start_frame, activity_run_start)
|
||||
) / SOURCE_FPS
|
||||
),
|
||||
"transcription": row.transcription,
|
||||
"target_of_speech": list(row.target_of_speech),
|
||||
} for row in nearby_other_rows]
|
||||
next_rows = sorted(
|
||||
[
|
||||
row for row in rows
|
||||
if frame <= row.dense_start_frame <= frame + relation_frames
|
||||
],
|
||||
key=lambda row: (row.dense_start_frame, row.participant_id, row.uid),
|
||||
)
|
||||
first_next_frame = next_rows[0].dense_start_frame if next_rows else None
|
||||
first_next_rows = (
|
||||
[] if first_next_frame is None
|
||||
else [row for row in next_rows if row.dense_start_frame == first_next_frame]
|
||||
)
|
||||
active_other_ids_after_offset = _active_other_ids(
|
||||
rows, frame, wearer_id=wearer_id
|
||||
)
|
||||
next_speaker_ids = (
|
||||
list(active_other_ids_after_offset)
|
||||
if active_other_ids_after_offset
|
||||
else [row.participant_id for row in first_next_rows]
|
||||
)
|
||||
next_speaker_roles = (
|
||||
["other"] * len(active_other_ids_after_offset)
|
||||
if active_other_ids_after_offset
|
||||
else [
|
||||
"wearer" if row.participant_id == wearer_id else "other"
|
||||
for row in first_next_rows
|
||||
]
|
||||
)
|
||||
evidence = _relation_evidence(
|
||||
rows, frame=frame, wearer_id=wearer_id, window_frames=relation_frames
|
||||
)
|
||||
evidence.update(
|
||||
wearer_floor_episode_start_frame=floor_start,
|
||||
wearer_floor_episode_start_seconds=(
|
||||
None if floor_start is None else floor_start / SOURCE_FPS
|
||||
wearer_activity_run_start_frame=activity_run_start,
|
||||
wearer_activity_run_start_seconds=(
|
||||
None if activity_run_start is None else activity_run_start / SOURCE_FPS
|
||||
),
|
||||
wearer_activity_run_duration_to_offset_seconds=(
|
||||
None if activity_run_start is None
|
||||
else (frame - activity_run_start) / SOURCE_FPS
|
||||
),
|
||||
wearer_offset_frame=frame,
|
||||
wearer_offset_seconds=frame / SOURCE_FPS,
|
||||
wearer_speech_duration_before_offset_seconds=_run_before(wearer, index, True) / SOURCE_FPS,
|
||||
wearer_silence_duration_after_offset_seconds=_run_after(wearer, index, False) / SOURCE_FPS,
|
||||
raw_offset_is_within_merged_floor_episode=internal_pause,
|
||||
active_other_ids_before_offset=list(_active_other_ids(rows, frame - 1, wearer_id=wearer_id)),
|
||||
active_other_ids_after_offset=list(_active_other_ids(rows, frame, wearer_id=wearer_id)),
|
||||
active_other_ids_after_offset=list(active_other_ids_after_offset),
|
||||
strict_crossing_other_ids=list(strict_crossing_ids),
|
||||
tolerance_crossing_other_ids=list(tolerance_crossing_ids),
|
||||
eligible_new_other_onset_ids=list(eligible_ids),
|
||||
eligible_new_other_onsets=[{
|
||||
new_other_onset_ids_during_wearer=list(new_other_ids),
|
||||
new_other_onsets_during_wearer=[{
|
||||
"participant_id": row.participant_id,
|
||||
"onset_seconds": row.dense_start_seconds,
|
||||
"offset_seconds": row.dense_end_seconds,
|
||||
"uid": row.uid,
|
||||
} for row in eligible_rows],
|
||||
preexisting_crossing_other_ids=list(preexisting_ids),
|
||||
} for row in new_other_rows],
|
||||
new_other_onset_ids_crossing_offset=list(new_crossing_ids),
|
||||
preexisting_other_ids_at_wearer_activity_run_start=list(preexisting_ids),
|
||||
preexisting_other_intervals=[{
|
||||
"participant_id": row.participant_id,
|
||||
"onset_seconds": row.dense_start_seconds,
|
||||
"offset_seconds": row.dense_end_seconds,
|
||||
"uid": row.uid,
|
||||
} for row in preexisting_rows],
|
||||
nearby_other_intervals=nearby_other_intervals,
|
||||
next_speech_onsets=[{
|
||||
"participant_id": row.participant_id,
|
||||
"participant_role": "wearer" if row.participant_id == wearer_id else "other",
|
||||
"onset_seconds": row.dense_start_seconds,
|
||||
"offset_seconds": row.dense_end_seconds,
|
||||
"uid": row.uid,
|
||||
"transcription": row.transcription,
|
||||
"target_of_speech": list(row.target_of_speech),
|
||||
} for row in next_rows],
|
||||
next_speaker_ids=next_speaker_ids,
|
||||
next_speaker_roles=next_speaker_roles,
|
||||
next_speaker_basis=(
|
||||
"active_across_offset"
|
||||
if active_other_ids_after_offset
|
||||
else ("first_new_onset_after_offset" if first_next_rows else None)
|
||||
),
|
||||
next_speech_onset_seconds=(
|
||||
frame / SOURCE_FPS
|
||||
if active_other_ids_after_offset
|
||||
else (None if first_next_frame is None else first_next_frame / SOURCE_FPS)
|
||||
),
|
||||
other_crosses_wearer_offset=bool(tolerance_crossing_ids),
|
||||
int_ordering_satisfied=bool(eligible_ids),
|
||||
new_other_onset_during_wearer=bool(new_other_rows),
|
||||
new_other_onset_crosses_offset=bool(new_crossing_rows),
|
||||
int_offset_tolerance_seconds=cfg.int_offset_tolerance_seconds,
|
||||
original_timestamp_seconds=frame / SOURCE_FPS,
|
||||
)
|
||||
control_reason = (
|
||||
"raw wearer offset is internal to a merged floor episode"
|
||||
if internal_pause
|
||||
else "no new non-wearer onset satisfies t_w < t_o < t_y and crosses the wearer offset"
|
||||
)
|
||||
add(_candidate(
|
||||
session=session,
|
||||
task="int_yield" if is_int_candidate else "int_yield_control",
|
||||
source="WEARER_OFFSET" if is_int_candidate else "WEARER_OFFSET_CONTROL",
|
||||
rule_type="YIELD_CANDIDATE" if is_int_candidate else "WEARER_OFFSET_CONTROL",
|
||||
session=session, task="int_yield", source="WEARER_OFFSET",
|
||||
rule_type="YIELD_CANDIDATE",
|
||||
anchor_type="wearer_offset", frame=frame,
|
||||
region_start=max(segment_start, frame - relation_frames),
|
||||
region_end=min(segment_end, frame + relation_frames),
|
||||
label="candidate" if is_int_candidate else "control", subtype=None,
|
||||
label="candidate", subtype=None,
|
||||
previous_ids=_active_other_ids(rows, frame - 1, wearer_id=wearer_id),
|
||||
incoming_ids=eligible_ids,
|
||||
incoming_ids=new_other_ids,
|
||||
evidence=evidence, clips=segment_clips, rows=rows, wearer_id=wearer_id,
|
||||
context_frames=context_frames,
|
||||
reason=(
|
||||
"new non-wearer onset occurs inside the wearer floor episode and crosses the yield boundary; semantic takeover status is unresolved"
|
||||
if is_int_candidate else control_reason
|
||||
"raw wearer offset retained without timing-based takeover gating; "
|
||||
"all nearby activity relations are descriptive evidence"
|
||||
),
|
||||
), ("WEARER_OFFSET" if is_int_candidate else "WEARER_OFFSET_CONTROL", frame, eligible_ids))
|
||||
), ("WEARER_OFFSET", frame, new_other_ids))
|
||||
|
||||
# B1: same speaker pauses and resumes while the wearer remains silent.
|
||||
by_speaker: dict[int, list[Utterance]] = defaultdict(list)
|
||||
@@ -762,7 +781,10 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument("--short-vocalization-max-seconds", type=float, default=1.0)
|
||||
parser.add_argument("--continuation-min-seconds", type=float, default=0.4)
|
||||
parser.add_argument("--int-offset-tolerance-seconds", type=float, default=0.05)
|
||||
parser.add_argument("--wearer-resume-merge-max-seconds", type=float, default=0.8)
|
||||
parser.add_argument(
|
||||
"--wearer-resume-merge-max-seconds", type=float,
|
||||
help=argparse.SUPPRESS,
|
||||
) # Deprecated compatibility option; raw wearer boundaries are never merged.
|
||||
parser.add_argument("--candidate-sources", nargs="+")
|
||||
parser.add_argument("--max-candidates", type=int)
|
||||
parser.add_argument("--selection-seed", type=int, default=2027)
|
||||
@@ -782,7 +804,6 @@ def main() -> None:
|
||||
short_vocalization_max_seconds=args.short_vocalization_max_seconds,
|
||||
continuation_min_seconds=args.continuation_min_seconds,
|
||||
int_offset_tolerance_seconds=args.int_offset_tolerance_seconds,
|
||||
wearer_resume_merge_max_seconds=args.wearer_resume_merge_max_seconds,
|
||||
)
|
||||
dataset_root, output_dir = args.dataset_root.expanduser().resolve(), args.output_dir.expanduser().resolve()
|
||||
all_rows: list[ProtocolCandidate] = []
|
||||
|
||||
+23
-12
@@ -27,6 +27,9 @@ from threading import Lock
|
||||
from typing import Any, Iterable
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
from easycom_event_taxonomy import (
|
||||
EXCLUDE_REASONS, LEGACY_EXCLUDE_REASONS, exclude_reason_group,
|
||||
)
|
||||
from prepare_easycom_eot_int_review import requires_semantic_adjudication
|
||||
|
||||
|
||||
@@ -34,7 +37,7 @@ SOURCE_FPS = 20.0
|
||||
STATIC_ROOT = Path(__file__).resolve().parent / "floor_event_review_web"
|
||||
VIDEO_ROOT_PARTS = ("Main", "Video_Compressed")
|
||||
TRANSCRIPT_ROOT_PARTS = ("Main", "Speech_Transcriptions")
|
||||
REVIEW_SCHEMA_VERSION = 5
|
||||
REVIEW_SCHEMA_VERSION = 6
|
||||
MAX_REQUEST_BYTES = 128 * 1024
|
||||
|
||||
PROTOCOL_DISPOSITION_VALUES = {"positive", "hard_negative", "exclude"}
|
||||
@@ -71,11 +74,7 @@ HUMAN_SUBTYPE_VALUES = {
|
||||
"INT_OTHER_BACKCHANNEL", "INT_NON_TAKEOVER_OVERLAP", "INT_WEARER_HOLDS_FLOOR",
|
||||
"OTHER_HARD_NEGATIVE",
|
||||
}
|
||||
EXCLUDE_REASON_VALUES = {
|
||||
"ANNOTATION_ERROR", "RECORDING_BOUNDARY", "MISSING_CONTEXT", "NON_SPEECH_ARTIFACT",
|
||||
"TIMESTAMP_CORRUPTION", "OUT_OF_SCOPE_NORMAL_COMPLETION", "OUT_OF_SCOPE_NON_EVENT",
|
||||
"UNRESOLVABLE",
|
||||
}
|
||||
EXCLUDE_REASON_VALUES = EXCLUDE_REASONS | LEGACY_EXCLUDE_REASONS
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -341,6 +340,8 @@ class CandidateRepository:
|
||||
"ai_verdict": row.get("ai_verdict"),
|
||||
"ai_event_type": row.get("ai_event_type"),
|
||||
"ai_subtype": row.get("ai_subtype"),
|
||||
"ai_exclude_reason": row.get("ai_exclude_reason"),
|
||||
"ai_exclude_reason_group": row.get("ai_exclude_reason_group"),
|
||||
"participant_role": row.get("participant_role") or row.get("holder_role"),
|
||||
"event_time_seconds": event_time,
|
||||
"region_start_seconds": row.get("region_start_seconds", event_time),
|
||||
@@ -458,14 +459,19 @@ class ReviewStore:
|
||||
selected_opportunity_uid = candidate.get("ai_selected_opportunity_uid") or (
|
||||
candidate.get("llm_suggestion") or {}
|
||||
).get("selected_opportunity_uid")
|
||||
exclude_reason = candidate.get("ai_exclude_reason") or (
|
||||
candidate.get("llm_suggestion") or {}
|
||||
).get("exclude_reason")
|
||||
if verdict not in HUMAN_VERDICT_VALUES:
|
||||
raise ValueError("Accept requires an AI pre-label; use Relabel when no AI label is available")
|
||||
elif action == "exclude":
|
||||
verdict, event_type, subtype = "EXCLUDE", None, None
|
||||
selected_opportunity_uid = None
|
||||
exclude_reason = payload.get("exclude_reason")
|
||||
else:
|
||||
verdict, event_type, subtype = payload.get("human_verdict"), payload.get("human_event_type"), payload.get("human_subtype")
|
||||
selected_opportunity_uid = payload.get("selected_opportunity_uid")
|
||||
exclude_reason = payload.get("exclude_reason")
|
||||
if verdict not in HUMAN_VERDICT_VALUES:
|
||||
raise ValueError(f"Invalid human_verdict: {verdict!r}")
|
||||
if verdict == "POSITIVE" and event_type not in HUMAN_EVENT_TYPE_VALUES:
|
||||
@@ -514,11 +520,12 @@ class ReviewStore:
|
||||
candidate.get("event_time", candidate.get("dense_start_seconds")),
|
||||
),
|
||||
)
|
||||
exclude_reason = payload.get("exclude_reason")
|
||||
if action == "exclude" and exclude_reason not in EXCLUDE_REASON_VALUES:
|
||||
raise ValueError("Exclude requires an exclude_reason")
|
||||
if action != "exclude" and exclude_reason is not None:
|
||||
raise ValueError("exclude_reason is only valid for Exclude")
|
||||
if verdict == "EXCLUDE" and exclude_reason not in EXCLUDE_REASON_VALUES:
|
||||
raise ValueError(
|
||||
"EXCLUDE requires an exclude_reason; legacy AI exclusions must be manually categorized"
|
||||
)
|
||||
if verdict != "EXCLUDE" and exclude_reason is not None:
|
||||
raise ValueError("exclude_reason is only valid for EXCLUDE")
|
||||
correction = payload.get("boundary_correction_seconds")
|
||||
if correction not in {-0.2, 0, 0.2, None}:
|
||||
raise ValueError("boundary_correction_seconds must be -0.2, 0, 0.2, or null")
|
||||
@@ -535,6 +542,7 @@ class ReviewStore:
|
||||
else [selected_opportunity_seconds, behavior_anchor_seconds]
|
||||
),
|
||||
exclude_reason=exclude_reason,
|
||||
exclude_reason_group=exclude_reason_group(exclude_reason),
|
||||
relabeled=action == "relabel",
|
||||
boundary_correction_seconds=correction,
|
||||
original_event_time=candidate.get("event_time", candidate.get("anchor_dense_seconds")),
|
||||
@@ -597,6 +605,8 @@ class ReviewStore:
|
||||
"ai_verdict": candidate.payload.get("ai_verdict"),
|
||||
"ai_event_type": candidate.payload.get("ai_event_type"),
|
||||
"ai_subtype": candidate.payload.get("ai_subtype"),
|
||||
"ai_exclude_reason": candidate.payload.get("ai_exclude_reason"),
|
||||
"ai_exclude_reason_group": candidate.payload.get("ai_exclude_reason_group"),
|
||||
"review": normalized,
|
||||
"updated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"candidate": candidate.payload,
|
||||
@@ -650,9 +660,10 @@ class ReviewStore:
|
||||
"automatic_protocol_label",
|
||||
"automatic_hard_negative_type",
|
||||
"ai_verdict", "ai_event_type", "ai_subtype",
|
||||
"ai_exclude_reason", "ai_exclude_reason_group",
|
||||
"human_action", "human_status", "human_verdict", "human_event_type", "human_subtype",
|
||||
"selected_opportunity_uid", "selected_opportunity_seconds",
|
||||
"valid_prediction_interval_seconds", "exclude_reason", "relabeled",
|
||||
"valid_prediction_interval_seconds", "exclude_reason", "exclude_reason_group", "relabeled",
|
||||
"boundary_correction_seconds", "original_event_time",
|
||||
"confidence",
|
||||
"reviewer",
|
||||
|
||||
@@ -8,7 +8,8 @@ import json
|
||||
import tempfile
|
||||
|
||||
from adjudicate_easycom_eot_int_candidates import (
|
||||
PROMPT_VERSION, SYSTEM_PROMPT, _progress_line, _transport_rows, build_user_prompt, read_candidates,
|
||||
PROMPT_VERSION, SYSTEM_PROMPT, _progress_line, _transport_rows, build_user_prompt,
|
||||
exclude_reason_group, read_candidates,
|
||||
requires_semantic_adjudication, validate_decisions,
|
||||
)
|
||||
|
||||
@@ -32,9 +33,10 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
self.assertIn("Yes.", prompt)
|
||||
|
||||
def test_system_prompt_treats_laughter_as_nonlexical_floor_feedback(self) -> None:
|
||||
self.assertEqual(PROMPT_VERSION, "easycom-wearer-entry-int-prelabel-v12")
|
||||
self.assertEqual(PROMPT_VERSION, "easycom-wearer-boundary-semantic-prelabel-v14")
|
||||
self.assertIn("opportunity_options", SYSTEM_PROMPT)
|
||||
self.assertIn("t_w < t_o < t_y", SYSTEM_PROMPT)
|
||||
self.assertIn("t_w < t_o < t_y never proves takeover", SYSTEM_PROMPT)
|
||||
self.assertIn("Every supplied WEARER_ONSET and WEARER_OFFSET is a raw observed", SYSTEM_PROMPT)
|
||||
self.assertIn("a transcription consisting of [L] denotes laughter-only vocalization", SYSTEM_PROMPT)
|
||||
self.assertIn("ENTRY_WEARER_BACKCHANNEL", SYSTEM_PROMPT)
|
||||
self.assertIn("INT_OTHER_BACKCHANNEL", SYSTEM_PROMPT)
|
||||
@@ -45,11 +47,13 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
self.assertIn('"Nope." beginning 0.15 s after an addressed question ends', SYSTEM_PROMPT)
|
||||
self.assertIn("does not require a preceding non-wearer release", SYSTEM_PROMPT)
|
||||
self.assertIn("duration alone never determines floor function", SYSTEM_PROMPT)
|
||||
self.assertIn("out-of-scope normal completion", SYSTEM_PROMPT)
|
||||
self.assertIn("same incoming non-wearer actually takes", SYSTEM_PROMPT)
|
||||
self.assertIn("normal wearer completion", SYSTEM_PROMPT)
|
||||
self.assertIn("relevant non-wearer actually takes", SYSTEM_PROMPT)
|
||||
self.assertIn("must not retroactively invalidate the preceding INT_YIELD", SYSTEM_PROMPT)
|
||||
self.assertIn("do not infer that a visible smile occurred", SYSTEM_PROMPT)
|
||||
self.assertIn("POSITIVE / INT_YIELD: subtype=null; selected_opportunity_uid=null", SYSTEM_PROMPT)
|
||||
self.assertIn("POSITIVE / INT_YIELD: subtype=null; exclude_reason=null", SYSTEM_PROMPT)
|
||||
self.assertIn("OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION", SYSTEM_PROMPT)
|
||||
self.assertIn("DATA_TIMESTAMP_CORRUPTION", SYSTEM_PROMPT)
|
||||
self.assertNotIn("NOT_APPLICABLE", SYSTEM_PROMPT)
|
||||
self.assertNotIn("UNSURE", SYSTEM_PROMPT)
|
||||
self.assertNotIn('"needs_human_review"', SYSTEM_PROMPT)
|
||||
@@ -78,7 +82,8 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
payload = {
|
||||
"items": [{
|
||||
"uid": "u1", "verdict": "EXCLUDE", "event_type": None,
|
||||
"subtype": None, "confidence": 0.4, "reason": "Ambiguous context.",
|
||||
"subtype": None, "exclude_reason": "INSUFFICIENT_EVIDENCE",
|
||||
"confidence": 0.4, "reason": "Ambiguous context.",
|
||||
"needs_human_review": "true",
|
||||
}]
|
||||
}
|
||||
@@ -128,6 +133,9 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
"verdict": verdict,
|
||||
"event_type": event_type,
|
||||
"subtype": subtype,
|
||||
"exclude_reason": (
|
||||
"INSUFFICIENT_EVIDENCE" if verdict == "EXCLUDE" else None
|
||||
),
|
||||
"selected_opportunity_uid": "copied-release",
|
||||
"confidence": 0.8,
|
||||
"reason": "Evidence-based decision.",
|
||||
@@ -139,6 +147,28 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
)[0]
|
||||
self.assertIsNone(decision["selected_opportunity_uid"])
|
||||
|
||||
def test_exclude_reason_taxonomy_separates_scope_evidence_and_data(self) -> None:
|
||||
self.assertEqual(
|
||||
exclude_reason_group("OUT_OF_SCOPE_NORMAL_COMPLETION"), "OUT_OF_SCOPE"
|
||||
)
|
||||
self.assertEqual(
|
||||
exclude_reason_group("INSUFFICIENT_EVIDENCE"), "INSUFFICIENT_EVIDENCE"
|
||||
)
|
||||
self.assertEqual(
|
||||
exclude_reason_group("DATA_ANNOTATION_ERROR"), "DATA_QUALITY"
|
||||
)
|
||||
base = {
|
||||
"uid": "u1", "verdict": "EXCLUDE", "event_type": None,
|
||||
"subtype": None, "selected_opportunity_uid": None,
|
||||
"confidence": 0.4, "reason": "Not a scored event.",
|
||||
}
|
||||
with self.assertRaisesRegex(ValueError, "valid exclude_reason"):
|
||||
validate_decisions({"items": [base]}, ["u1"])
|
||||
with self.assertRaisesRegex(ValueError, "valid exclude_reason"):
|
||||
validate_decisions(
|
||||
{"items": [{**base, "exclude_reason": "ANNOTATION_ERROR"}]}, ["u1"]
|
||||
)
|
||||
|
||||
def test_event_and_hard_negative_subtypes_match_candidate_family(self) -> None:
|
||||
positive = {
|
||||
"uid": "u1", "verdict": "POSITIVE", "event_type": "INT_YIELD",
|
||||
@@ -195,7 +225,7 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
self.assertEqual([row["uid"] for row in transported], ["item-0000", "item-0001"])
|
||||
self.assertEqual(mapping["item-0001"], rows[1]["uid"])
|
||||
|
||||
def test_old_nonoverlap_wearer_offset_is_not_sent_for_adjudication(self) -> None:
|
||||
def test_every_current_wearer_offset_is_sent_for_adjudication(self) -> None:
|
||||
control = {
|
||||
"candidate_source": "WEARER_OFFSET",
|
||||
"temporal_evidence": {
|
||||
@@ -217,9 +247,9 @@ class EntryINTAdjudicationTests(unittest.TestCase):
|
||||
"active_other_ids_after_offset": [4],
|
||||
},
|
||||
}
|
||||
self.assertFalse(requires_semantic_adjudication(control))
|
||||
self.assertTrue(requires_semantic_adjudication(control))
|
||||
self.assertTrue(requires_semantic_adjudication(crossing))
|
||||
self.assertFalse(requires_semantic_adjudication(speaker_change))
|
||||
self.assertTrue(requires_semantic_adjudication(speaker_change))
|
||||
self.assertFalse(requires_semantic_adjudication({"candidate_source": "WEARER_OFFSET_CONTROL"}))
|
||||
self.assertFalse(requires_semantic_adjudication({"candidate_source": "WEARER_ONSET_CONTROL"}))
|
||||
self.assertFalse(requires_semantic_adjudication({"candidate_source": "WEARER_RESUME"}))
|
||||
|
||||
@@ -106,36 +106,47 @@ class EntryINTReviewCandidateTests(unittest.TestCase):
|
||||
self.assertEqual(onset.task, "wearer_entry")
|
||||
self.assertEqual(onset.automatic_protocol_label, "candidate")
|
||||
self.assertEqual(onset.temporal_evidence["entry_context_timing"], "OTHER_ACTIVE")
|
||||
self.assertFalse(onset.temporal_evidence["preceding_release_paired"])
|
||||
self.assertTrue(onset.temporal_evidence["preceding_release_options_present"])
|
||||
self.assertTrue(
|
||||
any(
|
||||
option["other_onset_between_release_and_onset"]
|
||||
for option in onset.temporal_evidence["opportunity_options"]
|
||||
)
|
||||
)
|
||||
|
||||
def test_short_wearer_pause_without_other_takeover_becomes_resume_control(self) -> None:
|
||||
def test_short_wearer_pause_still_becomes_raw_onset_candidate(self) -> None:
|
||||
rows = [
|
||||
utterance("wearer-1", 2, 0.2, 1.0, "I was saying"),
|
||||
utterance("wearer-2", 2, 1.5, 2.5, "and then this."),
|
||||
]
|
||||
derived = self.derive(rows)
|
||||
resume = next(row for row in derived if row.candidate_source == "WEARER_RESUME")
|
||||
self.assertEqual(resume.task, "wearer_entry_control")
|
||||
self.assertEqual(resume.rule_candidate_type, "WITHIN_TURN_RESUME_CONTROL")
|
||||
self.assertEqual(resume.automatic_protocol_label, "control")
|
||||
self.assertTrue(resume.temporal_evidence["floor_episode_merged"])
|
||||
self.assertAlmostEqual(resume.temporal_evidence["previous_wearer_silence_duration_seconds"], 0.5)
|
||||
self.assertFalse(any(
|
||||
row.candidate_source == "WEARER_ONSET" and row.anchor_dense_seconds == 1.5
|
||||
for row in derived
|
||||
))
|
||||
onset = next(
|
||||
row for row in derived
|
||||
if row.candidate_source == "WEARER_ONSET" and row.anchor_dense_seconds == 1.5
|
||||
)
|
||||
self.assertEqual(onset.task, "wearer_entry")
|
||||
self.assertEqual(onset.rule_candidate_type, "WEARER_ENTRY_CANDIDATE")
|
||||
self.assertEqual(onset.automatic_protocol_label, "candidate")
|
||||
self.assertAlmostEqual(onset.temporal_evidence["previous_wearer_silence_duration_seconds"], 0.5)
|
||||
self.assertFalse(any(row.candidate_source == "WEARER_RESUME" for row in derived))
|
||||
|
||||
def test_short_feedback_does_not_split_wearer_floor_episode(self) -> None:
|
||||
def test_gap_other_activity_is_recorded_without_feedback_semantics(self) -> None:
|
||||
rows = [
|
||||
utterance("wearer-1", 2, 0.2, 1.0, "First clause"),
|
||||
utterance("feedback", 4, 1.1, 1.4, "Yeah."),
|
||||
utterance("wearer-2", 2, 1.6, 2.5, "same turn continues."),
|
||||
]
|
||||
resume = next(row for row in self.derive(rows) if row.candidate_source == "WEARER_RESUME")
|
||||
self.assertEqual(resume.temporal_evidence["feedback_like_other_ids_during_gap"], [4])
|
||||
self.assertFalse(resume.temporal_evidence["sustained_other_takeover_during_gap"])
|
||||
onset = next(
|
||||
row for row in self.derive(rows)
|
||||
if row.candidate_source == "WEARER_ONSET" and row.anchor_dense_seconds == 1.6
|
||||
)
|
||||
gap = onset.temporal_evidence["gap_other_intervals"]
|
||||
self.assertEqual([item["participant_id"] for item in gap], [4])
|
||||
self.assertEqual(gap[0]["transcription"], "Yeah.")
|
||||
self.assertAlmostEqual(gap[0]["gap_overlap_seconds"], 0.3)
|
||||
self.assertNotIn("feedback_like_other_ids_during_gap", onset.temporal_evidence)
|
||||
|
||||
def test_short_gap_with_sustained_other_takeover_preserves_positive_entry_candidate(self) -> None:
|
||||
def test_short_gap_with_other_activity_preserves_unresolved_entry_candidate(self) -> None:
|
||||
rows = [
|
||||
utterance("wearer-1", 2, 0.2, 1.0, "First wearer turn."),
|
||||
utterance("other", 4, 0.8, 2.2, "I have the floor now."),
|
||||
@@ -146,16 +157,17 @@ class EntryINTReviewCandidateTests(unittest.TestCase):
|
||||
row for row in derived
|
||||
if row.candidate_source == "WEARER_ONSET" and row.anchor_dense_seconds == 1.6
|
||||
)
|
||||
self.assertFalse(reentry.temporal_evidence["floor_episode_merged"])
|
||||
self.assertTrue(reentry.temporal_evidence["sustained_other_takeover_during_gap"])
|
||||
self.assertEqual(reentry.temporal_evidence["sustained_competing_other_ids_during_gap"], [4])
|
||||
self.assertEqual(reentry.temporal_evidence["gap_other_participant_ids"], [4])
|
||||
self.assertTrue(
|
||||
reentry.temporal_evidence["gap_other_intervals"][0]["continues_across_wearer_onset"]
|
||||
)
|
||||
self.assertEqual(reentry.temporal_evidence["entry_context_timing"], "OTHER_ACTIVE")
|
||||
self.assertEqual(reentry.automatic_protocol_label, "candidate")
|
||||
preceding_yield = next(
|
||||
row for row in derived
|
||||
if row.candidate_source == "WEARER_OFFSET" and row.anchor_dense_seconds == 1.0
|
||||
)
|
||||
self.assertTrue(preceding_yield.temporal_evidence["int_ordering_satisfied"])
|
||||
self.assertTrue(preceding_yield.temporal_evidence["new_other_onset_during_wearer"])
|
||||
|
||||
def test_same_other_resume_is_entry_midturn_hard_negative(self) -> None:
|
||||
rows = [
|
||||
@@ -199,32 +211,43 @@ class EntryINTReviewCandidateTests(unittest.TestCase):
|
||||
self.assertEqual(event.anchor_type, "wearer_offset")
|
||||
self.assertAlmostEqual(event.event_time, 2.0)
|
||||
self.assertEqual(event.temporal_evidence["tolerance_crossing_other_ids"], [4])
|
||||
self.assertEqual(event.temporal_evidence["eligible_new_other_onset_ids"], [4])
|
||||
self.assertEqual(event.temporal_evidence["new_other_onset_ids_during_wearer"], [4])
|
||||
self.assertAlmostEqual(
|
||||
event.temporal_evidence["eligible_new_other_onsets"][0]["onset_seconds"], 1.0
|
||||
event.temporal_evidence["new_other_onsets_during_wearer"][0]["onset_seconds"], 1.0
|
||||
)
|
||||
self.assertTrue(event.temporal_evidence["int_ordering_satisfied"])
|
||||
self.assertTrue(event.temporal_evidence["new_other_onset_during_wearer"])
|
||||
self.assertTrue(event.temporal_evidence["new_other_onset_crosses_offset"])
|
||||
self.assertEqual(event.temporal_evidence["next_speaker_ids"], [4])
|
||||
self.assertEqual(event.temporal_evidence["next_speaker_basis"], "active_across_offset")
|
||||
nearby = event.temporal_evidence["nearby_other_intervals"][0]
|
||||
self.assertAlmostEqual(nearby["overlap_with_wearer_before_offset_seconds"], 1.0)
|
||||
|
||||
def test_preexisting_other_crossing_offset_is_not_an_int_candidate(self) -> None:
|
||||
def test_preexisting_other_crossing_offset_remains_an_int_candidate(self) -> None:
|
||||
rows = [
|
||||
utterance("other", 4, 0.0, 3.0, "Other already holds the floor."),
|
||||
utterance("wearer", 2, 1.0, 2.0, "Wearer enters and stops."),
|
||||
]
|
||||
event = next(row for row in self.derive(rows) if row.anchor_type == "wearer_offset")
|
||||
self.assertEqual(event.candidate_source, "WEARER_OFFSET_CONTROL")
|
||||
self.assertEqual(event.temporal_evidence["preexisting_crossing_other_ids"], [4])
|
||||
self.assertFalse(event.temporal_evidence["int_ordering_satisfied"])
|
||||
self.assertEqual(event.candidate_source, "WEARER_OFFSET")
|
||||
self.assertEqual(event.task, "int_yield")
|
||||
self.assertEqual(
|
||||
event.temporal_evidence["preexisting_other_ids_at_wearer_activity_run_start"], [4]
|
||||
)
|
||||
self.assertFalse(event.temporal_evidence["new_other_onset_during_wearer"])
|
||||
self.assertTrue(event.temporal_evidence["other_crosses_wearer_offset"])
|
||||
|
||||
def test_other_starting_after_wearer_offset_is_a_separate_control(self) -> None:
|
||||
def test_other_starting_after_wearer_offset_remains_an_int_candidate(self) -> None:
|
||||
rows = [
|
||||
utterance("wearer", 2, 0.0, 2.0, "Wearer finishes."),
|
||||
utterance("other", 4, 2.2, 3.0, "Later turn."),
|
||||
]
|
||||
event = next(row for row in self.derive(rows) if row.anchor_type == "wearer_offset")
|
||||
self.assertEqual(event.candidate_source, "WEARER_OFFSET_CONTROL")
|
||||
self.assertEqual(event.rule_candidate_type, "WEARER_OFFSET_CONTROL")
|
||||
self.assertEqual(event.automatic_protocol_label, "control")
|
||||
self.assertEqual(event.candidate_source, "WEARER_OFFSET")
|
||||
self.assertEqual(event.rule_candidate_type, "YIELD_CANDIDATE")
|
||||
self.assertEqual(event.automatic_protocol_label, "candidate")
|
||||
self.assertFalse(event.temporal_evidence["other_crosses_wearer_offset"])
|
||||
self.assertEqual(event.temporal_evidence["next_speaker_ids"], [4])
|
||||
self.assertAlmostEqual(event.temporal_evidence["next_speech_onset_seconds"], 2.2)
|
||||
|
||||
def test_one_frame_boundary_tolerance_is_configurable(self) -> None:
|
||||
rows = [
|
||||
@@ -241,9 +264,11 @@ class EntryINTReviewCandidateTests(unittest.TestCase):
|
||||
tolerant_offset = next(row for row in tolerant if row.anchor_type == "wearer_offset")
|
||||
strict_offset = next(row for row in strict if row.anchor_type == "wearer_offset")
|
||||
self.assertEqual(tolerant_offset.candidate_source, "WEARER_OFFSET")
|
||||
self.assertEqual(strict_offset.candidate_source, "WEARER_OFFSET_CONTROL")
|
||||
self.assertEqual(strict_offset.candidate_source, "WEARER_OFFSET")
|
||||
self.assertTrue(tolerant_offset.temporal_evidence["other_crosses_wearer_offset"])
|
||||
self.assertFalse(strict_offset.temporal_evidence["other_crosses_wearer_offset"])
|
||||
|
||||
def test_internal_raw_offset_is_not_an_int_candidate(self) -> None:
|
||||
def test_internal_raw_offset_remains_an_int_candidate(self) -> None:
|
||||
rows = [
|
||||
utterance("wearer-1", 2, 0.0, 1.0, "First clause."),
|
||||
utterance("other", 4, 0.8, 1.3, "Yeah."),
|
||||
@@ -253,8 +278,8 @@ class EntryINTReviewCandidateTests(unittest.TestCase):
|
||||
row for row in self.derive(rows)
|
||||
if row.anchor_type == "wearer_offset" and row.anchor_dense_seconds == 1.0
|
||||
)
|
||||
self.assertEqual(event.candidate_source, "WEARER_OFFSET_CONTROL")
|
||||
self.assertTrue(event.temporal_evidence["raw_offset_is_within_merged_floor_episode"])
|
||||
self.assertEqual(event.candidate_source, "WEARER_OFFSET")
|
||||
self.assertEqual(event.automatic_protocol_label, "candidate")
|
||||
|
||||
def test_brief_feedback_is_int_backchannel_hard_negative(self) -> None:
|
||||
rows = [
|
||||
|
||||
@@ -147,7 +147,7 @@ class FloorEventReviewTests(unittest.TestCase):
|
||||
[candidate.resolve()],
|
||||
)
|
||||
|
||||
def test_primary_queue_skips_legacy_nonoverlap_wearer_offset(self) -> None:
|
||||
def test_primary_queue_keeps_nonoverlap_wearer_offset_for_semantics(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
path = Path(temporary) / "legacy.jsonl"
|
||||
rows = [
|
||||
@@ -171,7 +171,7 @@ class FloorEventReviewTests(unittest.TestCase):
|
||||
)
|
||||
primary = load_candidates([path])
|
||||
all_rows = load_candidates([path], include_offset_controls=True)
|
||||
self.assertEqual([row.uid for row in primary], [rows[1]["uid"]])
|
||||
self.assertEqual([row.uid for row in primary], [row["uid"] for row in rows])
|
||||
self.assertEqual(len(all_rows), 2)
|
||||
|
||||
def test_relabel_preserves_ai_and_records_human_decision(self) -> None:
|
||||
@@ -259,6 +259,37 @@ class FloorEventReviewTests(unittest.TestCase):
|
||||
self.assertEqual(
|
||||
record["review"]["exclude_reason"], "OUT_OF_SCOPE_NORMAL_COMPLETION"
|
||||
)
|
||||
self.assertEqual(record["review"]["exclude_reason_group"], "OUT_OF_SCOPE")
|
||||
|
||||
def test_accept_ai_exclude_preserves_reason_taxonomy(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
_, events = self._fixture(root)
|
||||
candidates = load_candidates([events])
|
||||
candidate = candidates[0]
|
||||
candidate.payload.update(
|
||||
ai_verdict="EXCLUDE",
|
||||
ai_event_type=None,
|
||||
ai_subtype=None,
|
||||
ai_exclude_reason="DATA_MISSING_CONTEXT",
|
||||
ai_exclude_reason_group="DATA_QUALITY",
|
||||
llm_suggestion={
|
||||
"verdict": "EXCLUDE",
|
||||
"event_type": None,
|
||||
"subtype": None,
|
||||
"exclude_reason": "DATA_MISSING_CONTEXT",
|
||||
},
|
||||
)
|
||||
store = ReviewStore(root / "reviews.jsonl", candidates)
|
||||
record = store.save(candidate.candidate_id, {
|
||||
"human_action": "accept", "human_verdict": None,
|
||||
"human_event_type": None, "human_subtype": None,
|
||||
"selected_opportunity_uid": None, "exclude_reason": None,
|
||||
"boundary_correction_seconds": None, "confidence": "medium",
|
||||
"notes": "context is truncated",
|
||||
})
|
||||
self.assertEqual(record["review"]["exclude_reason"], "DATA_MISSING_CONTEXT")
|
||||
self.assertEqual(record["review"]["exclude_reason_group"], "DATA_QUALITY")
|
||||
|
||||
def test_entry_review_may_select_only_a_supplied_opportunity_anchor(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
|
||||
Reference in New Issue
Block a user