wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revise wearer entry and yield annotation protocol

Browse Source
This commit is contained in:
Xin Wang
2026-09-15 01:25:39 +08:00
parent 026d89978f
commit 71dfd8808a
12 changed files with 478 additions and 248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
adjudicate_easycom_eot_int_candidates.py
+72 -35
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""Add conservative DeepSeek pre-labels to wearer-centric EOT/INT candidates.
"""Add conservative DeepSeek pre-labels to wearer entry/INT candidates.
The model receives the candidate source, structured participant timing and the
transcript excerpt. It cannot change any source timestamp. Its output is
@@ -26,15 +26,15 @@ import urllib.request
from prepare_easycom_eot_int_review import requires_semantic_adjudication
PROMPT_VERSION = "easycom-wearer-anchor-first-eot-int-prelabel-v7"
PROMPT_VERSION = "easycom-wearer-entry-int-prelabel-v12"
DEFAULT_MODEL = "deepseek-v4-flash"
DEFAULT_BASE_URL = "https://api.deepseek.com"
VERDICTS = {"POSITIVE", "HARD_NEGATIVE", "NOT_APPLICABLE", "EXCLUDE", "UNSURE"}
EVENT_TYPES = {"EOT_RESPONSE", "INT_YIELD"}
VERDICTS = {"POSITIVE", "HARD_NEGATIVE", "EXCLUDE"}
EVENT_TYPES = {"WEARER_ENTRY", "INT_YIELD"}
HARD_NEGATIVE_TYPES = {
"EOT_MIDTURN_PAUSE",
"EOT_OTHER_TO_OTHER",
"EOT_WEARER_BACKCHANNEL",
"ENTRY_OTHER_MIDTURN_PAUSE",
"ENTRY_OTHER_TO_OTHER",
"ENTRY_WEARER_BACKCHANNEL",
"INT_OTHER_BACKCHANNEL",
"INT_NON_TAKEOVER_OVERLAP",
"INT_WEARER_HOLDS_FLOOR",
@@ -89,31 +89,47 @@ class _ProgressBar:
SYSTEM_PROMPT = """You conservatively pre-label candidate regions for a scientific study of wearer-relative conversational floor control.
Evidence rules:
1. Use candidate source, structured participant timing, per-speaker activity, and transcript together. Timing evidence has priority over a speculative transcript interpretation.
1. Reconstruct the event chronologically before assigning semantics. Use exact start/end timestamps to identify (a) the proposed release or competing onset, (b) the wearer behavior anchor, (c) which same participant continues across the anchor, and (d) who produces the next substantive turn. Then use transcript content and Target_of_Speech, when supplied, to decide conversational function. Timing has priority over speculative transcript interpretation, but duration alone never determines floor function.
2. Never alter or propose an event timestamp. Do not infer prosody, gaze, intention, or acoustic cues that were not supplied.
3. A wearer onset is not automatically a floor take; a wearer offset is not automatically a yield; an other onset is not automatically a takeover. A pause is not automatically EOT, overlap is not automatically interruption, and words such as "yeah" are not automatically backchannels. Check who speaks sustainably afterward and who resumes.
4. Use UNSURE when the evidence does not establish conversational function. Never force a label for coverage.
5. Every supplied WEARER_ONSET was found by starting from an observed floor-level wearer entry and looking backward. Its immutable behavior anchor is fixed. Select the true preceding release only from temporal_evidence.opportunity_options; never invent a trigger or timestamp. A POSITIVE EOT_RESPONSE must return that option's uid as selected_opportunity_uid. Do not reject an option solely because its response delay is long. If none is a defensible trigger, return another verdict and null.
6. Every supplied WEARER_OFFSET has already passed both deterministic INT prerequisites: a new non-wearer onset occurs strictly inside the current wearer floor episode (t_w < t_o < t_y), and that same non-wearer interval crosses the wearer yield boundary within the recorded annotation tolerance. Decide only whether it is a genuine takeover or a semantic hard negative. Pre-existing other activity, post-completion other onset, within-turn wearer pauses, and completion-to-silence are controls outside this queue.
7. Transcript event markers are evidence, not lexical turns. In EasyCom, a transcription consisting of [L] denotes laughter-only vocalization. Do not treat [L] as a sustained floor take, an EOT response, or an interruption takeover merely because its activity interval creates an onset, offset, or overlap candidate. Specifically:
- wearer [L] during a non-wearer's retained turn is normally HARD_NEGATIVE / EOT_WEARER_BACKCHANNEL, not EOT_RESPONSE;
3. A raw wearer onset is not automatically a floor entry; a wearer offset is not automatically a yield; an other onset is not automatically a takeover. A pause is not automatically a release, overlap is not automatically interruption, and words such as "yeah" are not automatically backchannels. Determine what the utterance does in context, not only how long it lasts.
4. EXCLUDE is the sole non-scoring outcome. Use it both for insufficient/unresolvable evidence and for clear out-of-scope non-events that should not enter evaluation, including a normal wearer completion that merely coincides with laughter or feedback. Never force such a region into HARD_NEGATIVE for coverage.
5. Every supplied WEARER_ONSET is an observed floor-level onset after deterministic within-turn pause merging. Decide whether the wearer genuinely enters or re-enters the conversational floor. A POSITIVE WEARER_ENTRY does not require a preceding non-wearer release: it may occur after a release, from silence, or while another speaker is active. temporal_evidence.opportunity_options and entry_context_timing are context metadata only. If one supplied release clearly prompted the entry, selected_opportunity_uid may contain its uid; otherwise return null. Never invent a trigger or timestamp. A direct, semantically complete answer can be a genuine entry even when short. Do not call it a backchannel merely because it lasts less than one second or because another participant speaks later. For example, "Nope." beginning 0.15 s after an addressed question ends is normally WEARER_ENTRY, not ENTRY_WEARER_BACKCHANNEL. Later conversation does not retroactively cancel a completed entry.
6. Every supplied WEARER_OFFSET has already passed both deterministic INT prerequisites: a new non-wearer onset occurs strictly inside the current wearer floor episode (t_w < t_o < t_y), and that same non-wearer interval crosses the wearer yield boundary within the recorded annotation tolerance. This is only a retrieval gate. Verify that the same incoming non-wearer actually takes and sustains the next floor after the wearer offset. A different participant's later substantive turn is not evidence that the incoming speaker took over. Do not label INT_WEARER_HOLDS_FLOOR merely because the wearer speaks again later: if the competing other establishes an intervening substantive floor and the wearer has an intervening silent interval, the later wearer onset is a new WEARER_ENTRY and does not invalidate the earlier INT_YIELD. If the wearer reaches a semantically complete natural ending while the overlapping contribution is only [L] or feedback, and that contributor does not take the next substantive floor, return EXCLUDE: out-of-scope normal completion, not INT_YIELD and not a scored hard negative. Pre-existing other activity, post-completion other onset, within-turn wearer pauses, and completion-to-silence are controls outside this queue.
7. Transcript event markers are evidence, not lexical turns. In EasyCom, a transcription consisting of [L] denotes laughter-only vocalization. Do not treat [L] as a sustained floor entry or interruption takeover merely because its activity interval creates an onset, offset, or overlap candidate. Specifically:
- wearer [L] during a non-wearer's retained turn is normally HARD_NEGATIVE / ENTRY_WEARER_BACKCHANNEL, not WEARER_ENTRY;
- non-wearer [L] overlapping a wearer's retained turn is normally HARD_NEGATIVE / INT_OTHER_BACKCHANNEL, not INT_YIELD;
- wearer [L] ending while a non-wearer continues does not establish that the wearer held and yielded the floor.
Use UNSURE if the surrounding timing does not establish whose floor was retained. Do not EXCLUDE a valid laughter-only annotation merely because it is non-lexical, and do not infer that a visible smile occurred: [L] supplies laughter evidence only. Do not assign semantics to other bracketed markers unless the supplied context establishes them.
Use EXCLUDE if the surrounding timing does not establish whose floor was retained. Do not EXCLUDE a valid laughter-only annotation merely because it is non-lexical, and do not infer that a visible smile occurred: [L] supplies laughter evidence only. Do not assign semantics to other bracketed markers unless the supplied context establishes them.
Top-level verdicts:
- POSITIVE: the system should commit.
- HARD_NEGATIVE: a valid, interpretable region where the system should not commit.
- NOT_APPLICABLE: a real speech transition outside the committed EOT/INT task, such as a wearer starting after long silence or naturally ending into silence.
- EXCLUDE: the region cannot be scored reliably because of annotation corruption, a recording boundary, missing context, a non-speech artifact, timestamp corruption, or another unresolvable defect.
- UNSURE: evidence is insufficient for a reliable automatic pre-label and needs human judgment.
- POSITIVE: a target WEARER_ENTRY or INT_YIELD occurs and the system should commit.
- HARD_NEGATIVE: a deliberately retained, valid false-positive opportunity matching one of the behavioral counterexamples below; it enters evaluation as a negative.
- EXCLUDE: a non-scoring region. This includes clear out-of-scope/non-target opportunities such as natural wearer completion coincident with feedback, as well as insufficient evidence, annotation corruption, a recording boundary, missing context, a non-speech artifact, timestamp corruption, or another unresolvable defect.
POSITIVE event_type must be exactly EOT_RESPONSE or INT_YIELD. EOT_RESPONSE means a non-wearer genuinely releases the floor and the wearer then takes it with a sustained response. INT_YIELD means a non-wearer genuinely takes over during wearer speech and the wearer relinquishes the floor.
POSITIVE event_type must be exactly WEARER_ENTRY or INT_YIELD. WEARER_ENTRY means the wearer genuinely enters or re-enters the floor with a contextually complete response or substantive contribution; no prior release is required and the contribution need not be long. INT_YIELD means a non-wearer genuinely takes over during wearer speech and the wearer relinquishes the floor. A later WEARER_ENTRY after an intervening other floor is a new event and must not retroactively invalidate the preceding INT_YIELD.
HARD_NEGATIVE subtype must be exactly one of EOT_MIDTURN_PAUSE, EOT_OTHER_TO_OTHER, EOT_WEARER_BACKCHANNEL, INT_OTHER_BACKCHANNEL, INT_NON_TAKEOVER_OVERLAP, INT_WEARER_HOLDS_FLOOR, or OTHER_HARD_NEGATIVE.
HARD_NEGATIVE subtypes are behavioral counterexamples, not data-quality labels:
- ENTRY_OTHER_MIDTURN_PAUSE: a non-wearer merely pauses inside that same speaker's continuing floor episode; the same non-wearer resumes or clearly retains the turn, while the wearer does not enter.
- ENTRY_OTHER_TO_OTHER: one non-wearer releases and a different non-wearer takes the next floor; the wearer does not enter.
- ENTRY_WEARER_BACKCHANNEL: the wearer vocalizes, but only as brief feedback, acknowledgement, laughter, or another non-floor-taking response; the non-wearer retains or resumes the floor.
- INT_OTHER_BACKCHANNEL: the competing non-wearer contribution during wearer speech is brief feedback, acknowledgement, laughter, or another backchannel, and the wearer clearly continues or resumes the same floor episode. If the wearer instead naturally completes at the overlap boundary and the feedback speaker does not take the next substantive floor, use EXCLUDE.
- INT_WEARER_HOLDS_FLOOR: the competing contribution is not best characterized as a backchannel, but the wearer demonstrably continues or promptly resumes the same floor episode after the overlap, so no yield occurs.
- INT_NON_TAKEOVER_OVERLAP: the competing contribution is not merely a backchannel, yet it still fails to establish a sustained takeover (for example, a collision, abandoned attempt, or overlap that dies out); use this only when wearer continuation is not the clearer defining evidence for INT_WEARER_HOLDS_FLOOR.
- OTHER_HARD_NEGATIVE: a valid, interpretable false-positive opportunity in the correct ENTRY or INT family that does not fit any named subtype. State the unmatched behavior explicitly. This is not an uncertainty fallback; use EXCLUDE when the evidence is insufficient or unresolvable.
Subtype precedence when descriptions overlap:
1. Use the family matching the candidate task: ENTRY candidates take only ENTRY_* subtypes and INT candidates take only INT_* subtypes.
2. For ENTRY, prefer ENTRY_WEARER_BACKCHANNEL when an actual wearer vocalization is feedback rather than a floor entry; otherwise distinguish same-other resumption (ENTRY_OTHER_MIDTURN_PAUSE) from a different-other takeover (ENTRY_OTHER_TO_OTHER).
3. For INT, use INT_OTHER_BACKCHANNEL when the other contribution is feedback and wearer continuation/resumption makes it a scored false-positive opportunity. If feedback merely coincides with a natural wearer completion, use EXCLUDE. Otherwise use INT_WEARER_HOLDS_FLOOR when wearer continuation/resumption clearly demonstrates retained floor; use INT_NON_TAKEOVER_OVERLAP for the remaining interpretable failed-takeover overlaps.
4. Use OTHER_HARD_NEGATIVE only after the named subtypes have been ruled out. Never use it merely because confidence is low.
Return exactly one JSON object with this shape and no markdown:
{"items":[{"uid":"...","verdict":"UNSURE","event_type":null,"subtype":null,"selected_opportunity_uid":null,"confidence":0.25,"reason":"brief evidence-based sentence"}]}
{"items":[{"uid":"...","verdict":"EXCLUDE","event_type":null,"subtype":null,"selected_opportunity_uid":null,"confidence":0.25,"reason":"brief evidence-based sentence"}]}
Field combinations are strict:
- POSITIVE / WEARER_ENTRY: subtype=null; selected_opportunity_uid is one supplied opportunity uid or null.
- POSITIVE / INT_YIELD: subtype=null; selected_opportunity_uid=null.
- HARD_NEGATIVE: event_type=null; selected_opportunity_uid=null; subtype is one valid subtype from the candidate's ENTRY or INT family.
- EXCLUDE: event_type=null; subtype=null; selected_opportunity_uid=null.
Return every supplied uid exactly once. Copy uid byte-for-byte and never add a backslash before an underscore. confidence must be a number from 0 to 1."""
@@ -191,7 +207,7 @@ def build_user_prompt(rows: Sequence[dict[str, Any]], pass_index: int) -> str:
instruction = (
"Primary pass: classify conservatively."
if pass_index == 0
else "Independent skeptical pass: seek alternative explanations and choose uncertain when semantics are unsupported."
else "Independent skeptical pass: seek alternative explanations and choose EXCLUDE when semantics are unsupported."
)
return (
instruction
@@ -215,6 +231,7 @@ def validate_decisions(
payload: Any,
expected_uids: Sequence[str],
allowed_opportunities: dict[str, set[str]] | None = None,
expected_families: dict[str, str] | None = None,
) -> list[dict[str, Any]]:
if not isinstance(payload, dict) or not isinstance(payload.get("items"), list):
raise ValueError("Response must be an object with an items list")
@@ -236,21 +253,31 @@ def validate_decisions(
if verdict not in VERDICTS:
raise ValueError(f"Invalid verdict for {uid}: {verdict!r}")
if verdict == "POSITIVE" and event_type not in EVENT_TYPES:
raise ValueError(f"POSITIVE requires EOT_RESPONSE or INT_YIELD for {uid}")
raise ValueError(f"POSITIVE requires WEARER_ENTRY or INT_YIELD for {uid}")
if verdict != "POSITIVE" and event_type is not None:
raise ValueError(f"Only POSITIVE may have event_type for {uid}")
if verdict == "HARD_NEGATIVE" and subtype not in HARD_NEGATIVE_TYPES:
raise ValueError(f"HARD_NEGATIVE requires a valid subtype for {uid}")
if verdict != "HARD_NEGATIVE" and subtype is not None:
raise ValueError(f"Only HARD_NEGATIVE may have subtype for {uid}")
family = None if expected_families is None else expected_families.get(str(uid))
if verdict == "POSITIVE" and family in {"ENTRY", "INT"}:
required_event_type = "WEARER_ENTRY" if family == "ENTRY" else "INT_YIELD"
if event_type != required_event_type:
raise ValueError(f"{family} candidate requires {required_event_type} for {uid}")
if verdict == "HARD_NEGATIVE" and family in {"ENTRY", "INT"}:
if subtype != "OTHER_HARD_NEGATIVE" and not str(subtype).startswith(f"{family}_"):
raise ValueError(f"{family} candidate requires an {family}_* hard-negative subtype for {uid}")
allowed = set() if allowed_opportunities is None else allowed_opportunities.get(str(uid), set())
if verdict == "POSITIVE" and event_type == "EOT_RESPONSE" and allowed:
if selected_opportunity_uid not in allowed:
raise ValueError(
f"EOT_RESPONSE must select a supplied opportunity uid for {uid}"
)
if verdict != "POSITIVE" or event_type != "WEARER_ENTRY":
# This field is auxiliary attribution metadata, not part of the
# semantic label. DeepSeek occasionally copies an opportunity uid
# into an INT/HN/EXCLUDE result despite the output contract. Drop
# that irrelevant value instead of retrying and losing the batch.
selected_opportunity_uid = None
elif selected_opportunity_uid is not None:
raise ValueError(f"selected_opportunity_uid is only valid for EOT_RESPONSE: {uid}")
if selected_opportunity_uid not in allowed:
raise ValueError(f"WEARER_ENTRY may select only a supplied opportunity uid for {uid}")
if isinstance(confidence, bool) or not isinstance(confidence, (int, float)) or not 0 <= confidence <= 1:
raise ValueError(f"confidence must be in [0, 1] for {uid}")
if not isinstance(reason, str) or not reason.strip():
@@ -310,6 +337,7 @@ def _post(
content = envelope["choices"][0]["message"]["content"]
parsed = json.loads(content)
allowed_opportunities = {}
expected_families = {}
for row in transported:
evidence = row.get("temporal_evidence") or {}
options = evidence.get("opportunity_options") or []
@@ -320,8 +348,13 @@ def _post(
}
if not allowed_opportunities[row["uid"]] and evidence.get("opportunity_source_uid"):
allowed_opportunities[row["uid"]].add(str(evidence["opportunity_source_uid"]))
task = str(row.get("task", "")).lower()
if task.startswith("wearer_entry") or task.startswith("eot"):
expected_families[row["uid"]] = "ENTRY"
elif task.startswith("int"):
expected_families[row["uid"]] = "INT"
decisions = validate_decisions(
parsed, list(original_by_transport), allowed_opportunities
parsed, list(original_by_transport), allowed_opportunities, expected_families
)
for decision in decisions:
decision["uid"] = original_by_transport[decision["uid"]]
@@ -404,17 +437,21 @@ def add_suggestions(
for source in rows:
decisions = by_uid[source["uid"]]
labels = {
(d["verdict"], d["event_type"], d["subtype"], d["selected_opportunity_uid"])
(d["verdict"], d["event_type"], d["subtype"])
for d in decisions
}
unanimous = len(labels) == 1
if unanimous:
verdict, event_type, subtype, selected_opportunity_uid = next(iter(labels))
verdict, event_type, subtype = next(iter(labels))
selected_options = {d["selected_opportunity_uid"] for d in decisions}
selected_opportunity_uid = (
next(iter(selected_options)) if len(selected_options) == 1 else None
)
confidence = min(float(item["confidence"]) for item in decisions)
reason = " | ".join(item["reason"] for item in decisions)
needs_human_review = any(item["needs_human_review"] for item in decisions)
else:
verdict, event_type, subtype, selected_opportunity_uid = "UNSURE", None, None, None
verdict, event_type, subtype, selected_opportunity_uid = "EXCLUDE", None, None, None
confidence = min(float(item["confidence"]) for item in decisions)
reason = "The independent DeepSeek passes disagreed."
needs_human_review = True