Clarify EasyCom boundary prelabel taxonomy

2026-09-15 11:41:13 +08:00
parent e2c305facc
commit bb8963b80d
16 changed files with 782 additions and 5 deletions
@@ -7,6 +7,10 @@ Current artifact:

 - [`egospeak_easycom_feature_audit.json`](egospeak_easycom_feature_audit.json):
  released EgoSpeak EasyCom feature-layout audit.
+- [`easycom_event_ai_prelabel_stats_v15/`](easycom_event_ai_prelabel_stats_v15/):
+  descriptive DeepSeek v15 pre-label counts for EasyCom Sessions 1--4. These
+  rows are all unreviewed and must not replace the manuscript's
+  human-adjudicated annotation statistics.

 When adding a report, record the generating script, command, source dataset
 version and timestamp in the payload or an adjacent README. Derived timing or
@@ -0,0 +1,32 @@
+# EasyCom event AI pre-label audit
+
+> These are DeepSeek pre-labels awaiting mandatory human adjudication, not ground truth and not final manuscript annotation counts.
+
+Inputs contain 2,128 unique candidates from Session_1, Session_2, Session_3, Session_4. Protocol: `easycom-wearer-entry-int-candidates-v8`; prompt: `easycom-wearer-boundary-semantic-prelabel-v15`.
+
+## Cohort overview
+
+| Cohort | Candidates | Positive | Hard negative | Exclude | Prelabel scoring-eligible | Two-pass semantic/full agreement | Human reviewed |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Session_1 | 496 | 116 | 182 | 198 | 298 (60.08%) | 458/496 (92.34%) / 453/496 (91.33%) | 0 |
+| Session_2 | 490 | 71 | 252 | 167 | 323 (65.92%) | 454/490 (92.65%) / 452/490 (92.24%) | 0 |
+| Session_3 | 545 | 117 | 184 | 244 | 301 (55.23%) | 489/545 (89.72%) / 486/545 (89.17%) | 0 |
+| Session_4 | 597 | 84 | 381 | 132 | 465 (77.89%) | 575/597 (96.31%) / 571/597 (95.64%) | 0 |
+| Test pooled (Sessions 1-3) | 1531 | 304 | 618 | 609 | 922 (60.22%) | 1401/1531 (91.51%) / 1391/1531 (90.86%) | 0 |
+| Session 4 audit (train split) | 597 | 84 | 381 | 132 | 465 (77.89%) | 575/597 (96.31%) / 571/597 (95.64%) | 0 |
+| All supplied (descriptive only) | 2128 | 388 | 999 | 741 | 1387 (65.18%) | 1976/2128 (92.86%) / 1962/2128 (92.20%) | 0 |
+
+The official EasyCom test sessions (1--3) are pooled only for descriptive reporting; these AI pre-labels must not be used to tune thresholds, taxonomy, prompts, or model choices. Session 4 is shown separately as a train-split audit and is not the complete Sessions 4--12 training corpus.
+
+Detailed counts are in `taxonomy_counts.csv` and `candidate_source_counts.csv`. `manuscript_ai_prelabel_table.tex` mirrors the manuscript row order but is deliberately labeled as an AI audit; retain the manuscript TODOs until human adjudication is frozen.
+
+## Reproduce
+
+```bash
+python3 summarize_easycom_event_prelabels.py \
+  --input reports/easycom_entry_int_session1_v14/easycom_eot_int_deepseek_official.jsonl \
+  --input reports/easycom_entry_int_session2_v14/easycom_eot_int_deepseek_official.jsonl \
+  --input reports/easycom_entry_int_session3_v14/easycom_eot_int_deepseek_official.jsonl \
+  --input reports/easycom_entry_int_session4_v14/easycom_eot_int_deepseek_official.jsonl \
+  --output-dir reports/easycom_event_ai_prelabel_stats_v15
+```
@@ -0,0 +1,43 @@
+cohort,candidate_source,count,percent_of_all_candidates
+Session_1,NON_TAKEOVER_OVERLAP,21,4.23
+Session_1,OTHER_BACKCHANNEL,20,4.03
+Session_1,OTHER_TO_OTHER,56,11.29
+Session_1,SAME_SPEAKER_PAUSE,60,12.1
+Session_1,WEARER_OFFSET,170,34.27
+Session_1,WEARER_ONSET,169,34.07
+Session_2,NON_TAKEOVER_OVERLAP,14,2.86
+Session_2,OTHER_BACKCHANNEL,15,3.06
+Session_2,OTHER_TO_OTHER,123,25.1
+Session_2,SAME_SPEAKER_PAUSE,84,17.14
+Session_2,WEARER_OFFSET,127,25.92
+Session_2,WEARER_ONSET,127,25.92
+Session_3,NON_TAKEOVER_OVERLAP,22,4.04
+Session_3,OTHER_BACKCHANNEL,19,3.49
+Session_3,OTHER_TO_OTHER,49,8.99
+Session_3,SAME_SPEAKER_PAUSE,63,11.56
+Session_3,WEARER_OFFSET,196,35.96
+Session_3,WEARER_ONSET,196,35.96
+Session_4,NON_TAKEOVER_OVERLAP,12,2.01
+Session_4,OTHER_BACKCHANNEL,3,0.5
+Session_4,OTHER_TO_OTHER,255,42.71
+Session_4,SAME_SPEAKER_PAUSE,105,17.59
+Session_4,WEARER_OFFSET,111,18.59
+Session_4,WEARER_ONSET,111,18.59
+Test pooled (Sessions 1-3),NON_TAKEOVER_OVERLAP,57,3.72
+Test pooled (Sessions 1-3),OTHER_BACKCHANNEL,54,3.53
+Test pooled (Sessions 1-3),OTHER_TO_OTHER,228,14.89
+Test pooled (Sessions 1-3),SAME_SPEAKER_PAUSE,207,13.52
+Test pooled (Sessions 1-3),WEARER_OFFSET,493,32.2
+Test pooled (Sessions 1-3),WEARER_ONSET,492,32.14
+Session 4 audit (train split),NON_TAKEOVER_OVERLAP,12,2.01
+Session 4 audit (train split),OTHER_BACKCHANNEL,3,0.5
+Session 4 audit (train split),OTHER_TO_OTHER,255,42.71
+Session 4 audit (train split),SAME_SPEAKER_PAUSE,105,17.59
+Session 4 audit (train split),WEARER_OFFSET,111,18.59
+Session 4 audit (train split),WEARER_ONSET,111,18.59
+All supplied (descriptive only),NON_TAKEOVER_OVERLAP,69,3.24
+All supplied (descriptive only),OTHER_BACKCHANNEL,57,2.68
+All supplied (descriptive only),OTHER_TO_OTHER,483,22.7
+All supplied (descriptive only),SAME_SPEAKER_PAUSE,312,14.66
+All supplied (descriptive only),WEARER_OFFSET,604,28.38
+All supplied (descriptive only),WEARER_ONSET,603,28.34
@@ -0,0 +1,8 @@
+cohort,sessions,num_candidates,num_entry_family,num_int_family,num_positive,num_hard_negative,num_exclude,num_scoring_eligible_prelabels,scoring_eligible_prelabel_percent,mean_ai_confidence,median_ai_confidence,num_rows_with_two_passes,num_two_pass_semantic_agreement,two_pass_semantic_agreement_percent,num_two_pass_full_decision_agreement,two_pass_full_decision_agreement_percent,num_human_reviewed
+Session_1,['Session_1'],496,285,211,116,182,198,298,60.08,0.7012,0.7,496,458,92.34,453,91.33,0
+Session_2,['Session_2'],490,334,156,71,252,167,323,65.92,0.7143,0.7,490,454,92.65,452,92.24,0
+Session_3,['Session_3'],545,308,237,117,184,244,301,55.23,0.7256,0.75,545,489,89.72,486,89.17,0
+Session_4,['Session_4'],597,471,126,84,381,132,465,77.89,0.7348,0.75,597,575,96.31,571,95.64,0
+Test pooled (Sessions 1-3),"['Session_1', 'Session_2', 'Session_3']",1531,927,604,304,618,609,922,60.22,0.7141,0.7,1531,1401,91.51,1391,90.86,0
+Session 4 audit (train split),['Session_4'],597,471,126,84,381,132,465,77.89,0.7348,0.75,597,575,96.31,571,95.64,0
+All supplied (descriptive only),"['Session_1', 'Session_2', 'Session_3', 'Session_4']",2128,1398,730,388,999,741,1387,65.18,0.7199,0.7,2128,1976,92.86,1962,92.2,0
@@ -0,0 +1,21 @@
+% AI pre-label audit only; do not substitute for the human-adjudicated manuscript table.
+\begin{tabular}{lrr}
+\hline
+AI pre-label outcome & Session 4 audit & Test Sessions 1--3 \\
+\hline
+Positive: wearer entry & 75 & 265 \\
+HN: same-speaker pause & 104 & 189 \\
+HN: other-to-other transition & 251 & 228 \\
+HN: wearer backchannel & 11 & 95 \\
+Positive: INT-yield & 9 & 39 \\
+HN: other backchannel & 9 & 71 \\
+HN: non-takeover overlap & 6 & 23 \\
+HN: wearer holds floor & 0 & 12 \\
+HN: other & 0 & 0 \\
+Excluded: valid but out of scope & 110 & 479 \\
+Excluded: insufficient evidence & 22 & 130 \\
+Excluded: data quality & 0 & 0 \\
+Scoring-eligible total & 465 & 922 \\
+All candidates & 597 & 1531 \\
+\hline
+\end{tabular}
@@ -0,0 +1,191 @@
+{
+  "artifact_type": "AI_PRELABEL_DESCRIPTIVE_STATISTICS_NOT_GROUND_TRUTH",
+  "warning": "All supplied rows require human adjudication. Do not place these counts in the manuscript table labeled after human adjudication.",
+  "inputs": [
+    "reports/easycom_entry_int_session1_v14/easycom_eot_int_deepseek_official.jsonl",
+    "reports/easycom_entry_int_session2_v14/easycom_eot_int_deepseek_official.jsonl",
+    "reports/easycom_entry_int_session3_v14/easycom_eot_int_deepseek_official.jsonl",
+    "reports/easycom_entry_int_session4_v14/easycom_eot_int_deepseek_official.jsonl"
+  ],
+  "metadata": {
+    "protocol_versions": [
+      "easycom-wearer-entry-int-candidates-v8"
+    ],
+    "schema_versions": [
+      "6"
+    ],
+    "prompt_versions": [
+      "easycom-wearer-boundary-semantic-prelabel-v15"
+    ],
+    "models": [
+      "deepseek-flash"
+    ],
+    "passes": [
+      "2"
+    ],
+    "human_statuses": {
+      "UNREVIEWED": 2128
+    }
+  },
+  "cohorts": [
+    {
+      "cohort": "Session_1",
+      "sessions": [
+        "Session_1"
+      ],
+      "num_candidates": 496,
+      "num_entry_family": 285,
+      "num_int_family": 211,
+      "num_positive": 116,
+      "num_hard_negative": 182,
+      "num_exclude": 198,
+      "num_scoring_eligible_prelabels": 298,
+      "scoring_eligible_prelabel_percent": 60.08,
+      "mean_ai_confidence": 0.7012,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 496,
+      "num_two_pass_semantic_agreement": 458,
+      "two_pass_semantic_agreement_percent": 92.34,
+      "num_two_pass_full_decision_agreement": 453,
+      "two_pass_full_decision_agreement_percent": 91.33,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session_2",
+      "sessions": [
+        "Session_2"
+      ],
+      "num_candidates": 490,
+      "num_entry_family": 334,
+      "num_int_family": 156,
+      "num_positive": 71,
+      "num_hard_negative": 252,
+      "num_exclude": 167,
+      "num_scoring_eligible_prelabels": 323,
+      "scoring_eligible_prelabel_percent": 65.92,
+      "mean_ai_confidence": 0.7143,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 490,
+      "num_two_pass_semantic_agreement": 454,
+      "two_pass_semantic_agreement_percent": 92.65,
+      "num_two_pass_full_decision_agreement": 452,
+      "two_pass_full_decision_agreement_percent": 92.24,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session_3",
+      "sessions": [
+        "Session_3"
+      ],
+      "num_candidates": 545,
+      "num_entry_family": 308,
+      "num_int_family": 237,
+      "num_positive": 117,
+      "num_hard_negative": 184,
+      "num_exclude": 244,
+      "num_scoring_eligible_prelabels": 301,
+      "scoring_eligible_prelabel_percent": 55.23,
+      "mean_ai_confidence": 0.7256,
+      "median_ai_confidence": 0.75,
+      "num_rows_with_two_passes": 545,
+      "num_two_pass_semantic_agreement": 489,
+      "two_pass_semantic_agreement_percent": 89.72,
+      "num_two_pass_full_decision_agreement": 486,
+      "two_pass_full_decision_agreement_percent": 89.17,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session_4",
+      "sessions": [
+        "Session_4"
+      ],
+      "num_candidates": 597,
+      "num_entry_family": 471,
+      "num_int_family": 126,
+      "num_positive": 84,
+      "num_hard_negative": 381,
+      "num_exclude": 132,
+      "num_scoring_eligible_prelabels": 465,
+      "scoring_eligible_prelabel_percent": 77.89,
+      "mean_ai_confidence": 0.7348,
+      "median_ai_confidence": 0.75,
+      "num_rows_with_two_passes": 597,
+      "num_two_pass_semantic_agreement": 575,
+      "two_pass_semantic_agreement_percent": 96.31,
+      "num_two_pass_full_decision_agreement": 571,
+      "two_pass_full_decision_agreement_percent": 95.64,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Test pooled (Sessions 1-3)",
+      "sessions": [
+        "Session_1",
+        "Session_2",
+        "Session_3"
+      ],
+      "num_candidates": 1531,
+      "num_entry_family": 927,
+      "num_int_family": 604,
+      "num_positive": 304,
+      "num_hard_negative": 618,
+      "num_exclude": 609,
+      "num_scoring_eligible_prelabels": 922,
+      "scoring_eligible_prelabel_percent": 60.22,
+      "mean_ai_confidence": 0.7141,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 1531,
+      "num_two_pass_semantic_agreement": 1401,
+      "two_pass_semantic_agreement_percent": 91.51,
+      "num_two_pass_full_decision_agreement": 1391,
+      "two_pass_full_decision_agreement_percent": 90.86,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session 4 audit (train split)",
+      "sessions": [
+        "Session_4"
+      ],
+      "num_candidates": 597,
+      "num_entry_family": 471,
+      "num_int_family": 126,
+      "num_positive": 84,
+      "num_hard_negative": 381,
+      "num_exclude": 132,
+      "num_scoring_eligible_prelabels": 465,
+      "scoring_eligible_prelabel_percent": 77.89,
+      "mean_ai_confidence": 0.7348,
+      "median_ai_confidence": 0.75,
+      "num_rows_with_two_passes": 597,
+      "num_two_pass_semantic_agreement": 575,
+      "two_pass_semantic_agreement_percent": 96.31,
+      "num_two_pass_full_decision_agreement": 571,
+      "two_pass_full_decision_agreement_percent": 95.64,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "All supplied (descriptive only)",
+      "sessions": [
+        "Session_1",
+        "Session_2",
+        "Session_3",
+        "Session_4"
+      ],
+      "num_candidates": 2128,
+      "num_entry_family": 1398,
+      "num_int_family": 730,
+      "num_positive": 388,
+      "num_hard_negative": 999,
+      "num_exclude": 741,
+      "num_scoring_eligible_prelabels": 1387,
+      "scoring_eligible_prelabel_percent": 65.18,
+      "mean_ai_confidence": 0.7199,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 2128,
+      "num_two_pass_semantic_agreement": 1976,
+      "two_pass_semantic_agreement_percent": 92.86,
+      "num_two_pass_full_decision_agreement": 1962,
+      "two_pass_full_decision_agreement_percent": 92.2,
+      "num_human_reviewed": 0
+    }
+  ]
+}
@@ -0,0 +1,125 @@
+cohort,dimension,label,count,percent_of_all_candidates
+Session_1,verdict,EXCLUDE,198,39.92
+Session_1,verdict,HARD_NEGATIVE,182,36.69
+Session_1,verdict,POSITIVE,116,23.39
+Session_1,positive_event,INT_YIELD,20,4.03
+Session_1,positive_event,WEARER_ENTRY,96,19.35
+Session_1,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,58,11.69
+Session_1,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,56,11.29
+Session_1,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,28,5.65
+Session_1,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,10,2.02
+Session_1,hard_negative_subtype,INT_OTHER_BACKCHANNEL,29,5.85
+Session_1,hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,1,0.2
+Session_1,exclude_reason,INSUFFICIENT_EVIDENCE,38,7.66
+Session_1,exclude_reason,OUT_OF_SCOPE_NON_EVENT,8,1.61
+Session_1,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,104,20.97
+Session_1,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,21,4.23
+Session_1,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,27,5.44
+Session_1,exclude_group,INSUFFICIENT_EVIDENCE,38,7.66
+Session_1,exclude_group,OUT_OF_SCOPE,160,32.26
+Session_2,verdict,EXCLUDE,167,34.08
+Session_2,verdict,HARD_NEGATIVE,252,51.43
+Session_2,verdict,POSITIVE,71,14.49
+Session_2,positive_event,INT_YIELD,5,1.02
+Session_2,positive_event,WEARER_ENTRY,66,13.47
+Session_2,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,76,15.51
+Session_2,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,123,25.1
+Session_2,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,27,5.51
+Session_2,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,7,1.43
+Session_2,hard_negative_subtype,INT_OTHER_BACKCHANNEL,15,3.06
+Session_2,hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,4,0.82
+Session_2,exclude_reason,INSUFFICIENT_EVIDENCE,36,7.35
+Session_2,exclude_reason,OUT_OF_SCOPE_NON_EVENT,3,0.61
+Session_2,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,97,19.8
+Session_2,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,4,0.82
+Session_2,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,27,5.51
+Session_2,exclude_group,INSUFFICIENT_EVIDENCE,36,7.35
+Session_2,exclude_group,OUT_OF_SCOPE,131,26.73
+Session_3,verdict,EXCLUDE,244,44.77
+Session_3,verdict,HARD_NEGATIVE,184,33.76
+Session_3,verdict,POSITIVE,117,21.47
+Session_3,positive_event,INT_YIELD,14,2.57
+Session_3,positive_event,WEARER_ENTRY,103,18.9
+Session_3,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,55,10.09
+Session_3,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,49,8.99
+Session_3,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,40,7.34
+Session_3,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,6,1.1
+Session_3,hard_negative_subtype,INT_OTHER_BACKCHANNEL,27,4.95
+Session_3,hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,7,1.28
+Session_3,exclude_reason,INSUFFICIENT_EVIDENCE,56,10.28
+Session_3,exclude_reason,OUT_OF_SCOPE_NON_EVENT,23,4.22
+Session_3,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,99,18.17
+Session_3,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,25,4.59
+Session_3,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,41,7.52
+Session_3,exclude_group,INSUFFICIENT_EVIDENCE,56,10.28
+Session_3,exclude_group,OUT_OF_SCOPE,188,34.5
+Session_4,verdict,EXCLUDE,132,22.11
+Session_4,verdict,HARD_NEGATIVE,381,63.82
+Session_4,verdict,POSITIVE,84,14.07
+Session_4,positive_event,INT_YIELD,9,1.51
+Session_4,positive_event,WEARER_ENTRY,75,12.56
+Session_4,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,104,17.42
+Session_4,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,251,42.04
+Session_4,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,11,1.84
+Session_4,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,6,1.01
+Session_4,hard_negative_subtype,INT_OTHER_BACKCHANNEL,9,1.51
+Session_4,exclude_reason,INSUFFICIENT_EVIDENCE,22,3.69
+Session_4,exclude_reason,OUT_OF_SCOPE_NON_EVENT,4,0.67
+Session_4,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,90,15.08
+Session_4,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,3,0.5
+Session_4,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,13,2.18
+Session_4,exclude_group,INSUFFICIENT_EVIDENCE,22,3.69
+Session_4,exclude_group,OUT_OF_SCOPE,110,18.43
+Test pooled (Sessions 1-3),verdict,EXCLUDE,609,39.78
+Test pooled (Sessions 1-3),verdict,HARD_NEGATIVE,618,40.37
+Test pooled (Sessions 1-3),verdict,POSITIVE,304,19.86
+Test pooled (Sessions 1-3),positive_event,INT_YIELD,39,2.55
+Test pooled (Sessions 1-3),positive_event,WEARER_ENTRY,265,17.31
+Test pooled (Sessions 1-3),hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,189,12.34
+Test pooled (Sessions 1-3),hard_negative_subtype,ENTRY_OTHER_TO_OTHER,228,14.89
+Test pooled (Sessions 1-3),hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,95,6.21
+Test pooled (Sessions 1-3),hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,23,1.5
+Test pooled (Sessions 1-3),hard_negative_subtype,INT_OTHER_BACKCHANNEL,71,4.64
+Test pooled (Sessions 1-3),hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,12,0.78
+Test pooled (Sessions 1-3),exclude_reason,INSUFFICIENT_EVIDENCE,130,8.49
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_NON_EVENT,34,2.22
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,300,19.6
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,50,3.27
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,95,6.21
+Test pooled (Sessions 1-3),exclude_group,INSUFFICIENT_EVIDENCE,130,8.49
+Test pooled (Sessions 1-3),exclude_group,OUT_OF_SCOPE,479,31.29
+Session 4 audit (train split),verdict,EXCLUDE,132,22.11
+Session 4 audit (train split),verdict,HARD_NEGATIVE,381,63.82
+Session 4 audit (train split),verdict,POSITIVE,84,14.07
+Session 4 audit (train split),positive_event,INT_YIELD,9,1.51
+Session 4 audit (train split),positive_event,WEARER_ENTRY,75,12.56
+Session 4 audit (train split),hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,104,17.42
+Session 4 audit (train split),hard_negative_subtype,ENTRY_OTHER_TO_OTHER,251,42.04
+Session 4 audit (train split),hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,11,1.84
+Session 4 audit (train split),hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,6,1.01
+Session 4 audit (train split),hard_negative_subtype,INT_OTHER_BACKCHANNEL,9,1.51
+Session 4 audit (train split),exclude_reason,INSUFFICIENT_EVIDENCE,22,3.69
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_NON_EVENT,4,0.67
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,90,15.08
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,3,0.5
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,13,2.18
+Session 4 audit (train split),exclude_group,INSUFFICIENT_EVIDENCE,22,3.69
+Session 4 audit (train split),exclude_group,OUT_OF_SCOPE,110,18.43
+All supplied (descriptive only),verdict,EXCLUDE,741,34.82
+All supplied (descriptive only),verdict,HARD_NEGATIVE,999,46.95
+All supplied (descriptive only),verdict,POSITIVE,388,18.23
+All supplied (descriptive only),positive_event,INT_YIELD,48,2.26
+All supplied (descriptive only),positive_event,WEARER_ENTRY,340,15.98
+All supplied (descriptive only),hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,293,13.77
+All supplied (descriptive only),hard_negative_subtype,ENTRY_OTHER_TO_OTHER,479,22.51
+All supplied (descriptive only),hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,106,4.98
+All supplied (descriptive only),hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,29,1.36
+All supplied (descriptive only),hard_negative_subtype,INT_OTHER_BACKCHANNEL,80,3.76
+All supplied (descriptive only),hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,12,0.56
+All supplied (descriptive only),exclude_reason,INSUFFICIENT_EVIDENCE,152,7.14
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_NON_EVENT,38,1.79
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,390,18.33
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,53,2.49
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,108,5.08
+All supplied (descriptive only),exclude_group,INSUFFICIENT_EVIDENCE,152,7.14
+All supplied (descriptive only),exclude_group,OUT_OF_SCOPE,589,27.68