Add EasyCom v16 AI prelabel results

2026-09-15 12:59:03 +08:00
parent bb8963b80d
commit c710b9e023
20 changed files with 2821 additions and 1 deletions
@@ -0,0 +1,32 @@
+# EasyCom event AI pre-label audit
+
+> These are DeepSeek pre-labels awaiting mandatory human adjudication, not ground truth and not final manuscript annotation counts.
+
+Inputs contain 2,128 unique candidates from Session_1, Session_2, Session_3, Session_4. Protocol: `easycom-wearer-entry-int-candidates-v8`; prompt: `easycom-wearer-boundary-semantic-prelabel-v16`.
+
+## Cohort overview
+
+| Cohort | Candidates | Positive | Hard negative | Exclude | Prelabel scoring-eligible | Two-pass semantic/full agreement | Human reviewed |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Session_1 | 496 | 113 | 192 | 191 | 305 (61.49%) | 458/496 (92.34%) / 456/496 (91.94%) | 0 |
+| Session_2 | 490 | 64 | 260 | 166 | 324 (66.12%) | 467/490 (95.31%) / 464/490 (94.69%) | 0 |
+| Session_3 | 545 | 112 | 200 | 233 | 312 (57.25%) | 507/545 (93.03%) / 504/545 (92.48%) | 0 |
+| Session_4 | 597 | 78 | 365 | 154 | 443 (74.20%) | 555/597 (92.96%) / 550/597 (92.13%) | 0 |
+| Test pooled (Sessions 1-3) | 1531 | 289 | 652 | 590 | 941 (61.46%) | 1432/1531 (93.53%) / 1424/1531 (93.01%) | 0 |
+| Session 4 audit (train split) | 597 | 78 | 365 | 154 | 443 (74.20%) | 555/597 (92.96%) / 550/597 (92.13%) | 0 |
+| All supplied (descriptive only) | 2128 | 367 | 1017 | 744 | 1384 (65.04%) | 1987/2128 (93.37%) / 1974/2128 (92.76%) | 0 |
+
+The official EasyCom test sessions (1--3) are pooled only for descriptive reporting; these AI pre-labels must not be used to tune thresholds, taxonomy, prompts, or model choices. Session 4 is shown separately as a train-split audit and is not the complete Sessions 4--12 training corpus.
+
+Detailed counts are in `taxonomy_counts.csv` and `candidate_source_counts.csv`. `manuscript_ai_prelabel_table.tex` mirrors the manuscript row order but is deliberately labeled as an AI audit; retain the manuscript TODOs until human adjudication is frozen.
+
+## Reproduce
+
+```bash
+python3 summarize_easycom_event_prelabels.py \
+  --input reports/easycom_entry_int_session1_v16/easycom_eot_int_deepseek_official.jsonl \
+  --input reports/easycom_entry_int_session2_v16/easycom_eot_int_deepseek_official.jsonl \
+  --input reports/easycom_entry_int_session3_v16/easycom_eot_int_deepseek_official.jsonl \
+  --input reports/easycom_entry_int_session4_v16/easycom_eot_int_deepseek_official.jsonl \
+  --output-dir reports/easycom_event_ai_prelabel_stats_v16
+```
@@ -0,0 +1,43 @@
+cohort,candidate_source,count,percent_of_all_candidates
+Session_1,NON_TAKEOVER_OVERLAP,21,4.23
+Session_1,OTHER_BACKCHANNEL,20,4.03
+Session_1,OTHER_TO_OTHER,56,11.29
+Session_1,SAME_SPEAKER_PAUSE,60,12.1
+Session_1,WEARER_OFFSET,170,34.27
+Session_1,WEARER_ONSET,169,34.07
+Session_2,NON_TAKEOVER_OVERLAP,14,2.86
+Session_2,OTHER_BACKCHANNEL,15,3.06
+Session_2,OTHER_TO_OTHER,123,25.1
+Session_2,SAME_SPEAKER_PAUSE,84,17.14
+Session_2,WEARER_OFFSET,127,25.92
+Session_2,WEARER_ONSET,127,25.92
+Session_3,NON_TAKEOVER_OVERLAP,22,4.04
+Session_3,OTHER_BACKCHANNEL,19,3.49
+Session_3,OTHER_TO_OTHER,49,8.99
+Session_3,SAME_SPEAKER_PAUSE,63,11.56
+Session_3,WEARER_OFFSET,196,35.96
+Session_3,WEARER_ONSET,196,35.96
+Session_4,NON_TAKEOVER_OVERLAP,12,2.01
+Session_4,OTHER_BACKCHANNEL,3,0.5
+Session_4,OTHER_TO_OTHER,255,42.71
+Session_4,SAME_SPEAKER_PAUSE,105,17.59
+Session_4,WEARER_OFFSET,111,18.59
+Session_4,WEARER_ONSET,111,18.59
+Test pooled (Sessions 1-3),NON_TAKEOVER_OVERLAP,57,3.72
+Test pooled (Sessions 1-3),OTHER_BACKCHANNEL,54,3.53
+Test pooled (Sessions 1-3),OTHER_TO_OTHER,228,14.89
+Test pooled (Sessions 1-3),SAME_SPEAKER_PAUSE,207,13.52
+Test pooled (Sessions 1-3),WEARER_OFFSET,493,32.2
+Test pooled (Sessions 1-3),WEARER_ONSET,492,32.14
+Session 4 audit (train split),NON_TAKEOVER_OVERLAP,12,2.01
+Session 4 audit (train split),OTHER_BACKCHANNEL,3,0.5
+Session 4 audit (train split),OTHER_TO_OTHER,255,42.71
+Session 4 audit (train split),SAME_SPEAKER_PAUSE,105,17.59
+Session 4 audit (train split),WEARER_OFFSET,111,18.59
+Session 4 audit (train split),WEARER_ONSET,111,18.59
+All supplied (descriptive only),NON_TAKEOVER_OVERLAP,69,3.24
+All supplied (descriptive only),OTHER_BACKCHANNEL,57,2.68
+All supplied (descriptive only),OTHER_TO_OTHER,483,22.7
+All supplied (descriptive only),SAME_SPEAKER_PAUSE,312,14.66
+All supplied (descriptive only),WEARER_OFFSET,604,28.38
+All supplied (descriptive only),WEARER_ONSET,603,28.34
@@ -0,0 +1,8 @@
+cohort,sessions,num_candidates,num_entry_family,num_int_family,num_positive,num_hard_negative,num_exclude,num_scoring_eligible_prelabels,scoring_eligible_prelabel_percent,mean_ai_confidence,median_ai_confidence,num_rows_with_two_passes,num_two_pass_semantic_agreement,two_pass_semantic_agreement_percent,num_two_pass_full_decision_agreement,two_pass_full_decision_agreement_percent,num_human_reviewed
+Session_1,['Session_1'],496,285,211,113,192,191,305,61.49,0.6863,0.7,496,458,92.34,456,91.94,0
+Session_2,['Session_2'],490,334,156,64,260,166,324,66.12,0.6966,0.7,490,467,95.31,464,94.69,0
+Session_3,['Session_3'],545,308,237,112,200,233,312,57.25,0.7134,0.7,545,507,93.03,504,92.48,0
+Session_4,['Session_4'],597,471,126,78,365,154,443,74.2,0.6997,0.7,597,555,92.96,550,92.13,0
+Test pooled (Sessions 1-3),"['Session_1', 'Session_2', 'Session_3']",1531,927,604,289,652,590,941,61.46,0.6992,0.7,1531,1432,93.53,1424,93.01,0
+Session 4 audit (train split),['Session_4'],597,471,126,78,365,154,443,74.2,0.6997,0.7,597,555,92.96,550,92.13,0
+All supplied (descriptive only),"['Session_1', 'Session_2', 'Session_3', 'Session_4']",2128,1398,730,367,1017,744,1384,65.04,0.6994,0.7,2128,1987,93.37,1974,92.76,0
@@ -0,0 +1,21 @@
+% AI pre-label audit only; do not substitute for the human-adjudicated manuscript table.
+\begin{tabular}{lrr}
+\hline
+AI pre-label outcome & Session 4 audit & Test Sessions 1--3 \\
+\hline
+Positive: wearer entry & 72 & 252 \\
+HN: same-speaker pause & 89 & 199 \\
+HN: other-to-other transition & 248 & 227 \\
+HN: wearer backchannel & 13 & 116 \\
+Positive: INT-yield & 6 & 37 \\
+HN: other backchannel & 9 & 77 \\
+HN: non-takeover overlap & 6 & 25 \\
+HN: wearer holds floor & 0 & 8 \\
+HN: other & 0 & 0 \\
+Excluded: valid but out of scope & 111 & 491 \\
+Excluded: insufficient evidence & 43 & 99 \\
+Excluded: data quality & 0 & 0 \\
+Scoring-eligible total & 443 & 941 \\
+All candidates & 597 & 1531 \\
+\hline
+\end{tabular}
@@ -0,0 +1,191 @@
+{
+  "artifact_type": "AI_PRELABEL_DESCRIPTIVE_STATISTICS_NOT_GROUND_TRUTH",
+  "warning": "All supplied rows require human adjudication. Do not place these counts in the manuscript table labeled after human adjudication.",
+  "inputs": [
+    "reports/easycom_entry_int_session1_v16/easycom_eot_int_deepseek_official.jsonl",
+    "reports/easycom_entry_int_session2_v16/easycom_eot_int_deepseek_official.jsonl",
+    "reports/easycom_entry_int_session3_v16/easycom_eot_int_deepseek_official.jsonl",
+    "reports/easycom_entry_int_session4_v16/easycom_eot_int_deepseek_official.jsonl"
+  ],
+  "metadata": {
+    "protocol_versions": [
+      "easycom-wearer-entry-int-candidates-v8"
+    ],
+    "schema_versions": [
+      "6"
+    ],
+    "prompt_versions": [
+      "easycom-wearer-boundary-semantic-prelabel-v16"
+    ],
+    "models": [
+      "deepseek-flash"
+    ],
+    "passes": [
+      "2"
+    ],
+    "human_statuses": {
+      "UNREVIEWED": 2128
+    }
+  },
+  "cohorts": [
+    {
+      "cohort": "Session_1",
+      "sessions": [
+        "Session_1"
+      ],
+      "num_candidates": 496,
+      "num_entry_family": 285,
+      "num_int_family": 211,
+      "num_positive": 113,
+      "num_hard_negative": 192,
+      "num_exclude": 191,
+      "num_scoring_eligible_prelabels": 305,
+      "scoring_eligible_prelabel_percent": 61.49,
+      "mean_ai_confidence": 0.6863,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 496,
+      "num_two_pass_semantic_agreement": 458,
+      "two_pass_semantic_agreement_percent": 92.34,
+      "num_two_pass_full_decision_agreement": 456,
+      "two_pass_full_decision_agreement_percent": 91.94,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session_2",
+      "sessions": [
+        "Session_2"
+      ],
+      "num_candidates": 490,
+      "num_entry_family": 334,
+      "num_int_family": 156,
+      "num_positive": 64,
+      "num_hard_negative": 260,
+      "num_exclude": 166,
+      "num_scoring_eligible_prelabels": 324,
+      "scoring_eligible_prelabel_percent": 66.12,
+      "mean_ai_confidence": 0.6966,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 490,
+      "num_two_pass_semantic_agreement": 467,
+      "two_pass_semantic_agreement_percent": 95.31,
+      "num_two_pass_full_decision_agreement": 464,
+      "two_pass_full_decision_agreement_percent": 94.69,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session_3",
+      "sessions": [
+        "Session_3"
+      ],
+      "num_candidates": 545,
+      "num_entry_family": 308,
+      "num_int_family": 237,
+      "num_positive": 112,
+      "num_hard_negative": 200,
+      "num_exclude": 233,
+      "num_scoring_eligible_prelabels": 312,
+      "scoring_eligible_prelabel_percent": 57.25,
+      "mean_ai_confidence": 0.7134,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 545,
+      "num_two_pass_semantic_agreement": 507,
+      "two_pass_semantic_agreement_percent": 93.03,
+      "num_two_pass_full_decision_agreement": 504,
+      "two_pass_full_decision_agreement_percent": 92.48,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session_4",
+      "sessions": [
+        "Session_4"
+      ],
+      "num_candidates": 597,
+      "num_entry_family": 471,
+      "num_int_family": 126,
+      "num_positive": 78,
+      "num_hard_negative": 365,
+      "num_exclude": 154,
+      "num_scoring_eligible_prelabels": 443,
+      "scoring_eligible_prelabel_percent": 74.2,
+      "mean_ai_confidence": 0.6997,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 597,
+      "num_two_pass_semantic_agreement": 555,
+      "two_pass_semantic_agreement_percent": 92.96,
+      "num_two_pass_full_decision_agreement": 550,
+      "two_pass_full_decision_agreement_percent": 92.13,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Test pooled (Sessions 1-3)",
+      "sessions": [
+        "Session_1",
+        "Session_2",
+        "Session_3"
+      ],
+      "num_candidates": 1531,
+      "num_entry_family": 927,
+      "num_int_family": 604,
+      "num_positive": 289,
+      "num_hard_negative": 652,
+      "num_exclude": 590,
+      "num_scoring_eligible_prelabels": 941,
+      "scoring_eligible_prelabel_percent": 61.46,
+      "mean_ai_confidence": 0.6992,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 1531,
+      "num_two_pass_semantic_agreement": 1432,
+      "two_pass_semantic_agreement_percent": 93.53,
+      "num_two_pass_full_decision_agreement": 1424,
+      "two_pass_full_decision_agreement_percent": 93.01,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "Session 4 audit (train split)",
+      "sessions": [
+        "Session_4"
+      ],
+      "num_candidates": 597,
+      "num_entry_family": 471,
+      "num_int_family": 126,
+      "num_positive": 78,
+      "num_hard_negative": 365,
+      "num_exclude": 154,
+      "num_scoring_eligible_prelabels": 443,
+      "scoring_eligible_prelabel_percent": 74.2,
+      "mean_ai_confidence": 0.6997,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 597,
+      "num_two_pass_semantic_agreement": 555,
+      "two_pass_semantic_agreement_percent": 92.96,
+      "num_two_pass_full_decision_agreement": 550,
+      "two_pass_full_decision_agreement_percent": 92.13,
+      "num_human_reviewed": 0
+    },
+    {
+      "cohort": "All supplied (descriptive only)",
+      "sessions": [
+        "Session_1",
+        "Session_2",
+        "Session_3",
+        "Session_4"
+      ],
+      "num_candidates": 2128,
+      "num_entry_family": 1398,
+      "num_int_family": 730,
+      "num_positive": 367,
+      "num_hard_negative": 1017,
+      "num_exclude": 744,
+      "num_scoring_eligible_prelabels": 1384,
+      "scoring_eligible_prelabel_percent": 65.04,
+      "mean_ai_confidence": 0.6994,
+      "median_ai_confidence": 0.7,
+      "num_rows_with_two_passes": 2128,
+      "num_two_pass_semantic_agreement": 1987,
+      "two_pass_semantic_agreement_percent": 93.37,
+      "num_two_pass_full_decision_agreement": 1974,
+      "two_pass_full_decision_agreement_percent": 92.76,
+      "num_human_reviewed": 0
+    }
+  ]
+}
@@ -0,0 +1,129 @@
+cohort,dimension,label,count,percent_of_all_candidates
+Session_1,verdict,EXCLUDE,191,38.51
+Session_1,verdict,HARD_NEGATIVE,192,38.71
+Session_1,verdict,POSITIVE,113,22.78
+Session_1,positive_event,INT_YIELD,20,4.03
+Session_1,positive_event,WEARER_ENTRY,93,18.75
+Session_1,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,60,12.1
+Session_1,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,56,11.29
+Session_1,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,35,7.06
+Session_1,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,11,2.22
+Session_1,hard_negative_subtype,INT_OTHER_BACKCHANNEL,30,6.05
+Session_1,exclude_reason,INSUFFICIENT_EVIDENCE,38,7.66
+Session_1,exclude_reason,OUT_OF_SCOPE_NON_EVENT,9,1.81
+Session_1,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,81,16.33
+Session_1,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,11,2.22
+Session_1,exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,28,5.65
+Session_1,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,24,4.84
+Session_1,exclude_group,INSUFFICIENT_EVIDENCE,38,7.66
+Session_1,exclude_group,OUT_OF_SCOPE,153,30.85
+Session_2,verdict,EXCLUDE,166,33.88
+Session_2,verdict,HARD_NEGATIVE,260,53.06
+Session_2,verdict,POSITIVE,64,13.06
+Session_2,positive_event,INT_YIELD,4,0.82
+Session_2,positive_event,WEARER_ENTRY,60,12.24
+Session_2,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,76,15.51
+Session_2,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,122,24.9
+Session_2,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,36,7.35
+Session_2,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,8,1.63
+Session_2,hard_negative_subtype,INT_OTHER_BACKCHANNEL,15,3.06
+Session_2,hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,3,0.61
+Session_2,exclude_reason,INSUFFICIENT_EVIDENCE,23,4.69
+Session_2,exclude_reason,OUT_OF_SCOPE_NON_EVENT,2,0.41
+Session_2,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,77,15.71
+Session_2,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,2,0.41
+Session_2,exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,36,7.35
+Session_2,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,26,5.31
+Session_2,exclude_group,INSUFFICIENT_EVIDENCE,23,4.69
+Session_2,exclude_group,OUT_OF_SCOPE,143,29.18
+Session_3,verdict,EXCLUDE,233,42.75
+Session_3,verdict,HARD_NEGATIVE,200,36.7
+Session_3,verdict,POSITIVE,112,20.55
+Session_3,positive_event,INT_YIELD,13,2.39
+Session_3,positive_event,WEARER_ENTRY,99,18.17
+Session_3,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,63,11.56
+Session_3,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,49,8.99
+Session_3,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,45,8.26
+Session_3,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,6,1.1
+Session_3,hard_negative_subtype,INT_OTHER_BACKCHANNEL,32,5.87
+Session_3,hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,5,0.92
+Session_3,exclude_reason,INSUFFICIENT_EVIDENCE,38,6.97
+Session_3,exclude_reason,OUT_OF_SCOPE_NON_EVENT,17,3.12
+Session_3,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,88,16.15
+Session_3,exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,11,2.02
+Session_3,exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,44,8.07
+Session_3,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,35,6.42
+Session_3,exclude_group,INSUFFICIENT_EVIDENCE,38,6.97
+Session_3,exclude_group,OUT_OF_SCOPE,195,35.78
+Session_4,verdict,EXCLUDE,154,25.8
+Session_4,verdict,HARD_NEGATIVE,365,61.14
+Session_4,verdict,POSITIVE,78,13.07
+Session_4,positive_event,INT_YIELD,6,1.01
+Session_4,positive_event,WEARER_ENTRY,72,12.06
+Session_4,hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,89,14.91
+Session_4,hard_negative_subtype,ENTRY_OTHER_TO_OTHER,248,41.54
+Session_4,hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,13,2.18
+Session_4,hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,6,1.01
+Session_4,hard_negative_subtype,INT_OTHER_BACKCHANNEL,9,1.51
+Session_4,exclude_reason,INSUFFICIENT_EVIDENCE,43,7.2
+Session_4,exclude_reason,OUT_OF_SCOPE_NON_EVENT,2,0.34
+Session_4,exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,83,13.9
+Session_4,exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,12,2.01
+Session_4,exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,14,2.35
+Session_4,exclude_group,INSUFFICIENT_EVIDENCE,43,7.2
+Session_4,exclude_group,OUT_OF_SCOPE,111,18.59
+Test pooled (Sessions 1-3),verdict,EXCLUDE,590,38.54
+Test pooled (Sessions 1-3),verdict,HARD_NEGATIVE,652,42.59
+Test pooled (Sessions 1-3),verdict,POSITIVE,289,18.88
+Test pooled (Sessions 1-3),positive_event,INT_YIELD,37,2.42
+Test pooled (Sessions 1-3),positive_event,WEARER_ENTRY,252,16.46
+Test pooled (Sessions 1-3),hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,199,13.0
+Test pooled (Sessions 1-3),hard_negative_subtype,ENTRY_OTHER_TO_OTHER,227,14.83
+Test pooled (Sessions 1-3),hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,116,7.58
+Test pooled (Sessions 1-3),hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,25,1.63
+Test pooled (Sessions 1-3),hard_negative_subtype,INT_OTHER_BACKCHANNEL,77,5.03
+Test pooled (Sessions 1-3),hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,8,0.52
+Test pooled (Sessions 1-3),exclude_reason,INSUFFICIENT_EVIDENCE,99,6.47
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_NON_EVENT,28,1.83
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,246,16.07
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,24,1.57
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,108,7.05
+Test pooled (Sessions 1-3),exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,85,5.55
+Test pooled (Sessions 1-3),exclude_group,INSUFFICIENT_EVIDENCE,99,6.47
+Test pooled (Sessions 1-3),exclude_group,OUT_OF_SCOPE,491,32.07
+Session 4 audit (train split),verdict,EXCLUDE,154,25.8
+Session 4 audit (train split),verdict,HARD_NEGATIVE,365,61.14
+Session 4 audit (train split),verdict,POSITIVE,78,13.07
+Session 4 audit (train split),positive_event,INT_YIELD,6,1.01
+Session 4 audit (train split),positive_event,WEARER_ENTRY,72,12.06
+Session 4 audit (train split),hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,89,14.91
+Session 4 audit (train split),hard_negative_subtype,ENTRY_OTHER_TO_OTHER,248,41.54
+Session 4 audit (train split),hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,13,2.18
+Session 4 audit (train split),hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,6,1.01
+Session 4 audit (train split),hard_negative_subtype,INT_OTHER_BACKCHANNEL,9,1.51
+Session 4 audit (train split),exclude_reason,INSUFFICIENT_EVIDENCE,43,7.2
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_NON_EVENT,2,0.34
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,83,13.9
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,12,2.01
+Session 4 audit (train split),exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,14,2.35
+Session 4 audit (train split),exclude_group,INSUFFICIENT_EVIDENCE,43,7.2
+Session 4 audit (train split),exclude_group,OUT_OF_SCOPE,111,18.59
+All supplied (descriptive only),verdict,EXCLUDE,744,34.96
+All supplied (descriptive only),verdict,HARD_NEGATIVE,1017,47.79
+All supplied (descriptive only),verdict,POSITIVE,367,17.25
+All supplied (descriptive only),positive_event,INT_YIELD,43,2.02
+All supplied (descriptive only),positive_event,WEARER_ENTRY,324,15.23
+All supplied (descriptive only),hard_negative_subtype,ENTRY_OTHER_MIDTURN_PAUSE,288,13.53
+All supplied (descriptive only),hard_negative_subtype,ENTRY_OTHER_TO_OTHER,475,22.32
+All supplied (descriptive only),hard_negative_subtype,ENTRY_WEARER_BACKCHANNEL,129,6.06
+All supplied (descriptive only),hard_negative_subtype,INT_NON_TAKEOVER_OVERLAP,31,1.46
+All supplied (descriptive only),hard_negative_subtype,INT_OTHER_BACKCHANNEL,86,4.04
+All supplied (descriptive only),hard_negative_subtype,INT_WEARER_HOLDS_FLOOR,8,0.38
+All supplied (descriptive only),exclude_reason,INSUFFICIENT_EVIDENCE,142,6.67
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_NON_EVENT,30,1.41
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_NORMAL_COMPLETION,329,15.46
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_POST_COMPLETION_ACTIVITY,24,1.13
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_WEARER_BACKCHANNEL_OFFSET,120,5.64
+All supplied (descriptive only),exclude_reason,OUT_OF_SCOPE_WITHIN_TURN_CONTINUATION,99,4.65
+All supplied (descriptive only),exclude_group,INSUFFICIENT_EVIDENCE,142,6.67
+All supplied (descriptive only),exclude_group,OUT_OF_SCOPE,602,28.29