33 lines
2.3 KiB
Markdown
33 lines
2.3 KiB
Markdown
|
# EasyCom event AI pre-label audit
|
||
|
|
|
||
|
|
> These are DeepSeek pre-labels awaiting mandatory human adjudication, not ground truth and not final manuscript annotation counts.
|
||
|
|
|
||
|
|
Inputs contain 2,128 unique candidates from Session_1, Session_2, Session_3, Session_4. Protocol: `easycom-wearer-entry-int-candidates-v8`; prompt: `easycom-wearer-boundary-semantic-prelabel-v15`.
|
||
|
|
|
||
|
|
## Cohort overview
|
||
|
|
|
||
|
|
| Cohort | Candidates | Positive | Hard negative | Exclude | Prelabel scoring-eligible | Two-pass semantic/full agreement | Human reviewed |
|
||
|
|
|---|---:|---:|---:|---:|---:|---:|---:|
|
||
|
|
| Session_1 | 496 | 116 | 182 | 198 | 298 (60.08%) | 458/496 (92.34%) / 453/496 (91.33%) | 0 |
|
||
|
|
| Session_2 | 490 | 71 | 252 | 167 | 323 (65.92%) | 454/490 (92.65%) / 452/490 (92.24%) | 0 |
|
||
|
|
| Session_3 | 545 | 117 | 184 | 244 | 301 (55.23%) | 489/545 (89.72%) / 486/545 (89.17%) | 0 |
|
||
|
|
| Session_4 | 597 | 84 | 381 | 132 | 465 (77.89%) | 575/597 (96.31%) / 571/597 (95.64%) | 0 |
|
||
|
|
| Test pooled (Sessions 1-3) | 1531 | 304 | 618 | 609 | 922 (60.22%) | 1401/1531 (91.51%) / 1391/1531 (90.86%) | 0 |
|
||
|
|
| Session 4 audit (train split) | 597 | 84 | 381 | 132 | 465 (77.89%) | 575/597 (96.31%) / 571/597 (95.64%) | 0 |
|
||
|
|
| All supplied (descriptive only) | 2128 | 388 | 999 | 741 | 1387 (65.18%) | 1976/2128 (92.86%) / 1962/2128 (92.20%) | 0 |
|
||
|
|
|
||
|
|
The official EasyCom test sessions (1--3) are pooled only for descriptive reporting; these AI pre-labels must not be used to tune thresholds, taxonomy, prompts, or model choices. Session 4 is shown separately as a train-split audit and is not the complete Sessions 4--12 training corpus.
|
||
|
|
|
||
|
|
Detailed counts are in `taxonomy_counts.csv` and `candidate_source_counts.csv`. `manuscript_ai_prelabel_table.tex` mirrors the manuscript row order but is deliberately labeled as an AI audit; retain the manuscript TODOs until human adjudication is frozen.
|
||
|
|
|
||
|
|
## Reproduce
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python3 summarize_easycom_event_prelabels.py \
|
||
|
|
--input reports/easycom_entry_int_session1_v14/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--input reports/easycom_entry_int_session2_v14/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--input reports/easycom_entry_int_session3_v14/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--input reports/easycom_entry_int_session4_v14/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--output-dir reports/easycom_event_ai_prelabel_stats_v15
|
||
|
|
```
|