33 lines
2.3 KiB
Markdown
33 lines
2.3 KiB
Markdown
|
# EasyCom event AI pre-label audit
|
||
|
|
|
||
|
|
> These are DeepSeek pre-labels awaiting mandatory human adjudication, not ground truth and not final manuscript annotation counts.
|
||
|
|
|
||
|
|
Inputs contain 2,128 unique candidates from Session_1, Session_2, Session_3, Session_4. Protocol: `easycom-wearer-entry-int-candidates-v8`; prompt: `easycom-wearer-boundary-semantic-prelabel-v16`.
|
||
|
|
|
||
|
|
## Cohort overview
|
||
|
|
|
||
|
|
| Cohort | Candidates | Positive | Hard negative | Exclude | Prelabel scoring-eligible | Two-pass semantic/full agreement | Human reviewed |
|
||
|
|
|---|---:|---:|---:|---:|---:|---:|---:|
|
||
|
|
| Session_1 | 496 | 113 | 192 | 191 | 305 (61.49%) | 458/496 (92.34%) / 456/496 (91.94%) | 0 |
|
||
|
|
| Session_2 | 490 | 64 | 260 | 166 | 324 (66.12%) | 467/490 (95.31%) / 464/490 (94.69%) | 0 |
|
||
|
|
| Session_3 | 545 | 112 | 200 | 233 | 312 (57.25%) | 507/545 (93.03%) / 504/545 (92.48%) | 0 |
|
||
|
|
| Session_4 | 597 | 78 | 365 | 154 | 443 (74.20%) | 555/597 (92.96%) / 550/597 (92.13%) | 0 |
|
||
|
|
| Test pooled (Sessions 1-3) | 1531 | 289 | 652 | 590 | 941 (61.46%) | 1432/1531 (93.53%) / 1424/1531 (93.01%) | 0 |
|
||
|
|
| Session 4 audit (train split) | 597 | 78 | 365 | 154 | 443 (74.20%) | 555/597 (92.96%) / 550/597 (92.13%) | 0 |
|
||
|
|
| All supplied (descriptive only) | 2128 | 367 | 1017 | 744 | 1384 (65.04%) | 1987/2128 (93.37%) / 1974/2128 (92.76%) | 0 |
|
||
|
|
|
||
|
|
The official EasyCom test sessions (1--3) are pooled only for descriptive reporting; these AI pre-labels must not be used to tune thresholds, taxonomy, prompts, or model choices. Session 4 is shown separately as a train-split audit and is not the complete Sessions 4--12 training corpus.
|
||
|
|
|
||
|
|
Detailed counts are in `taxonomy_counts.csv` and `candidate_source_counts.csv`. `manuscript_ai_prelabel_table.tex` mirrors the manuscript row order but is deliberately labeled as an AI audit; retain the manuscript TODOs until human adjudication is frozen.
|
||
|
|
|
||
|
|
## Reproduce
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python3 summarize_easycom_event_prelabels.py \
|
||
|
|
--input reports/easycom_entry_int_session1_v16/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--input reports/easycom_entry_int_session2_v16/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--input reports/easycom_entry_int_session3_v16/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--input reports/easycom_entry_int_session4_v16/easycom_eot_int_deepseek_official.jsonl \
|
||
|
|
--output-dir reports/easycom_event_ai_prelabel_stats_v16
|
||
|
|
```
|