Add annotation workload statistics to manuscript

manuscript/main/main.tex

@@ -240,7 +240,9 @@ converted into two parallel representations. First, they are downsampled to
 5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background,
 camera-wearer, and aggregate non-wearer activity. Second, the original
 participant timeline is retained to derive adjudicated interaction events for
-behavioral evaluation.
+behavioral evaluation. We additionally apply the annotation workflow to the
+3,608 Ego4D AV validation candidates as a protocol stress test; Ego4D is not
+used to train, tune, or compare the forecasting models reported here.
 
 \textbf{Behavioral event annotation.}
 We use every observed wearer activity transition as a high-recall semantic
@@ -259,35 +261,26 @@ wearer response that nevertheless constitute false-positive opportunities:
 same-speaker pauses and other-to-other transitions for entry, and backchannels or
 non-takeover overlaps for INT. Only human-confirmed positives and hard negatives
 enter evaluation. Appendix~\ref{app:event_annotation} gives the full protocol.
+Table~\ref{tab:behavior_annotation_stats} reports the current annotation
+workload compactly; the EasyCom snapshot covers Sessions 1--4 only and is not
+the final train/test annotation inventory.
 
-\begin{table}[t]
+\begin{table}[h]
-\caption{Behavioral-event annotation statistics after human adjudication.
+\caption{LLM pre-label workload before human adjudication. Agreement is
-Counts must be filled only after the candidate protocol and annotation set are
+between two independent passes.}
-frozen. Scoring-eligible totals include confirmed positives and hard negatives
-only.}
 \label{tab:behavior_annotation_stats}
 \centering
-\small
+\scriptsize
-\begin{tabular}{lcc}
+\begin{tabular}{@{}lrr@{}}
 \hline
-Annotation outcome & Train/dev & Test \\
+Measure & EasyCom S1--4 & Ego4D AV val \\
 \hline
-\multicolumn{3}{l}{\emph{Wearer entry}}\\
+Candidates & 2,128 & 3,608 \\
-Positive: wearer entry              & \todo{--} & \todo{--} \\
+AI positive & 367 & 541 \\
-HN: same-speaker pause              & \todo{--} & \todo{--} \\
+AI hard negative & 1,017 & 1,174 \\
-HN: other-to-other transition       & \todo{--} & \todo{--} \\
+AI excluded & 744 & 1,893 \\
-HN: wearer backchannel              & \todo{--} & \todo{--} \\
+Two-pass agreement & 93.4\% & 82.1\% \\
-\hline
+Human-confirmed & 0 & 0 \\
-\multicolumn{3}{l}{\emph{INT-yield}}\\
-Positive: INT-yield                 & \todo{--} & \todo{--} \\
-HN: other backchannel               & \todo{--} & \todo{--} \\
-HN: non-takeover overlap            & \todo{--} & \todo{--} \\
-HN: wearer holds floor              & \todo{--} & \todo{--} \\
-\hline
-Excluded: valid but out of scope    & \todo{--} & \todo{--} \\
-Excluded: insufficient evidence     & \todo{--} & \todo{--} \\
-Excluded: data quality              & \todo{--} & \todo{--} \\
-\textbf{Scoring-eligible total}     & \todo{--} & \todo{--} \\
 \hline
 \end{tabular}
 \end{table}
@@ -658,9 +651,11 @@ intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not
 by itself a sustained floor take or takeover. Preceding release options are
 optional metadata for a positive wearer-entry pre-label.
 
-We use \texttt{deepseek-v4-flash} at temperature zero with a primary and a
+We use temperature-zero DeepSeek pre-labeling with a primary and a skeptical
-skeptical pass. Disagreement on verdict, event type, or hard-negative subtype
+pass (\texttt{deepseek-flash} for the EasyCom snapshot and
-produces an \textsc{Exclude} suggestion. Every candidate then receives mandatory
+\texttt{deepseek-v4-flash} for the Ego4D AV snapshot). Disagreement on verdict,
+event type, or hard-negative subtype produces an \textsc{Exclude} suggestion.
+Every candidate then receives mandatory
 human adjudication: accept the complete pre-label and optional release metadata,
 relabel it, or exclude an
 unscorable region. Only human-confirmed \textsc{Wearer-Entry},
@@ -674,6 +669,49 @@ to insufficient evidence and is never counted as corrupt data.
 Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
 and future observations are never supplied to the evaluated forecasting model.
 
+% Snapshot source: reports/easycom_event_ai_prelabel_stats_v16/summary.json
+% and reports/ego4d_av_event_candidates_2026-09-15/ego4d_av_val_candidates_ai.jsonl.
+\begin{table}[t]
+\caption{Descriptive LLM pre-label audit before human adjudication. Counts in
+parentheses are percentages of all candidates in that column; agreement uses
+all candidates as its denominator. These values describe annotation workload,
+not ground truth or model performance.}
+\label{tab:llm_prelabel_audit}
+\centering
+\scriptsize
+\begin{tabular}{@{}lrr@{}}
+\hline
+Pre-label outcome & EasyCom S1--4 & Ego4D AV val \\
+\hline
+All candidates                    & 2,128         & 3,608 \\
+Positive: wearer entry            & 324 (15.2\%)  & 503 (13.9\%) \\
+Positive: INT-yield               & 43 (2.0\%)    & 38 (1.1\%) \\
+HN: entry family                  & 892 (41.9\%)  & 940 (26.1\%) \\
+HN: INT family                    & 125 (5.9\%)   & 234 (6.5\%) \\
+Excluded: valid/out of scope      & 602 (28.3\%)  & 775 (21.5\%) \\
+Excluded: insufficient evidence   & 142 (6.7\%)   & 1,098 (30.4\%) \\
+Excluded: data quality            & 0 (0.0\%)     & 20 (0.6\%) \\
+Two-pass semantic agreement       & 1,987 (93.4\%)& 2,961 (82.1\%) \\
+Human-confirmed                   & 0              & 0 \\
+\hline
+\end{tabular}
+\end{table}
+
+Table~\ref{tab:llm_prelabel_audit} expands the unreviewed AI summary in
+Table~\ref{tab:behavior_annotation_stats}; neither table reports final human
+annotation results.
+The EasyCom column covers only Sessions 1--4, while the Ego4D AV column covers
+the 3,608 validation candidates; the 32,768 Ego4D AV training candidates have
+not yet received LLM pre-labels. All 5,736 reported rows remain unreviewed.
+Because the candidate sets deliberately include mined hard-negative regions,
+their class proportions are not estimates of natural event prevalence. The
+higher-level counts are descriptive only and were not used for forecasting
+threshold or model selection. The
+higher Ego4D insufficient-evidence rate (30.4\% versus 6.7\%) and lower
+two-pass agreement (82.1\% versus 93.4\%) identify a larger human-review burden;
+they do not establish a cross-dataset accuracy difference without adjudicated
+reference labels.
+
 \todo{Before freezing this annotation protocol, audit randomly sampled
 continuous timeline segments without candidate hints to estimate candidate
 recall for entry, INT, and each hard-negative family. Independently double-label