Add annotation workload statistics to manuscript
This commit is contained in:
Xin Wang
+66
-28
@@ -240,7 +240,9 @@ converted into two parallel representations. First, they are downsampled to
|
||||
5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background,
|
||||
camera-wearer, and aggregate non-wearer activity. Second, the original
|
||||
participant timeline is retained to derive adjudicated interaction events for
|
||||
behavioral evaluation.
|
||||
behavioral evaluation. We additionally apply the annotation workflow to the
|
||||
3,608 Ego4D AV validation candidates as a protocol stress test; Ego4D is not
|
||||
used to train, tune, or compare the forecasting models reported here.
|
||||
|
||||
\textbf{Behavioral event annotation.}
|
||||
We use every observed wearer activity transition as a high-recall semantic
|
||||
@@ -259,35 +261,26 @@ wearer response that nevertheless constitute false-positive opportunities:
|
||||
same-speaker pauses and other-to-other transitions for entry, and backchannels or
|
||||
non-takeover overlaps for INT. Only human-confirmed positives and hard negatives
|
||||
enter evaluation. Appendix~\ref{app:event_annotation} gives the full protocol.
|
||||
Table~\ref{tab:behavior_annotation_stats} reports the current annotation
|
||||
workload compactly; the EasyCom snapshot covers Sessions 1--4 only and is not
|
||||
the final train/test annotation inventory.
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Behavioral-event annotation statistics after human adjudication.
|
||||
Counts must be filled only after the candidate protocol and annotation set are
|
||||
frozen. Scoring-eligible totals include confirmed positives and hard negatives
|
||||
only.}
|
||||
\begin{table}[h]
|
||||
\caption{LLM pre-label workload before human adjudication. Agreement is
|
||||
between two independent passes.}
|
||||
\label{tab:behavior_annotation_stats}
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{lcc}
|
||||
\scriptsize
|
||||
\begin{tabular}{@{}lrr@{}}
|
||||
\hline
|
||||
Annotation outcome & Train/dev & Test \\
|
||||
Measure & EasyCom S1--4 & Ego4D AV val \\
|
||||
\hline
|
||||
\multicolumn{3}{l}{\emph{Wearer entry}}\\
|
||||
Positive: wearer entry & \todo{--} & \todo{--} \\
|
||||
HN: same-speaker pause & \todo{--} & \todo{--} \\
|
||||
HN: other-to-other transition & \todo{--} & \todo{--} \\
|
||||
HN: wearer backchannel & \todo{--} & \todo{--} \\
|
||||
\hline
|
||||
\multicolumn{3}{l}{\emph{INT-yield}}\\
|
||||
Positive: INT-yield & \todo{--} & \todo{--} \\
|
||||
HN: other backchannel & \todo{--} & \todo{--} \\
|
||||
HN: non-takeover overlap & \todo{--} & \todo{--} \\
|
||||
HN: wearer holds floor & \todo{--} & \todo{--} \\
|
||||
\hline
|
||||
Excluded: valid but out of scope & \todo{--} & \todo{--} \\
|
||||
Excluded: insufficient evidence & \todo{--} & \todo{--} \\
|
||||
Excluded: data quality & \todo{--} & \todo{--} \\
|
||||
\textbf{Scoring-eligible total} & \todo{--} & \todo{--} \\
|
||||
Candidates & 2,128 & 3,608 \\
|
||||
AI positive & 367 & 541 \\
|
||||
AI hard negative & 1,017 & 1,174 \\
|
||||
AI excluded & 744 & 1,893 \\
|
||||
Two-pass agreement & 93.4\% & 82.1\% \\
|
||||
Human-confirmed & 0 & 0 \\
|
||||
\hline
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
@@ -658,9 +651,11 @@ intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not
|
||||
by itself a sustained floor take or takeover. Preceding release options are
|
||||
optional metadata for a positive wearer-entry pre-label.
|
||||
|
||||
We use \texttt{deepseek-v4-flash} at temperature zero with a primary and a
|
||||
skeptical pass. Disagreement on verdict, event type, or hard-negative subtype
|
||||
produces an \textsc{Exclude} suggestion. Every candidate then receives mandatory
|
||||
We use temperature-zero DeepSeek pre-labeling with a primary and a skeptical
|
||||
pass (\texttt{deepseek-flash} for the EasyCom snapshot and
|
||||
\texttt{deepseek-v4-flash} for the Ego4D AV snapshot). Disagreement on verdict,
|
||||
event type, or hard-negative subtype produces an \textsc{Exclude} suggestion.
|
||||
Every candidate then receives mandatory
|
||||
human adjudication: accept the complete pre-label and optional release metadata,
|
||||
relabel it, or exclude an
|
||||
unscorable region. Only human-confirmed \textsc{Wearer-Entry},
|
||||
@@ -674,6 +669,49 @@ to insufficient evidence and is never counted as corrupt data.
|
||||
Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
|
||||
and future observations are never supplied to the evaluated forecasting model.
|
||||
|
||||
% Snapshot source: reports/easycom_event_ai_prelabel_stats_v16/summary.json
|
||||
% and reports/ego4d_av_event_candidates_2026-09-15/ego4d_av_val_candidates_ai.jsonl.
|
||||
\begin{table}[t]
|
||||
\caption{Descriptive LLM pre-label audit before human adjudication. Counts in
|
||||
parentheses are percentages of all candidates in that column; agreement uses
|
||||
all candidates as its denominator. These values describe annotation workload,
|
||||
not ground truth or model performance.}
|
||||
\label{tab:llm_prelabel_audit}
|
||||
\centering
|
||||
\scriptsize
|
||||
\begin{tabular}{@{}lrr@{}}
|
||||
\hline
|
||||
Pre-label outcome & EasyCom S1--4 & Ego4D AV val \\
|
||||
\hline
|
||||
All candidates & 2,128 & 3,608 \\
|
||||
Positive: wearer entry & 324 (15.2\%) & 503 (13.9\%) \\
|
||||
Positive: INT-yield & 43 (2.0\%) & 38 (1.1\%) \\
|
||||
HN: entry family & 892 (41.9\%) & 940 (26.1\%) \\
|
||||
HN: INT family & 125 (5.9\%) & 234 (6.5\%) \\
|
||||
Excluded: valid/out of scope & 602 (28.3\%) & 775 (21.5\%) \\
|
||||
Excluded: insufficient evidence & 142 (6.7\%) & 1,098 (30.4\%) \\
|
||||
Excluded: data quality & 0 (0.0\%) & 20 (0.6\%) \\
|
||||
Two-pass semantic agreement & 1,987 (93.4\%)& 2,961 (82.1\%) \\
|
||||
Human-confirmed & 0 & 0 \\
|
||||
\hline
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Table~\ref{tab:llm_prelabel_audit} expands the unreviewed AI summary in
|
||||
Table~\ref{tab:behavior_annotation_stats}; neither table reports final human
|
||||
annotation results.
|
||||
The EasyCom column covers only Sessions 1--4, while the Ego4D AV column covers
|
||||
the 3,608 validation candidates; the 32,768 Ego4D AV training candidates have
|
||||
not yet received LLM pre-labels. All 5,736 reported rows remain unreviewed.
|
||||
Because the candidate sets deliberately include mined hard-negative regions,
|
||||
their class proportions are not estimates of natural event prevalence. The
|
||||
higher-level counts are descriptive only and were not used for forecasting
|
||||
threshold or model selection. The
|
||||
higher Ego4D insufficient-evidence rate (30.4\% versus 6.7\%) and lower
|
||||
two-pass agreement (82.1\% versus 93.4\%) identify a larger human-review burden;
|
||||
they do not establish a cross-dataset accuracy difference without adjudicated
|
||||
reference labels.
|
||||
|
||||
\todo{Before freezing this annotation protocol, audit randomly sampled
|
||||
continuous timeline segments without candidate hints to estimate candidate
|
||||
recall for entry, INT, and each hard-negative family. Independently double-label
|
||||
|
||||
Reference in New Issue
Block a user