|
|
|
@@ -107,117 +107,125 @@ corresponding TODO gates pass:
|
|
|
|
|
positives and explicitly annotated hard-negative opportunity regions.
|
|
|
|
|
\end{itemize}
|
|
|
|
|
|
|
|
|
|
\begin{figure}[t]
|
|
|
|
|
\begin{figure*}[t]
|
|
|
|
|
\centering
|
|
|
|
|
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
|
|
|
|
|
\todo{Draw only the selected method: causal AV to the long-context Omni path;
|
|
|
|
|
the short-window mixed-audio and anonymous person-set path; residual fusion;
|
|
|
|
|
the common $10\times3$ head; and the separate wearer-entry/INT-yield evaluation.
|
|
|
|
|
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
|
|
|
|
|
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
|
|
|
|
|
\caption{Long-context Omni understanding and short-window local floor evidence
|
|
|
|
|
for wearer-relative future-activity prediction.}
|
|
|
|
|
\includegraphics[width=0.98\textwidth]{figures/method_architecture.pdf}
|
|
|
|
|
\caption{Our strictly causal global--local architecture (top) and streaming
|
|
|
|
|
alignment (bottom). At every 5-Hz cutoff $t_n$, the global state includes the
|
|
|
|
|
latest partial 2-s Omni block, while every fifth causal 25-Hz local state
|
|
|
|
|
provides shared audio and an anonymous visible-person set. Masked person
|
|
|
|
|
attention contributes only through a zero-initialized residual to the matched
|
|
|
|
|
$G+U$ baseline.}
|
|
|
|
|
\label{fig:method}
|
|
|
|
|
\end{figure}
|
|
|
|
|
\end{figure*}
|
|
|
|
|
|
|
|
|
|
\section{Method}
|
|
|
|
|
\label{sec:method}
|
|
|
|
|
|
|
|
|
|
\subsection{Causal wearer-relative projection}
|
|
|
|
|
\subsection{Model Architecture}
|
|
|
|
|
\label{sec:model_arch}
|
|
|
|
|
|
|
|
|
|
At decision time $t$, the model observes only the stream
|
|
|
|
|
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. For
|
|
|
|
|
bin $k$, the multilabel target
|
|
|
|
|
At decision time $t$, the model observes only the audiovisual stream
|
|
|
|
|
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins spaced by
|
|
|
|
|
$\Delta=0.2$~s. For horizon $k$, the multilabel target
|
|
|
|
|
$\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
|
|
|
|
|
background, camera-wearer activity, and activity by one or more other people.
|
|
|
|
|
Wearer--other overlap is allowed. The model outputs
|
|
|
|
|
background, camera-wearer activity, and activity by one or more other people;
|
|
|
|
|
wearer--other overlap is allowed. The output is
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t})
|
|
|
|
|
\in[0,1]^{K\times3}.
|
|
|
|
|
\label{eq:task}
|
|
|
|
|
\end{equation}
|
|
|
|
|
The wearer-activity forecast also provides causal behavioral scores at every
|
|
|
|
|
horizon. At decision time $t$, if the wearer is silent,
|
|
|
|
|
$\hat y^{\mathrm w}_{t,k}$ acts as a take-turn score for future bin $k$; if
|
|
|
|
|
the wearer is active, $1-\hat y^{\mathrm w}_{t,k}$ acts as a yield score.
|
|
|
|
|
These scores are trained only from future activity targets and are not event
|
|
|
|
|
hazards or committed controller actions.
|
|
|
|
|
Adjudicated wearer-entry and INT-yield events used for behavioral evaluation are
|
|
|
|
|
defined separately from participant-resolved annotations in
|
|
|
|
|
Sec.~\ref{sec:experiments}.
|
|
|
|
|
|
|
|
|
|
\subsection{Global Omni baseline}
|
|
|
|
|
|
|
|
|
|
Let $\vect g_t\in\mathbb R^{d_g}$ be the causal global readout of
|
|
|
|
|
Qwen2.5-Omni at the physical cutoff $t$ \cite{xu2025qwenomni}. The frozen
|
|
|
|
|
baseline trains only a dimension-matched prediction head. A matched PEFT/LoRA
|
|
|
|
|
baseline updates a pre-registered subset of the Omni model under the same
|
|
|
|
|
examples, context, optimizer budget, head, and evaluator. Its purpose is to
|
|
|
|
|
test whether generic parameter adaptation is sufficient; one negative recipe
|
|
|
|
|
is not interpreted as a general limitation of LoRA or all Omni models.
|
|
|
|
|
|
|
|
|
|
\todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates,
|
|
|
|
|
streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank,
|
|
|
|
|
trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a
|
|
|
|
|
capacity-scaling control. MiniCPM-o 4.5 is restricted to the diagnostic
|
|
|
|
|
streaming/cache ablation below and does not enter the core method or support a
|
|
|
|
|
cross-family superiority claim.}
|
|
|
|
|
|
|
|
|
|
\subsection{Causal local floor adapter}
|
|
|
|
|
|
|
|
|
|
The candidate local path operates on the most recent $L_\ell$ seconds. At local instant
|
|
|
|
|
$\tau\leq t$, a causal audio encoder produces $\vect a_\tau$ from mixed audio.
|
|
|
|
|
For the anonymous visible-person set
|
|
|
|
|
$\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
|
|
|
|
|
a shared visual encoder and masked set operator produce
|
|
|
|
|
Our architecture complements globally integrated Omni context with explicitly
|
|
|
|
|
accessible local floor evidence (Fig.~\ref{fig:method}). Let
|
|
|
|
|
$\vect g_t\in\mathbb R^{d_g}$ denote the causal Qwen2.5-Omni Thinker readout
|
|
|
|
|
at $t$ \cite{xu2025qwenomni}. In parallel, the frozen MuVAP ASD frontend
|
|
|
|
|
\cite{qi2026muvap} produces one shared mixed-audio CPC state $\vect u_t$ and a
|
|
|
|
|
variable-size set of visible-person states
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\vect v_\tau=\mathrm{SetPool}
|
|
|
|
|
\left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}
|
|
|
|
|
\cup\{\vect v_\varnothing\}\right),
|
|
|
|
|
\label{eq:set}
|
|
|
|
|
\mathcal P_t=\{\vect p_{t,i}\}_{i=1}^{N_t},\qquad
|
|
|
|
|
\vect p_{t,i}=[\vect v_{t,i};s_{t,i}],
|
|
|
|
|
\label{eq:person_state}
|
|
|
|
|
\end{equation}
|
|
|
|
|
where $\vect v_\varnothing$ handles an empty or offscreen visible set. The
|
|
|
|
|
operator is permutation invariant and uses no participant-ID embedding or
|
|
|
|
|
cross-time identity. A causal short-window encoder summarizes local dynamics,
|
|
|
|
|
where $\vect v_{t,i}$ is a causal visual state and $s_{t,i}$ is one
|
|
|
|
|
audio--visual speaking logit. The shared audio state is not duplicated in
|
|
|
|
|
each person token. The prediction model receives neither participant IDs nor
|
|
|
|
|
cross-time track identity; provided causal face boxes are used only to isolate
|
|
|
|
|
the representation and fusion question.
|
|
|
|
|
|
|
|
|
|
A matched global--audio head first maps $[\vect g_t;\vect u_t]$ to baseline
|
|
|
|
|
logits $\vect Z_t^{\mathrm{base}}\in\mathbb R^{K\times3}$ and an intermediate
|
|
|
|
|
state $\vect h_t$. The same inputs query the current person set:
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\vect \ell_t=F_\ell
|
|
|
|
|
\left(\{[\vect a_\tau;\vect v_\tau]\}_{\tau\in(t-L_\ell,t]}\right).
|
|
|
|
|
\label{eq:local}
|
|
|
|
|
\begin{aligned}
|
|
|
|
|
\vect q_t &= W_g\vect g_t+W_a\vect u_t,\\
|
|
|
|
|
\mathcal M_t &= \{W_p\vect p_{t,i}\}_{i=1}^{N_t}
|
|
|
|
|
\cup\{\vect p_\varnothing\},\\
|
|
|
|
|
\vect c_t &= \mathrm{MHA}(\vect q_t,\mathcal M_t,\mathcal M_t),
|
|
|
|
|
\end{aligned}
|
|
|
|
|
\label{eq:person_attention}
|
|
|
|
|
\end{equation}
|
|
|
|
|
The adapter preserves the global initialization through a zero-initialized
|
|
|
|
|
residual output,
|
|
|
|
|
where masking supports variable cardinality and the learned null token
|
|
|
|
|
$\vect p_\varnothing$ makes the attention well defined for an empty set. If
|
|
|
|
|
$m_t$ indicates that at least one valid person observation exists, the person
|
|
|
|
|
branch supplies only a residual logit correction,
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\vect z_t=\vect g_t+W_o\,
|
|
|
|
|
\sigma\!\left(W_g\vect g_t+W_\ell\vect\ell_t\right),
|
|
|
|
|
\qquad W_o\vert_{\mathrm{init}}=0,
|
|
|
|
|
\label{eq:fusion}
|
|
|
|
|
\vect Z_t=\vect Z_t^{\mathrm{base}}
|
|
|
|
|
{}+m_t\,\mathrm{reshape}\!\left(W_r[\vect h_t;\vect c_t;m_t]\right),
|
|
|
|
|
\qquad W_r\vert_{\mathrm{init}}=0,
|
|
|
|
|
\label{eq:residual}
|
|
|
|
|
\end{equation}
|
|
|
|
|
and the common head maps $\vect z_t$ to Eq.~\eqref{eq:task}. This implements an
|
|
|
|
|
evidence-access hypothesis rather than adding an unconstrained second model:
|
|
|
|
|
the global path retains broad context, while the local path preserves timely
|
|
|
|
|
floor evidence.
|
|
|
|
|
and $\hat{\vect Y}_t=\mathrm{softmax}_c(\vect Z_t)$. Thus, after copying and
|
|
|
|
|
freezing the trained $G+U$ head, the complete model initially reproduces that
|
|
|
|
|
baseline exactly; only the person projection, query, masked attention, and
|
|
|
|
|
residual readout are trained in the adaptation stage.
|
|
|
|
|
|
|
|
|
|
\todo{Freeze $L_\ell$, local audio/video rates, crop source and resolution,
|
|
|
|
|
set pooling, causal temporal encoder, masks, reset rules, parameter count, and
|
|
|
|
|
latency. If the local path runs faster than the 5-Hz global/output grid, compare
|
|
|
|
|
it with a same-parameter 5-Hz variant before using the term ``dual-rate.'' Use
|
|
|
|
|
provided causal boxes only as a clearly labeled method-isolation condition;
|
|
|
|
|
online detection/tracking is not a paper contribution unless actually run.}
|
|
|
|
|
|
|
|
|
|
\subsection{Training}
|
|
|
|
|
|
|
|
|
|
The primary objective is class-balanced binary cross entropy over valid
|
|
|
|
|
horizons and activity channels,
|
|
|
|
|
We use the EgoSpeak/LSTR multi-positive cross-entropy over causal-valid rows,
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\mathcal L_{\mathrm{act}}=
|
|
|
|
|
\sum_{t,k,c}m_{t,k}w_c\,
|
|
|
|
|
\mathrm{BCE}(\hat y^c_{t,k},y^c_{t,k}).
|
|
|
|
|
\begin{aligned}
|
|
|
|
|
\mathcal L&=-\frac{1}{M}\sum_{t,k,c}m_{t,k}y^c_{t,k}
|
|
|
|
|
\log\mathrm{softmax}_c(\vect Z_{t,k}),\\
|
|
|
|
|
M&=\sum_{t,k}m_{t,k}.
|
|
|
|
|
\end{aligned}
|
|
|
|
|
\label{eq:loss}
|
|
|
|
|
\end{equation}
|
|
|
|
|
No transcript, target-of-speech label, close microphone, future frame, or
|
|
|
|
|
per-person speech label enters the deployable model. \todo{Add an active-speaker
|
|
|
|
|
or event auxiliary loss only if it improves the frozen primary protocol and
|
|
|
|
|
its label/input boundary is fully reported.}
|
|
|
|
|
For an overlap target, both active channels contribute a log-probability term.
|
|
|
|
|
No transcript, target-of-speech annotation, close-microphone signal, speech
|
|
|
|
|
label, or future observation is supplied to the forecasting model. The
|
|
|
|
|
wearer forecast is interpreted only at evaluation time: it is an entry proxy
|
|
|
|
|
when the wearer is silent and its complement is a yield proxy when the wearer
|
|
|
|
|
is active, not a trained event hazard or committed controller action.
|
|
|
|
|
|
|
|
|
|
\subsection{Causal Streaming Architecture}
|
|
|
|
|
\label{sec:causal_streaming}
|
|
|
|
|
|
|
|
|
|
We distinguish encoder processing granularity from the common forecasting
|
|
|
|
|
rate. Predictions are emitted at $t_n=n\Delta$, $n\geq1$, while every
|
|
|
|
|
representation used at $t_n$ is restricted to media available by that physical
|
|
|
|
|
cutoff.
|
|
|
|
|
|
|
|
|
|
The global path uses an 8-s rolling causal presentation budget with
|
|
|
|
|
source-aligned 2-s audiovisual blocks. Completed blocks are cached and
|
|
|
|
|
replayed, while the currently incomplete block is a temporary tail. At every
|
|
|
|
|
200-ms cutoff, the tail is rebuilt with all newly observed audio and video and
|
|
|
|
|
included in computing $\vect g_{t_n}$; when it reaches 2~s, it is committed to
|
|
|
|
|
the reusable history. Consequently, the Omni state is not held constant for
|
|
|
|
|
2~s: it includes the latest available content at every decision cutoff without
|
|
|
|
|
observing media after $t_n$. We retain the final four Thinker query-layer
|
|
|
|
|
states and fuse them in the matched head with a concatenation MLP.
|
|
|
|
|
|
|
|
|
|
The local frontend operates causally at 25~Hz. State $j$ consumes the
|
|
|
|
|
half-open interval $[j/25,(j+1)/25)$ and is timestamped at its right edge.
|
|
|
|
|
Mixed audio is encoded once as a continuous shared CPC stream, while each
|
|
|
|
|
available face crop is processed by the causal visual and audio--visual
|
|
|
|
|
speaking modules. Ten-second extraction chunks may prepend up to 2~s of past
|
|
|
|
|
context, never future context. At a 5-Hz forecasting cutoff, we select the
|
|
|
|
|
local state whose right edge equals $t_n$; hence every fifth 25-Hz state aligns
|
|
|
|
|
with one Omni prediction row. The two paths therefore describe the same
|
|
|
|
|
causal prefix: the Omni path preserves integrated conversational context,
|
|
|
|
|
whereas the local path exposes fine-grained audio and visible-person floor
|
|
|
|
|
evidence. Fusion and prediction occur only on the common 5-Hz grid.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Experiments}
|
|
|
|
@@ -232,7 +240,9 @@ converted into two parallel representations. First, they are downsampled to
|
|
|
|
|
5~Hz to construct the EgoSpeak-compatible ten-horizon targets for background,
|
|
|
|
|
camera-wearer, and aggregate non-wearer activity. Second, the original
|
|
|
|
|
participant timeline is retained to derive adjudicated interaction events for
|
|
|
|
|
behavioral evaluation.
|
|
|
|
|
behavioral evaluation. We additionally apply the annotation workflow to the
|
|
|
|
|
3,608 Ego4D AV validation candidates as a protocol stress test; Ego4D is not
|
|
|
|
|
used to train, tune, or compare the forecasting models reported here.
|
|
|
|
|
|
|
|
|
|
\textbf{Behavioral event annotation.}
|
|
|
|
|
We use every observed wearer activity transition as a high-recall semantic
|
|
|
|
@@ -251,35 +261,26 @@ wearer response that nevertheless constitute false-positive opportunities:
|
|
|
|
|
same-speaker pauses and other-to-other transitions for entry, and backchannels or
|
|
|
|
|
non-takeover overlaps for INT. Only human-confirmed positives and hard negatives
|
|
|
|
|
enter evaluation. Appendix~\ref{app:event_annotation} gives the full protocol.
|
|
|
|
|
Table~\ref{tab:behavior_annotation_stats} reports the current annotation
|
|
|
|
|
workload compactly; the EasyCom snapshot covers Sessions 1--4 only and is not
|
|
|
|
|
the final train/test annotation inventory.
|
|
|
|
|
|
|
|
|
|
\begin{table}[t]
|
|
|
|
|
\caption{Behavioral-event annotation statistics after human adjudication.
|
|
|
|
|
Counts must be filled only after the candidate protocol and annotation set are
|
|
|
|
|
frozen. Scoring-eligible totals include confirmed positives and hard negatives
|
|
|
|
|
only.}
|
|
|
|
|
\begin{table}[h]
|
|
|
|
|
\caption{LLM pre-label workload before human adjudication. Agreement is
|
|
|
|
|
between two independent passes.}
|
|
|
|
|
\label{tab:behavior_annotation_stats}
|
|
|
|
|
\centering
|
|
|
|
|
\small
|
|
|
|
|
\begin{tabular}{lcc}
|
|
|
|
|
\scriptsize
|
|
|
|
|
\begin{tabular}{@{}lrr@{}}
|
|
|
|
|
\hline
|
|
|
|
|
Annotation outcome & Train/dev & Test \\
|
|
|
|
|
Measure & EasyCom S1--4 & Ego4D AV val \\
|
|
|
|
|
\hline
|
|
|
|
|
\multicolumn{3}{l}{\emph{Wearer entry}}\\
|
|
|
|
|
Positive: wearer entry & \todo{--} & \todo{--} \\
|
|
|
|
|
HN: same-speaker pause & \todo{--} & \todo{--} \\
|
|
|
|
|
HN: other-to-other transition & \todo{--} & \todo{--} \\
|
|
|
|
|
HN: wearer backchannel & \todo{--} & \todo{--} \\
|
|
|
|
|
\hline
|
|
|
|
|
\multicolumn{3}{l}{\emph{INT-yield}}\\
|
|
|
|
|
Positive: INT-yield & \todo{--} & \todo{--} \\
|
|
|
|
|
HN: other backchannel & \todo{--} & \todo{--} \\
|
|
|
|
|
HN: non-takeover overlap & \todo{--} & \todo{--} \\
|
|
|
|
|
HN: wearer holds floor & \todo{--} & \todo{--} \\
|
|
|
|
|
\hline
|
|
|
|
|
Excluded: valid but out of scope & \todo{--} & \todo{--} \\
|
|
|
|
|
Excluded: insufficient evidence & \todo{--} & \todo{--} \\
|
|
|
|
|
Excluded: data quality & \todo{--} & \todo{--} \\
|
|
|
|
|
\textbf{Scoring-eligible total} & \todo{--} & \todo{--} \\
|
|
|
|
|
Candidates & 2,128 & 3,608 \\
|
|
|
|
|
AI positive & 367 & 541 \\
|
|
|
|
|
AI hard negative & 1,017 & 1,174 \\
|
|
|
|
|
AI excluded & 744 & 1,893 \\
|
|
|
|
|
Two-pass agreement & 93.4\% & 82.1\% \\
|
|
|
|
|
Human-confirmed & 0 & 0 \\
|
|
|
|
|
\hline
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\end{table}
|
|
|
|
@@ -301,8 +302,8 @@ matched PEFT/LoRA adaptation, global features with recent local audio, a
|
|
|
|
|
parameter- and history-matched temporal local branch, and the proposed causal
|
|
|
|
|
local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling
|
|
|
|
|
control. All learned variants use identical causal examples, prediction
|
|
|
|
|
horizons, prediction heads, and the class-balanced binary cross-entropy
|
|
|
|
|
objective in Eq.~\eqref{eq:loss}.
|
|
|
|
|
horizons, prediction heads, and the multi-positive cross-entropy objective in
|
|
|
|
|
Eq.~\eqref{eq:loss}.
|
|
|
|
|
|
|
|
|
|
The diagnostic streaming ablation additionally uses MiniCPM-o 4.5
|
|
|
|
|
\cite{cui2026minicpmo45}. Its native cached stream is evaluated at its native
|
|
|
|
@@ -658,9 +659,11 @@ intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not
|
|
|
|
|
by itself a sustained floor take or takeover. Preceding release options are
|
|
|
|
|
optional metadata for a positive wearer-entry pre-label.
|
|
|
|
|
|
|
|
|
|
We use \texttt{deepseek-v4-flash} at temperature zero with a primary and a
|
|
|
|
|
skeptical pass. Disagreement on verdict, event type, or hard-negative subtype
|
|
|
|
|
produces an \textsc{Exclude} suggestion. Every candidate then receives mandatory
|
|
|
|
|
We use temperature-zero DeepSeek pre-labeling with a primary and a skeptical
|
|
|
|
|
pass (\texttt{deepseek-flash} for the EasyCom snapshot and
|
|
|
|
|
\texttt{deepseek-v4-flash} for the Ego4D AV snapshot). Disagreement on verdict,
|
|
|
|
|
event type, or hard-negative subtype produces an \textsc{Exclude} suggestion.
|
|
|
|
|
Every candidate then receives mandatory
|
|
|
|
|
human adjudication: accept the complete pre-label and optional release metadata,
|
|
|
|
|
relabel it, or exclude an
|
|
|
|
|
unscorable region. Only human-confirmed \textsc{Wearer-Entry},
|
|
|
|
@@ -674,6 +677,49 @@ to insufficient evidence and is never counted as corrupt data.
|
|
|
|
|
Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
|
|
|
|
|
and future observations are never supplied to the evaluated forecasting model.
|
|
|
|
|
|
|
|
|
|
% Snapshot source: reports/easycom_event_ai_prelabel_stats_v16/summary.json
|
|
|
|
|
% and reports/ego4d_av_event_candidates_2026-09-15/ego4d_av_val_candidates_ai.jsonl.
|
|
|
|
|
\begin{table}[t]
|
|
|
|
|
\caption{Descriptive LLM pre-label audit before human adjudication. Counts in
|
|
|
|
|
parentheses are percentages of all candidates in that column; agreement uses
|
|
|
|
|
all candidates as its denominator. These values describe annotation workload,
|
|
|
|
|
not ground truth or model performance.}
|
|
|
|
|
\label{tab:llm_prelabel_audit}
|
|
|
|
|
\centering
|
|
|
|
|
\scriptsize
|
|
|
|
|
\begin{tabular}{@{}lrr@{}}
|
|
|
|
|
\hline
|
|
|
|
|
Pre-label outcome & EasyCom S1--4 & Ego4D AV val \\
|
|
|
|
|
\hline
|
|
|
|
|
All candidates & 2,128 & 3,608 \\
|
|
|
|
|
Positive: wearer entry & 324 (15.2\%) & 503 (13.9\%) \\
|
|
|
|
|
Positive: INT-yield & 43 (2.0\%) & 38 (1.1\%) \\
|
|
|
|
|
HN: entry family & 892 (41.9\%) & 940 (26.1\%) \\
|
|
|
|
|
HN: INT family & 125 (5.9\%) & 234 (6.5\%) \\
|
|
|
|
|
Excluded: valid/out of scope & 602 (28.3\%) & 775 (21.5\%) \\
|
|
|
|
|
Excluded: insufficient evidence & 142 (6.7\%) & 1,098 (30.4\%) \\
|
|
|
|
|
Excluded: data quality & 0 (0.0\%) & 20 (0.6\%) \\
|
|
|
|
|
Two-pass semantic agreement & 1,987 (93.4\%)& 2,961 (82.1\%) \\
|
|
|
|
|
Human-confirmed & 0 & 0 \\
|
|
|
|
|
\hline
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\end{table}
|
|
|
|
|
|
|
|
|
|
Table~\ref{tab:llm_prelabel_audit} expands the unreviewed AI summary in
|
|
|
|
|
Table~\ref{tab:behavior_annotation_stats}; neither table reports final human
|
|
|
|
|
annotation results.
|
|
|
|
|
The EasyCom column covers only Sessions 1--4, while the Ego4D AV column covers
|
|
|
|
|
the 3,608 validation candidates; the 32,768 Ego4D AV training candidates have
|
|
|
|
|
not yet received LLM pre-labels. All 5,736 reported rows remain unreviewed.
|
|
|
|
|
Because the candidate sets deliberately include mined hard-negative regions,
|
|
|
|
|
their class proportions are not estimates of natural event prevalence. The
|
|
|
|
|
higher-level counts are descriptive only and were not used for forecasting
|
|
|
|
|
threshold or model selection. The
|
|
|
|
|
higher Ego4D insufficient-evidence rate (30.4\% versus 6.7\%) and lower
|
|
|
|
|
two-pass agreement (82.1\% versus 93.4\%) identify a larger human-review burden;
|
|
|
|
|
they do not establish a cross-dataset accuracy difference without adjudicated
|
|
|
|
|
reference labels.
|
|
|
|
|
|
|
|
|
|
\todo{Before freezing this annotation protocol, audit randomly sampled
|
|
|
|
|
continuous timeline segments without candidate hints to estimate candidate
|
|
|
|
|
recall for entry, INT, and each hard-negative family. Independently double-label
|
|
|
|
|
Xin Wang