wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revise wearer entry and yield annotation protocol

Browse Source
This commit is contained in:
Xin Wang
2026-09-15 01:25:39 +08:00
parent 026d89978f
commit 71dfd8808a
12 changed files with 478 additions and 248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/main/main.tex
+49 -42
View File
@@ -34,7 +34,7 @@ We therefore study a candidate causal local floor adapter that combines the glob
Omni state with short-window mixed audio and an anonymous set of visible-person
observations, without assuming persistent speaker identity. On EasyCom,
\todo{report the final three-seed dense and committed-event
results: EOT-response and INT-yield recall, false-positive rate, latency,
results: wearer-entry and INT-yield recall, false-positive rate, latency,
confidence intervals, and oracle-gap recovery}. Controlled
\todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
show that \todo{state the supported mechanism, without claiming addressee or
@@ -103,7 +103,7 @@ corresponding TODO gates pass:
long-context Omni information with short-window audio and anonymous
visible-person evidence.
\item We evaluate whether the adapter reduces erroneous wearer commits
during multiparty floor transfers under an EOT/INT event protocol with
during multiparty floor transfers under an entry/INT event protocol with
explicitly annotated hard-negative regions.
\end{itemize}
@@ -112,7 +112,7 @@ corresponding TODO gates pass:
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
\todo{Draw only the selected method: causal AV to the long-context Omni path;
the short-window mixed-audio and anonymous person-set path; residual fusion;
the common $10\times3$ head; and the separate EOT-response/INT-yield evaluation.
the common $10\times3$ head; and the separate wearer-entry/INT-yield evaluation.
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
\caption{Long-context Omni understanding and short-window local floor evidence
@@ -140,7 +140,7 @@ The wearer-activity forecast also provides a causal control score. When the
wearer is silent, $\hat y^{\mathrm w}_{t,1}$ acts as a near-term take-turn
score; while the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ acts as a
yield score. These scores are trained only from future activity targets.
Committed EOT-response and INT-yield events used for behavioral evaluation are
Committed wearer-entry and INT-yield events used for behavioral evaluation are
defined separately from participant-resolved annotations in
Sec.~\ref{sec:experiments}.
@@ -235,15 +235,16 @@ behavioral evaluation.
\textbf{Behavioral event annotation.}
We use floor-level wearer transitions as high-recall anchors for positive-event
annotation. Each wearer onset is examined retrospectively for a preceding
non-wearer floor release and labeled as an EOT-response only when the subsequent
wearer speech constitutes a genuine floor take. Similarly, each wearer offset
non-wearer floor release as contextual metadata and labeled as a wearer-entry
whenever the subsequent wearer speech constitutes a genuine floor take,
including re-entry while another speaker is active. Similarly, each wearer offset
is examined for a new competing non-wearer onset during the current wearer turn
and labeled as INT-yield only when the competing speaker genuinely takes over
the floor. We use permissive temporal retrieval and let an LLM jointly inspect
participant-resolved timing and transcripts to provide a semantic pre-label,
followed by mandatory human adjudication. Separately, we mine regions with no
wearer response that nevertheless constitute false-positive opportunities:
same-speaker pauses and other-to-other transitions for EOT, and backchannels or
same-speaker pauses and other-to-other transitions for entry, and backchannels or
non-takeover overlaps for INT. Only human-confirmed positives and hard negatives
enter evaluation. Appendix~\ref{app:event_annotation} gives the full protocol.
@@ -259,8 +260,8 @@ only.}
\hline
Annotation outcome & Train/dev & Test \\
\hline
\multicolumn{3}{l}{\emph{EOT-response}}\\
Positive: EOT-response & \todo{--} & \todo{--} \\
\multicolumn{3}{l}{\emph{Wearer entry}}\\
Positive: wearer entry & \todo{--} & \todo{--} \\
HN: same-speaker pause & \todo{--} & \todo{--} \\
HN: other-to-other transition & \todo{--} & \todo{--} \\
HN: wearer backchannel & \todo{--} & \todo{--} \\
@@ -271,9 +272,7 @@ HN: other backchannel & \todo{--} & \todo{--} \\
HN: non-takeover overlap & \todo{--} & \todo{--} \\
HN: wearer holds floor & \todo{--} & \todo{--} \\
\hline
Not applicable & \todo{--} & \todo{--} \\
Excluded / unscorable & \todo{--} & \todo{--} \\
Unresolved & \todo{--} & \todo{--} \\
Excluded / fallback & \todo{--} & \todo{--} \\
\textbf{Scoring-eligible total} & \todo{--} & \todo{--} \\
\hline
\end{tabular}
@@ -283,10 +282,10 @@ Participant identity is used only to construct these evaluation annotations
and is never provided to the prediction model. Provided person boxes are used
only for the causal visible-person input; transcripts, target-of-speech labels,
close-microphone audio, and future observations are unavailable to the model.
\todo{Report duration and event statistics: EOT positives/hard negatives,
\todo{Report duration and event statistics: entry positives/hard negatives,
INT positives/hard negatives, excluded candidates, and audited-region counts
for train/development and test. Freeze the event-construction and region-audit
protocol before reporting semantic EOT/INT results; until then, retain the
protocol before reporting semantic entry/INT results; until then, retain the
timing-derived proxy terminology.}
\subsection{Model Configuration and Training}
@@ -324,7 +323,7 @@ We first retain dense future-activity evaluation for direct comparison with
EgoSpeak, reporting AP at the 0.2--2.0~s horizons and their mean mAP.
Behavioral evaluation follows a committed-event protocol. For both
EOT-response and INT-yield, each gold event is matched to at most one committed
wearer-entry and INT-yield, each gold event is matched to at most one committed
prediction and is scored by recall and signed detection latency.
\textbf{False-positive scoring rule.}
@@ -335,12 +334,12 @@ situations in which the correct wearer behavior is to remain silent or retain
the floor.
We report the recall--false-positive trade-off and a common operating point at
matched recall $R=0.8$. The primary behavioral quantities are therefore EOT
matched recall $R=0.8$. The primary behavioral quantities are therefore entry
false-positive rate, INT false-positive rate, and signed latency at the frozen
operating point. We additionally report performance on the other-to-other
subset to test whether local visual evidence specifically reduces erroneous
wearer turn-taking during multiparty floor transfers. \todo{Freeze the FPR
denominators and report signed median latency separately for EOT and INT.}
denominators and report signed median latency separately for entry and INT.}
\section{Results and Analysis}
@@ -358,8 +357,8 @@ frozen before test evaluation. \todo{Freeze all baseline implementations and
verify identical targets, causal validity, and evaluation.}
\begin{table}[t]
\caption{Confirmatory EasyCom results. EOT and INT false-positive rates are
measured at matched recall $R=0.8$. E/I Lat. gives signed median EOT/INT
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates are
measured at matched recall $R=0.8$. E/I Lat. gives signed median entry/INT
latency in seconds; values closer to zero are better.}
\label{tab:main}
\centering
@@ -368,7 +367,7 @@ latency in seconds; values closer to zero are better.}
\begin{tabular}{lccccc}
\hline
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
& EOT FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I Lat.\\
& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I Lat.\\
\hline
EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
@@ -390,9 +389,9 @@ frames / specify actual observation}, indicating that local audiovisual
evidence primarily improves \todo{the interpretation supported by the data}.
\textbf{Committed behavior.}
At matched EOT-response recall $R=0.8$, the proposed adapter reduces EOT FPR
At matched wearer-entry recall $R=0.8$, the proposed adapter reduces entry FPR
from \todo{$X$} to \todo{$Y$}; at matched INT-yield recall, it changes INT FPR
from \todo{$A$} to \todo{$B$}. Its signed median EOT/INT latencies are
from \todo{$A$} to \todo{$B$}. Its signed median entry/INT latencies are
\todo{$+0.XX/+0.XX$~s}. These recall-matched error rates and signed latencies
test whether fewer erroneous commits arise from better floor decisions rather
than indefinite waiting.
@@ -450,7 +449,7 @@ Cross-family raw scores are diagnostic.}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lcccc}
\hline
Condition & Mean mAP $\uparrow$ & EOT FPR $\downarrow$
Condition & Mean mAP $\uparrow$ & Entry FPR $\downarrow$
& O$\rightarrow$O FPR $\downarrow$ & INT FPR $\downarrow$\\
\hline
\multicolumn{5}{l}{\emph{Local-evidence controls (Qwen2.5-Omni-3B)}}\\
@@ -472,7 +471,7 @@ MiniCPM-o, audited cached 5-Hz branch
\end{tabular}}
\end{table}
Removing local vision changes \todo{EOT/O$\rightarrow$O/INT FPR by $X/Y/Z$},
Removing local vision changes \todo{entry/O$\rightarrow$O/INT FPR by $X/Y/Z$},
while shifting the visual stream by \todo{$\Delta t$} produces \todo{result}.
Replacing person-local evidence with full-frame vision produces \todo{result}.
If aligned local AV improves estimates of surrounding floor occupancy, its
@@ -489,7 +488,7 @@ not describe lower false entry as success.}
\subsection{Scaling and Streaming Controls}
Increasing the frozen Omni backbone from 3B to 7B yields
\todo{$X$ points} in 0.2-s mAP and \todo{EOT/INT FPR of $Y/Z$} at matched
\todo{$X$ points} in 0.2-s mAP and \todo{entry/INT FPR of $Y/Z$} at matched
recall.
The proposed 3B adapter \todo{remains better than / approaches / is surpassed
by} the 7B scaling control.
@@ -540,7 +539,7 @@ This study asks whether a general-purpose Omni representation can be adapted
for causal floor forecasting of an invisible egocentric wearer. The candidate
method separates long-context Omni understanding from short-window local floor
evidence without assuming persistent speaker identity. \todo{State the final
learned-method improvement, matched-recall EOT/INT false-positive rates,
learned-method improvement, matched-recall entry/INT false-positive rates,
signed latency, causal-control evidence, and main limitation. If no learned
variant passes these tests, reframe the submission as an information audit or
do not make the method claim.}
@@ -566,16 +565,20 @@ non-wearer activity. Short feedback-like activity does not by itself split the
wearer episode. This grouping changes only the behavioral-event layer, not the
source intervals or dense targets.
\textbf{EOT-response.}
Starting from a floor-level wearer onset at $t_b$, we search backward for
plausible non-wearer floor releases. A release at $t_r$ is retained as an option
\textbf{Wearer entry.}
Every floor-level wearer onset at $t_b$ is proposed as a wearer-entry candidate.
We additionally search backward for plausible non-wearer floor releases. A
release at $t_r$ is retained as a contextual option
only when the interval from $t_r$ to $t_b$ contains neither intervening wearer
speech nor a competing non-wearer onset. Each wearer onset contributes at most
one review item: $t_b$ is its immutable \emph{behavior anchor}, and the item
stores every admissible \emph{opportunity-anchor} option and response delay.
Silence duration alone does not reject an option. A positive EOT annotation
must select one supplied release; neither the LLM nor the human can invent or
move a trigger timestamp. Unpaired wearer onsets remain audit controls.
Silence duration alone does not reject an option. A preceding release is not
required for a positive entry: the wearer may enter after a release, from
silence, or while another speaker remains active. When a release clearly
prompts the entry, the LLM or human may select one supplied option; neither may
invent or move a trigger timestamp. The release relation is metadata for
response-oriented analyses, not part of the positive entry label.
\textbf{INT-yield.}
Starting from an episode-ending wearer offset $t_y$, we search backward inside
@@ -594,7 +597,7 @@ the competitor genuinely takes over and the wearer yields.
\subsection{False-positive opportunity mining}
Hard-negative regions are retrieved independently of positive wearer
transitions. For EOT, a same-speaker pause followed by resumption and an
transitions. For entry, a same-speaker pause followed by resumption and an
other-to-other handover while the wearer stays silent are false-entry
opportunities. For INT, a short non-wearer backchannel or a non-takeover overlap
during which the wearer retains and continues the floor is a false-yield
@@ -606,26 +609,30 @@ labels.
The annotation assistant jointly receives immutable timestamps,
participant-resolved activity, neighboring boundaries, gap and overlap
measurements, transcripts, and target-of-speech metadata when available. It is
instructed not to equate onset with entry, offset with yield, silence with EOT,
instructed not to equate raw onset with floor entry, offset with yield, or silence
with an entry opportunity,
or overlap with interruption, and not to invent unavailable prosody, gaze, or
intent. EasyCom's \texttt{[L]} marker supplies laughter evidence only and is not
by itself a sustained floor take or takeover. A positive EOT pre-label must
also select one of the supplied opportunity-anchor options.
by itself a sustained floor take or takeover. Preceding release options are
optional metadata for a positive wearer-entry pre-label.
We use \texttt{deepseek-v4-flash} at temperature zero with a primary and a
skeptical pass. Disagreement on verdict, event type, or hard-negative subtype
produces an \textsc{Unsure} suggestion. Every candidate then receives mandatory
human adjudication: accept the complete pre-label and selected EOT trigger,
relabel it (selecting a displayed trigger for a positive EOT), or exclude an
unscorable region. Only human-confirmed \textsc{EOT-Response},
produces an \textsc{Exclude} suggestion. Every candidate then receives mandatory
human adjudication: accept the complete pre-label and optional release metadata,
relabel it, or exclude an
unscorable region. Only human-confirmed \textsc{Wearer-Entry},
\textsc{INT-Yield}, and hard-negative opportunities enter evaluation;
\textsc{Not-Applicable}, \textsc{Exclude}, and unresolved candidates do not.
\textsc{Exclude} candidates do not. Thus \textsc{Exclude} is the sole
non-scoring outcome for out-of-scope non-events (e.g., natural wearer completion
that merely coincides with feedback), insufficient evidence, unresolvable data
quality, or pre-label disagreement.
Annotation-only transcripts, identities, target-of-speech metadata, LLM output,
and future observations are never supplied to the evaluated forecasting model.
\todo{Before freezing this annotation protocol, audit randomly sampled
continuous timeline segments without candidate hints to estimate candidate
recall for EOT, INT, and each hard-negative family. Independently double-label
recall for entry, INT, and each hard-negative family. Independently double-label
a pre-registered subset, report its size and agreement statistic, freeze all
candidate parameters and prompt/model identifiers on Sessions 4--12, and
report final candidate, control, exclusion, and adjudicated-label counts.}