revise manuscript method and add architecture figure
This commit is contained in:
Xin Wang
+97
-89
@@ -107,117 +107,125 @@ corresponding TODO gates pass:
|
||||
positives and explicitly annotated hard-negative opportunity regions.
|
||||
\end{itemize}
|
||||
|
||||
\begin{figure}[t]
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
|
||||
\todo{Draw only the selected method: causal AV to the long-context Omni path;
|
||||
the short-window mixed-audio and anonymous person-set path; residual fusion;
|
||||
the common $10\times3$ head; and the separate wearer-entry/INT-yield evaluation.
|
||||
Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
|
||||
Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
|
||||
\caption{Long-context Omni understanding and short-window local floor evidence
|
||||
for wearer-relative future-activity prediction.}
|
||||
\includegraphics[width=0.98\textwidth]{figures/method_architecture.pdf}
|
||||
\caption{Our strictly causal global--local architecture (top) and streaming
|
||||
alignment (bottom). At every 5-Hz cutoff $t_n$, the global state includes the
|
||||
latest partial 2-s Omni block, while every fifth causal 25-Hz local state
|
||||
provides shared audio and an anonymous visible-person set. Masked person
|
||||
attention contributes only through a zero-initialized residual to the matched
|
||||
$G+U$ baseline.}
|
||||
\label{fig:method}
|
||||
\end{figure}
|
||||
\end{figure*}
|
||||
|
||||
\section{Method}
|
||||
\label{sec:method}
|
||||
|
||||
\subsection{Causal wearer-relative projection}
|
||||
\subsection{Model Architecture}
|
||||
\label{sec:model_arch}
|
||||
|
||||
At decision time $t$, the model observes only the stream
|
||||
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. For
|
||||
bin $k$, the multilabel target
|
||||
At decision time $t$, the model observes only the audiovisual stream
|
||||
$\mathcal X_{\leq t}$ and predicts $K=10$ future bins spaced by
|
||||
$\Delta=0.2$~s. For horizon $k$, the multilabel target
|
||||
$\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
|
||||
background, camera-wearer activity, and activity by one or more other people.
|
||||
Wearer--other overlap is allowed. The model outputs
|
||||
background, camera-wearer activity, and activity by one or more other people;
|
||||
wearer--other overlap is allowed. The output is
|
||||
\begin{equation}
|
||||
\hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t})
|
||||
\in[0,1]^{K\times3}.
|
||||
\label{eq:task}
|
||||
\end{equation}
|
||||
The wearer-activity forecast also provides causal behavioral scores at every
|
||||
horizon. At decision time $t$, if the wearer is silent,
|
||||
$\hat y^{\mathrm w}_{t,k}$ acts as a take-turn score for future bin $k$; if
|
||||
the wearer is active, $1-\hat y^{\mathrm w}_{t,k}$ acts as a yield score.
|
||||
These scores are trained only from future activity targets and are not event
|
||||
hazards or committed controller actions.
|
||||
Adjudicated wearer-entry and INT-yield events used for behavioral evaluation are
|
||||
defined separately from participant-resolved annotations in
|
||||
Sec.~\ref{sec:experiments}.
|
||||
|
||||
\subsection{Global Omni baseline}
|
||||
|
||||
Let $\vect g_t\in\mathbb R^{d_g}$ be the causal global readout of
|
||||
Qwen2.5-Omni at the physical cutoff $t$ \cite{xu2025qwenomni}. The frozen
|
||||
baseline trains only a dimension-matched prediction head. A matched PEFT/LoRA
|
||||
baseline updates a pre-registered subset of the Omni model under the same
|
||||
examples, context, optimizer budget, head, and evaluator. Its purpose is to
|
||||
test whether generic parameter adaptation is sufficient; one negative recipe
|
||||
is not interpreted as a general limitation of LoRA or all Omni models.
|
||||
|
||||
\todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates,
|
||||
streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank,
|
||||
trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a
|
||||
capacity-scaling control. MiniCPM-o 4.5 is restricted to the diagnostic
|
||||
streaming/cache ablation below and does not enter the core method or support a
|
||||
cross-family superiority claim.}
|
||||
|
||||
\subsection{Causal local floor adapter}
|
||||
|
||||
The candidate local path operates on the most recent $L_\ell$ seconds. At local instant
|
||||
$\tau\leq t$, a causal audio encoder produces $\vect a_\tau$ from mixed audio.
|
||||
For the anonymous visible-person set
|
||||
$\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
|
||||
a shared visual encoder and masked set operator produce
|
||||
Our architecture complements globally integrated Omni context with explicitly
|
||||
accessible local floor evidence (Fig.~\ref{fig:method}). Let
|
||||
$\vect g_t\in\mathbb R^{d_g}$ denote the causal Qwen2.5-Omni Thinker readout
|
||||
at $t$ \cite{xu2025qwenomni}. In parallel, the frozen MuVAP ASD frontend
|
||||
\cite{qi2026muvap} produces one shared mixed-audio CPC state $\vect u_t$ and a
|
||||
variable-size set of visible-person states
|
||||
\begin{equation}
|
||||
\vect v_\tau=\mathrm{SetPool}
|
||||
\left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}
|
||||
\cup\{\vect v_\varnothing\}\right),
|
||||
\label{eq:set}
|
||||
\mathcal P_t=\{\vect p_{t,i}\}_{i=1}^{N_t},\qquad
|
||||
\vect p_{t,i}=[\vect v_{t,i};s_{t,i}],
|
||||
\label{eq:person_state}
|
||||
\end{equation}
|
||||
where $\vect v_\varnothing$ handles an empty or offscreen visible set. The
|
||||
operator is permutation invariant and uses no participant-ID embedding or
|
||||
cross-time identity. A causal short-window encoder summarizes local dynamics,
|
||||
where $\vect v_{t,i}$ is a causal visual state and $s_{t,i}$ is one
|
||||
audio--visual speaking logit. The shared audio state is not duplicated in
|
||||
each person token. The prediction model receives neither participant IDs nor
|
||||
cross-time track identity; provided causal face boxes are used only to isolate
|
||||
the representation and fusion question.
|
||||
|
||||
A matched global--audio head first maps $[\vect g_t;\vect u_t]$ to baseline
|
||||
logits $\vect Z_t^{\mathrm{base}}\in\mathbb R^{K\times3}$ and an intermediate
|
||||
state $\vect h_t$. The same inputs query the current person set:
|
||||
\begin{equation}
|
||||
\vect \ell_t=F_\ell
|
||||
\left(\{[\vect a_\tau;\vect v_\tau]\}_{\tau\in(t-L_\ell,t]}\right).
|
||||
\label{eq:local}
|
||||
\begin{aligned}
|
||||
\vect q_t &= W_g\vect g_t+W_a\vect u_t,\\
|
||||
\mathcal M_t &= \{W_p\vect p_{t,i}\}_{i=1}^{N_t}
|
||||
\cup\{\vect p_\varnothing\},\\
|
||||
\vect c_t &= \mathrm{MHA}(\vect q_t,\mathcal M_t,\mathcal M_t),
|
||||
\end{aligned}
|
||||
\label{eq:person_attention}
|
||||
\end{equation}
|
||||
The adapter preserves the global initialization through a zero-initialized
|
||||
residual output,
|
||||
where masking supports variable cardinality and the learned null token
|
||||
$\vect p_\varnothing$ makes the attention well defined for an empty set. If
|
||||
$m_t$ indicates that at least one valid person observation exists, the person
|
||||
branch supplies only a residual logit correction,
|
||||
\begin{equation}
|
||||
\vect z_t=\vect g_t+W_o\,
|
||||
\sigma\!\left(W_g\vect g_t+W_\ell\vect\ell_t\right),
|
||||
\qquad W_o\vert_{\mathrm{init}}=0,
|
||||
\label{eq:fusion}
|
||||
\vect Z_t=\vect Z_t^{\mathrm{base}}
|
||||
{}+m_t\,\mathrm{reshape}\!\left(W_r[\vect h_t;\vect c_t;m_t]\right),
|
||||
\qquad W_r\vert_{\mathrm{init}}=0,
|
||||
\label{eq:residual}
|
||||
\end{equation}
|
||||
and the common head maps $\vect z_t$ to Eq.~\eqref{eq:task}. This implements an
|
||||
evidence-access hypothesis rather than adding an unconstrained second model:
|
||||
the global path retains broad context, while the local path preserves timely
|
||||
floor evidence.
|
||||
and $\hat{\vect Y}_t=\mathrm{softmax}_c(\vect Z_t)$. Thus, after copying and
|
||||
freezing the trained $G+U$ head, the complete model initially reproduces that
|
||||
baseline exactly; only the person projection, query, masked attention, and
|
||||
residual readout are trained in the adaptation stage.
|
||||
|
||||
\todo{Freeze $L_\ell$, local audio/video rates, crop source and resolution,
|
||||
set pooling, causal temporal encoder, masks, reset rules, parameter count, and
|
||||
latency. If the local path runs faster than the 5-Hz global/output grid, compare
|
||||
it with a same-parameter 5-Hz variant before using the term ``dual-rate.'' Use
|
||||
provided causal boxes only as a clearly labeled method-isolation condition;
|
||||
online detection/tracking is not a paper contribution unless actually run.}
|
||||
|
||||
\subsection{Training}
|
||||
|
||||
The primary objective is class-balanced binary cross entropy over valid
|
||||
horizons and activity channels,
|
||||
We use the EgoSpeak/LSTR multi-positive cross-entropy over causal-valid rows,
|
||||
\begin{equation}
|
||||
\mathcal L_{\mathrm{act}}=
|
||||
\sum_{t,k,c}m_{t,k}w_c\,
|
||||
\mathrm{BCE}(\hat y^c_{t,k},y^c_{t,k}).
|
||||
\begin{aligned}
|
||||
\mathcal L&=-\frac{1}{M}\sum_{t,k,c}m_{t,k}y^c_{t,k}
|
||||
\log\mathrm{softmax}_c(\vect Z_{t,k}),\\
|
||||
M&=\sum_{t,k}m_{t,k}.
|
||||
\end{aligned}
|
||||
\label{eq:loss}
|
||||
\end{equation}
|
||||
No transcript, target-of-speech label, close microphone, future frame, or
|
||||
per-person speech label enters the deployable model. \todo{Add an active-speaker
|
||||
or event auxiliary loss only if it improves the frozen primary protocol and
|
||||
its label/input boundary is fully reported.}
|
||||
For an overlap target, both active channels contribute a log-probability term.
|
||||
No transcript, target-of-speech annotation, close-microphone signal, speech
|
||||
label, or future observation is supplied to the forecasting model. The
|
||||
wearer forecast is interpreted only at evaluation time: it is an entry proxy
|
||||
when the wearer is silent and its complement is a yield proxy when the wearer
|
||||
is active, not a trained event hazard or committed controller action.
|
||||
|
||||
\subsection{Causal Streaming Architecture}
|
||||
\label{sec:causal_streaming}
|
||||
|
||||
We distinguish encoder processing granularity from the common forecasting
|
||||
rate. Predictions are emitted at $t_n=n\Delta$, $n\geq1$, while every
|
||||
representation used at $t_n$ is restricted to media available by that physical
|
||||
cutoff.
|
||||
|
||||
The global path uses an 8-s rolling causal presentation budget with
|
||||
source-aligned 2-s audiovisual blocks. Completed blocks are cached and
|
||||
replayed, while the currently incomplete block is a temporary tail. At every
|
||||
200-ms cutoff, the tail is rebuilt with all newly observed audio and video and
|
||||
included in computing $\vect g_{t_n}$; when it reaches 2~s, it is committed to
|
||||
the reusable history. Consequently, the Omni state is not held constant for
|
||||
2~s: it includes the latest available content at every decision cutoff without
|
||||
observing media after $t_n$. We retain the final four Thinker query-layer
|
||||
states and fuse them in the matched head with a concatenation MLP.
|
||||
|
||||
The local frontend operates causally at 25~Hz. State $j$ consumes the
|
||||
half-open interval $[j/25,(j+1)/25)$ and is timestamped at its right edge.
|
||||
Mixed audio is encoded once as a continuous shared CPC stream, while each
|
||||
available face crop is processed by the causal visual and audio--visual
|
||||
speaking modules. Ten-second extraction chunks may prepend up to 2~s of past
|
||||
context, never future context. At a 5-Hz forecasting cutoff, we select the
|
||||
local state whose right edge equals $t_n$; hence every fifth 25-Hz state aligns
|
||||
with one Omni prediction row. The two paths therefore describe the same
|
||||
causal prefix: the Omni path preserves integrated conversational context,
|
||||
whereas the local path exposes fine-grained audio and visible-person floor
|
||||
evidence. Fusion and prediction occur only on the common 5-Hz grid.
|
||||
|
||||
|
||||
\section{Experiments}
|
||||
@@ -301,8 +309,8 @@ matched PEFT/LoRA adaptation, global features with recent local audio, a
|
||||
parameter- and history-matched temporal local branch, and the proposed causal
|
||||
local audio--visual adapter. Qwen2.5-Omni-7B is used as a capacity-scaling
|
||||
control. All learned variants use identical causal examples, prediction
|
||||
horizons, prediction heads, and the class-balanced binary cross-entropy
|
||||
objective in Eq.~\eqref{eq:loss}.
|
||||
horizons, prediction heads, and the multi-positive cross-entropy objective in
|
||||
Eq.~\eqref{eq:loss}.
|
||||
|
||||
The diagnostic streaming ablation additionally uses MiniCPM-o 4.5
|
||||
\cite{cui2026minicpmo45}. Its native cached stream is evaluated at its native
|
||||
|
||||
Reference in New Issue
Block a user