Refine targeted methods and freeze section structure

manuscript/targeted/main.tex

@@ -143,7 +143,7 @@ committed to persistent history.}
 \section{Method}
 \label{sec:method}
 
-\subsection{Task and Short-Window Encoder Evidence}
+\subsection{Task and Global--Local Fusion}
 
 At the 5-Hz cutoff $t_n=0.2n$~s, the model observes only media available by
 $t_n$ and predicts activity at $t_n+h$, where
@@ -152,32 +152,48 @@ horizon has three channels in fixed order: background, camera wearer, and
 aggregate others.  Wearer and others may overlap; no output identifies an
 individual speaker, addressee, or semantic end of turn.
 
-The global feature $\vect g_n$ concatenates the final four frozen
-Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions.  It
-already contains audio and video context; $\G$ is not audio-free.  SWE
-provides a separate short-window path so the head need not recover all
-cutoff-local detail from the integrated state. Local audio $\vect a_n$ crops
-the latest one-second mixed
-waveform \emph{before} preprocessing, runs the frozen Qwen audio encoder, and
-averages valid tokens.  Optional $\vect v_n$ averages frozen vision tokens
-inside provided person boxes, then across people and valid causal 2-FPS
-frames in the latest second. This pooling has no identity or mouth-motion trajectory;
-provided boxes do not imply an online detector.
+The global representation $\vect g_n$ concatenates the final four frozen
+Qwen2.5-Omni-3B Thinker readouts and projects them to 2048 dimensions. It
+already integrates audio and video over the available context; $\G$ is not
+an audio-free baseline. Short-window encoder evidence (SWE) adds a direct,
+cutoff-aligned path for recent audio $\vect a_n$ and, when enabled, visual
+evidence $\vect v_n$. This separates the question of what the global state
+encodes from whether a small prediction head can access transient cues.
 
-Each enabled local feature is layer-normalized, projected to 256 dimensions,
-and paired with a validity bit.  The trainable head is
+In the pooled fusion, each available local vector is layer-normalized,
+projected to 256 dimensions, and accompanied by a validity bit. The
+concatenated global and local features are passed to a layer-normalized MLP:
 \begin{equation}
- \widehat{\vect Y}_{n}=
- \operatorname{softmax}_{c}F_{\theta}
- \bigl[\vect g_n;\widetilde{\vect a}_n;m_n^a;
-       \widetilde{\vect v}_n;m_n^v\bigr],
+ \widehat{\vect Y}_{n,h,c}=\operatorname{softmax}_{c}
+ F_{\theta}\!\left([\vect g_n;\widetilde{\vect a}_n;m_n^a;
+       \widetilde{\vect v}_n;m_n^v]\right)_{h,c},
  \label{eq:fusion}
 \end{equation}
-where omitted modalities remove their terms and $F_\theta$ is a
-layer-normalized MLP with a 30-logit output.  We train only the projections and
-head with the EgoSpeak multi-positive cross-entropy over causal-valid
-source--horizon pairs.  The Omni and modality encoders remain frozen, so this
-is frozen-backbone adaptation rather than end-to-end training.
+where a disabled branch is omitted and $F_\theta$ produces ten sets of
+three logits. For token-preserving visual variants, $\vect v_n$ denotes
+an attention-pooled context: a query formed from $\vect g_n$ and valid
+$\vect a_n$ attends to the causal visual token set (and a learned null
+token), rather than averaging it beforehand. The resulting context and
+validity bit enter the same prediction head. We train the projections,
+visual aggregator when present, and head with the EgoSpeak multi-positive
+cross-entropy over causal-valid source--horizon pairs. The principal SWE
+configuration freezes all feature encoders and the Thinker: this is
+frozen-backbone adaptation, not task-level end-to-end training.
+
+We separate \emph{where} parameters are updated from \emph{which} Thinker
+layers are read. The primary comparison trains only the layer-combination
+projection, local-fusion modules, and 30-logit prediction head on cached
+frozen features. Parameter-adaptation controls instead attach low-rank
+adapters to Thinker attention projections, either across the configured
+Thinker or restricted to its upper layers; the prediction head may be
+updated jointly or held fixed to isolate representation changes. These
+controls require the same causal media, source cutoffs, supervision,
+initialization, and model-selection protocol as the frozen-head condition.
+Archived LoRA runs, including an upper-eight-layer setting, are
+exploratory and not a matched layer-scope ablation. Last-four-layer
+\emph{readout} fusion changes the head input, not the LoRA update scope;
+neither its gain nor an unmatched LoRA score establishes an adaptation
+advantage.
 
 \subsection{Causal Partial-Tail Refresh}
 
@@ -193,18 +209,42 @@ completed-only versus PTR readouts with $\A$ on/off at an identical 8-s cap;
 the separate 2--12-s cap sweep tests context length. Those comparisons are
 pending, so PTR is a causal access rule rather than a demonstrated gain.
 
-\subsection{Visible-Activity Information Audit}
+\subsection{Short-Window Feature Extraction}
 
-To distinguish missing information from ineffective learned vision, we build
-analysis-only features from transcript-derived \emph{past} activity.  At each
-cutoff, visible non-wearer participants receive two seconds of binary history
-sampled on the same 5-Hz grid.  The preparation fails if its aggregate
-activity disagrees with the released target.  We compare: (i) visibility with
-activity zeroed, (ii) per-person histories, (iii) identities shuffled while
-preserving active-person counts at every lag, (iv) histories delayed by one
-second, and (v) one OR-pooled aggregate-visible history.  These conditions
-never expose future activity, but their speech labels make them privileged
-diagnostics, not deployable features or upper bounds.
+We define every local observation by its prediction cutoff, before applying
+any encoder-specific preprocessing. The default audio branch encodes the
+mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni
+audio encoder and averages valid output tokens. A wav2vec~2.0 convolutional
+audio representation, as used by EgoSpeak \cite{kim2025egospeak}, is an
+encoder-choice control; it must use the same one-second input interval and
+forecasting head before differences can be attributed to the encoder. It is
+not the source of the reported $\G+\A$ result.
+
+For vision, a single-frame variant uses the latest frame strictly before
+$t_n$; the pooled person-crop variant averages frozen Qwen vision tokens
+within provided boxes, across visible people, and over valid 2-FPS frames
+in the preceding second. These pooled features summarize appearance but
+discard person identity and explicit motion. Boxes are dataset-provided,
+not predictions from an evaluated online detector. Two no-box full-frame
+alternatives retain the post-merger spatial-token grid: F2 lets the
+global--audio query attend to the latest causal frame; F3 first encodes up
+to two observations at each matching grid location in chronological order
+with a shared GRU, then applies the same query to the resulting grid. F3
+links image coordinates, not people or tracks. Its difference from F2
+confounds additional history and temporal capacity, so it cannot alone
+establish a motion or speaker-grounding mechanism. Empty or unavailable
+local evidence is indicated by the validity masks, never filled with a
+future observation.
+
+As a distinct, privileged short-window feature, we also derive visible
+non-wearer speech activity from transcript annotations \emph{before} each
+cutoff. Each visible participant contributes a two-second binary history
+on the 5-Hz grid. We test visibility-only, per-person, identity-shuffled,
+one-second-delayed, and OR-pooled aggregate histories to ask which timely
+activity information a learned visual branch would need to recover.
+Although strictly past-only, these annotations are not available from raw
+media at inference; this is an information audit, not a deployable feature
+or a performance upper bound.
 
 \input{tables_compact}
 % Historical full-precision table layouts retained as source audit only.