Streamline targeted methods and experiment presentation

AGENTS.md

@@ -50,7 +50,7 @@ Introduction 的故事或改换核心研究问题；可继续核实文献、修
    泛称所有 Omni 模型都缺乏局部感知。
 4. **本文观点与三项贡献。** 先提出保留 Omni 全局状态、同时在每个因果截止点显式接入
    近期模态证据的观点，再以冒号引出三项贡献：(1) SWE（short-window encoder
-   evidence，短窗编码器证据）将近期音频与可选的因果人物框视觉特征接入冻结 Omni
+   evidence，短窗编码器证据）将近期音频与可选的因果视觉编码特征接入冻结 Omni
    预测头；“短窗”指时间局部，不是浅层网络或任务级端到端训练。(2) PTR（partial-tail
    refresh，部分尾块刷新）在复用已完成块时只查询未完成块已观测前缀，不把它提交到
    持久历史；其收益须由相同 cutoff、相同上下文上限的配对实验验证。(3) EasyCom
@@ -70,7 +70,7 @@ Method 按以下三个小节展开，除非用户明确要求，不再改变这
 
 1. **Task and Global--Local Fusion。** 定义 5 Hz 因果截止点、十个未来 horizon 和
    background/wearer/others 输出；说明 Thinker 全局状态本身已含音视频，再定义 SWE
-   的近期音频及可选视觉旁路。区分 pooled 特征的拼接聚合与 F2/F3 保留视觉 token 的
+   的近期音频及可选视觉旁路。区分 pooled 特征的拼接聚合与 SVA/TSVA 保留视觉 token 的
    query attention。训练范围分开写：主方案只训练 readout/局部分支/预测头；Thinker
    LoRA、限制在上层的 LoRA、预测头冻结或联合训练是参数适配控制。最后四层
    **readout** 与 LoRA 更新哪些层是两个独立维度。已归档 LoRA 结果为非匹配探索，
@@ -78,13 +78,18 @@ Method 按以下三个小节展开，除非用户明确要求，不再改变这
 2. **Causal Partial-Tail Refresh。** 完成块进入持久历史；每个 200 ms 截止点只临时查询
    当前 2 秒块已观测的前缀，尾块不提交、不中途引入未来媒体。与 completed-only
    在同一截止点和上下文上限下配对比较；未完成实验前只称因果访问规则。
-3. **Short-Window Feature Extraction。** 默认局部音频为截止点前一秒混合波形经冻结
-   Qwen 音频编码器；EgoSpeak 式 wav2vec2 卷积编码是待做的同窗口、同预测头对照，
-   不能冒充已测 G+A。视觉区分最近单帧、最近一秒多帧人物框 pooled 特征、无框的
-   F2 最新全画面空间 token，以及 F3 同网格位置两帧 token 经共享 GRU 后注意力聚合。
-   另列一种特殊的短窗信息审计：由转写标注生成、仅含截止点之前的可见非 wearer
-   活动历史，比较逐人、打乱、延迟和 OR 聚合等形式。它不是部署时可获得的视觉特征，
-   也不是性能上界。F3 不是人物跟踪，F3--F2 差异也不能单独归因于运动理解。
+3. **Short-Window Feature Extraction。** 默认局部音频 `A` 为截止点前一秒混合波形经冻结
+   Qwen 音频编码器；EgoSpeak 式 wav2vec2 卷积编码记为 `A_E`，是待做的同窗口、同预测头对照，
+   不能冒充已测 G+A。视觉区分最近一秒多帧人物框池化的 Pool、无框的
+   SVA（spatial visual attention）最新单帧全画面空间 token，以及 TSVA（temporal
+   spatial visual attention）同网格位置两帧 token 经共享 GRU 后注意力聚合。
+   TSVA 不是人物跟踪，TSVA--SVA 差异也不能单独归因于运动理解。由转写标注生成的
+   过去可见人活动历史只在 Results 的信息审计中定义和分析，不放进 Method 2.3；
+   它不是部署时可获得的视觉特征，也不是性能上界。
+
+主结果表固定为 EasyCom 与 Ego4D 各四行：EgoSpeak A+V、Omni G、Omni G+A、
+Omni G+A+SVA，并保留十个 0.2--2.0 秒 horizon 与均值。尚未取得的 SVA/Ego4D
+分数用 `--` 占位，不插值、不把计划写成结果；pooled-V 与 TSVA 只进入视觉分析表。
 
 ## 不可破坏的研究契约
 

manuscript/targeted/main.tex

@@ -34,7 +34,7 @@ models and streaming Omni assistants address related problems, but the
 cutoff-level availability of local evidence in an integrated Omni state is
 underexplored for this wearer-centered task.  We examine two interfaces:
 short-window encoder evidence (SWE) supplies recent audio and optional
-person-crop vision directly to a frozen-Omni forecasting head; partial-tail
+causal visual evidence directly to a frozen-Omni forecasting head; partial-tail
 refresh (PTR) queries only the observed prefix of the current audiovisual
 block at each 200-ms cutoff.  On EasyCom, the exploratory global-plus-audio
 screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus
@@ -123,7 +123,7 @@ evidence at each causal cutoff (Fig.~\ref{fig:overview}). EasyCom screens
 suggest that global context and local audio are complementary; gains from
 pooled vision and isolated PTR effects remain unproven. This study makes
 three contributions: (1) short-window encoder evidence (SWE) supplies recent
-audio and optional person-crop vision to a frozen-Omni forecasting head;
+audio and optional visual encoder evidence to a frozen-Omni forecasting head;
 (2) partial-tail refresh (PTR) queries the observed current-block prefix
 without committing it to persistent history; and (3) detailed EasyCom
 analyses accompany a specified Ego4D transfer protocol. Matched PTR,
@@ -133,7 +133,7 @@ Ego4D scores, and event behavior remain pending.
 \centering
 \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
 \caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
-with recent audio and optional pooled person-crop evidence in the forecasting
+with recent audio and optional local visual evidence in the forecasting
 head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
 the observed current-block tail at each 200-ms cutoff. The tail is not
 committed to persistent history.}
@@ -211,40 +211,28 @@ pending, so PTR is a causal access rule rather than a demonstrated gain.
 
 \subsection{Short-Window Feature Extraction}
 
-We define every local observation by its prediction cutoff, before applying
-any encoder-specific preprocessing. The default audio branch encodes the
-mixed waveform in $[t_n-1\,\mathrm{s},t_n)$ with the frozen Qwen2.5-Omni
-audio encoder and averages valid output tokens. A wav2vec~2.0 convolutional
-audio representation, as used by EgoSpeak \cite{kim2025egospeak}, is an
-encoder-choice control; it must use the same one-second input interval and
-forecasting head before differences can be attributed to the encoder. It is
-not the source of the reported $\G+\A$ result.
+Each local branch observes only media preceding the prediction cutoff
+$t_n$; unavailable observations retain explicit validity masks rather
+than borrowing later samples. All encoders are frozen. The notation below
+distinguishes encoder choice from fusion choice: $\G$ is the global Thinker
+state, and local branches are appended to the same forecasting head.
 
-For vision, a single-frame variant uses the latest frame strictly before
-$t_n$; the pooled person-crop variant averages frozen Qwen vision tokens
-within provided boxes, across visible people, and over valid 2-FPS frames
-in the preceding second. These pooled features summarize appearance but
-discard person identity and explicit motion. Boxes are dataset-provided,
-not predictions from an evaluated online detector. Two no-box full-frame
-alternatives retain the post-merger spatial-token grid: F2 lets the
-global--audio query attend to the latest causal frame; F3 first encodes up
-to two observations at each matching grid location in chronological order
-with a shared GRU, then applies the same query to the resulting grid. F3
-links image coordinates, not people or tracks. Its difference from F2
-confounds additional history and temporal capacity, so it cannot alone
-establish a motion or speaker-grounding mechanism. Empty or unavailable
-local evidence is indicated by the validity masks, never filled with a
-future observation.
+The default audio feature $\A$ encodes the mixed waveform in
+$[t_n-1\,\mathrm{s},t_n)$ with the Qwen2.5-Omni audio encoder and averages
+valid tokens. $\A_{\rm E}$ denotes EgoSpeak's wav2vec~2.0 convolutional
+representation \cite{kim2025egospeak}; comparing $\G+\A$ with
+$\G+\A_{\rm E}$ requires the same window and head. Released EgoSpeak A+V
+remains an external baseline, not this encoder substitution.
 
-As a distinct, privileged short-window feature, we also derive visible
-non-wearer speech activity from transcript annotations \emph{before} each
-cutoff. Each visible participant contributes a two-second binary history
-on the 5-Hz grid. We test visibility-only, per-person, identity-shuffled,
-one-second-delayed, and OR-pooled aggregate histories to ask which timely
-activity information a learned visual branch would need to recover.
-Although strictly past-only, these annotations are not available from raw
-media at inference; this is an information audit, not a deployable feature
-or a performance upper bound.
+For vision, Pool averages Qwen tokens within provided person boxes and
+across valid 2-FPS frames in the preceding second; the boxes are not
+outputs of an evaluated online detector. Spatial visual attention (SVA)
+instead lets $\G+\A$ query the unpooled spatial tokens of the latest
+strictly pre-cutoff full frame. Temporal spatial visual attention (TSVA)
+first applies a shared GRU to up to two causal tokens at each matching
+grid position, then uses the same query. TSVA links image coordinates,
+not person tracks. We denote these variants by $\G+\A+$Pool,
+$\G+\A+$SVA, and $\G+\A+$TSVA.
 
 \input{tables_compact}
 % Historical full-precision table layouts retained as source audit only.
@@ -253,7 +241,7 @@ or a performance upper bound.
 \caption{Measured EasyCom screens (\%). Panel A: deployable models; Panel B:
 three-seed privileged past-activity audit, which is not deployable. Mean is
 ten-horizon dense mAP; On/Off are raw wearer endpoint AP, not semantic events.
-F3 is a no-box, two-frame causal full-frame screen; it and the historical
+TSVA is a no-box, two-frame causal full-frame screen; it and the historical
 deployable rows are exploratory seed-0 runs with unmatched head/training
 details. F1 has three seeds. Dashes are unavailable, not zero.}
 \label{tab:main}
@@ -272,7 +260,7 @@ Omni global $\G$ & 73.446 & 77.867 & 60.849 & -- & -- \\\\
 Omni $\G+\A$ & 81.723 & 85.066 & 65.109 & 25.960 & 45.251 \\\\
 Omni $\G+\A+$pooled-$\V$ & 82.027 & 85.086
 & 65.400 & 26.019 & 45.371 \\\\
-No-box full-frame F3 & 82.380 & -- & 65.540 & -- & -- \\\\
+No-box full-frame TSVA & 82.380 & -- & 65.540 & -- & -- \\\\
 Current-frame person set F1 & $81.777{\pm}0.156$ & --
 & $64.951{\pm}0.095$ & -- & -- \\\\
 Matched-width $\G$ (L2) & -- & -- & -- & -- & -- \\\\
@@ -396,23 +384,19 @@ Full path & -- & -- & -- \\
 \section{Experiments}
 \label{sec:experiments}
 
-\textbf{Data and protocol.}
-We use EasyCom with the EgoSpeak official split: Sessions~4--12 for training
-and Sessions~1--3 for testing.  Inputs and targets share a 5-Hz grid.  Heads
-are trained for four fixed epochs on all training sessions without test-time
-checkpoint selection.  The deployable feature matrix is an archived
-single-seed screen; the information audit uses seeds 0, 1, and 2 with an
-identical fixed recipe.  We therefore report standard deviations only for the
-latter and treat small differences in the former descriptively.
-
-\textbf{Ego4D transfer protocol (pending results).}
-Ego4D audiovisual clips \cite{grauman2022ego4d} use the existing EgoSpeak
-manifest and targets, with 346 training and 87 test clips.  The same 5-Hz
-cutoffs and local 1-s encoder windows define $\G$, $\G+\A$, $\G+\V$, and
-$\G+\A+\V$ comparisons; visual validity is retained rather than silently
-dropping no-box rows.  The repository contains the extraction and training
-workflow but no archived Ego4D score or matched PTR control.  Hence this
-protocol is specified, not reported as an observed cross-dataset gain.
+\textbf{Data Preparation.}
+We use the EgoSpeak EasyCom split (Sessions~4--12 for training and
+Sessions~1--3 for testing) and its Ego4D audiovisual manifest
+\cite{donley2021easycom,grauman2022ego4d}, which contains 346 training and
+87 test clips. Both datasets use 5-Hz causal cutoffs and targets, the same
+one-second local-audio window, and visual samples strictly preceding each
+cutoff; unavailable vision is masked rather than silently dropping rows.
+EasyCom heads use four fixed training epochs without test-set checkpoint
+selection. Its deployable feature matrix is an archived single-seed screen,
+whereas the privileged information audit uses three seeds under one recipe;
+small single-seed differences are descriptive. Ego4D extraction and training
+workflows exist, but test scores and a matched PTR comparison have not yet
+been archived, so cross-dataset gains are not claimed.
 
 \textbf{Metrics.}
 At each horizon we compute AP for the three activity channels over common
@@ -424,30 +408,16 @@ speech and scores future silence with $1-p(\mathrm{wearer})$.  These
 threshold-free quantities average ten horizons.  They are not one-event-one-
 vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding.
 
-\textbf{Comparisons.}
-We evaluate released EgoSpeak MiniROADA A+V; recent audio $\A$; global Omni
-$\G$; $\G+\A$; and $\G+\A+$pooled-$\V$.  A three-seed current-frame
-frame-set attention screen tests whether a richer person aggregation is
-already sufficient.  Because that screen used Session~12 model selection
-whereas the historical pooled row did not, their difference is diagnostic,
-not a matched causal attribution.
-
-\textbf{Pre-specified analyses.}
-We freeze six control families: (L) capacity and recency, (T) tail
-refresh crossed with local audio, (C) six context caps, (B) paired backbone
-transfer, (H/Q) error slices and timelines, and (S) full-path efficiency.
-They share the official split and physical cutoffs; the 8-s T/C row reuses
-one run.  The source retains unmeasured hypotheses and falsifiers.
-
 \section{Results and Analysis}
 \label{sec:results}
 
 \subsection{EasyCom: Local Encoder Evidence}
 
-Table~\ref{tab:main} shows that recent encoder-side audio $\A$ is strong at
-0.2~s while integrated $\G$ is slightly stronger on the ten-horizon mean.
-Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 over
-$\G$.  Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
+Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate
+audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and
+60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon
+mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26
+over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at
 0.2~s and 3.92 mean-mAP points.  Raw wearer offset AP rises by 7.23
 points, onset AP by only 1.47: dense gains do not imply equal improvement
 at both activity boundaries.
@@ -524,20 +494,26 @@ not real-time deployment. \\
 
 \subsection{EasyCom: Visual Evidence and Ablations}
 
-Adding pooled $\V$ to $\G+\A$ produces only +0.304 points at 0.2~s and
+Table~\ref{tab:visual} separates visual screens from the cross-dataset main
+matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and
 +0.291 mean mAP.  Its wearer increment is +0.020 at 0.2~s and +0.061 when
 averaged over horizons; onset/offset changes are +0.059/+0.120.  A richer
 current-frame person-set model is stable across seeds but reaches lower mean
 mAP than the historical pooled row under a non-matched selection protocol.
 These results do not support a claim of learned speaker grounding. A separate
-seed-0 no-box full-frame F3 screen reaches 82.38\% at 0.2~s and 65.54\%
+seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\%
 mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same
-causal-valid dense protocol. F3 adds a two-frame 2-FPS token history and a
+causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a
 shared GRU, while head capacity and training batch differ; raw boundary
 metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point
 difference does not isolate visual motion or floor behavior.
 
-Table~\ref{tab:oracle} locates the missing information. Aggregate visible
+The analysis-only audit in Table~\ref{tab:oracle} substitutes two seconds
+of transcript-derived, strictly past visible-person activity on the 5-Hz
+grid. It tests visibility alone, per-person histories, shuffled identities,
+a one-second delay, and OR-pooled activity. These privileged labels are
+neither deployable visual features nor a performance upper bound.
+Aggregate visible
 activity improves over the audio baseline by 4.826 points at 0.2~s and 1.775
 mean-mAP points.  Visibility alone changes mean mAP by only +0.040, and
 one-second-delayed activity by +0.164.  Thus availability or extra parameters

manuscript/targeted/tables_compact.tex

@@ -1,13 +1,11 @@
 % Active result tables. A dash denotes an unreported or unmeasured value.
 \begin{table*}[t]
-\caption{Dense future-activity mAP (\%) by prediction horizon. EasyCom
-scores use Sessions~1--3; the complete ten-horizon curves are reported for
-EgoSpeak, $\G+\A$, and F3. Other archived screens provide only 0.2-s and
-ten-horizon mean scores, so intermediate cells are not interpolated.
-Ego4D transfer is pending. Mean averages all ten horizons, including where
-the individual values are unavailable. F3 and the fusion rows are exploratory
-seed-0 runs; F1 reports a three-seed mean. These rows are not all
-capacity- or training-matched.}
+\caption{Dense future-activity mAP (\%) at ten horizons on EasyCom and
+Ego4D. Each dataset has the same four-method comparison. EasyCom values
+are measured on Sessions~1--3; $\G$ has only archived 0.2-s and mean
+scores, and SVA awaits evaluation. Ego4D scores are pending. Dashes are unreported or unmeasured,
+not zero; intermediate scores are never interpolated. The available Omni
+rows are exploratory and not all training-matched.}
 \label{tab:main}
 \centering\ninept
 \renewcommand{\arraystretch}{0.91}
@@ -17,22 +15,41 @@ capacity- or training-matched.}
 \cmidrule(lr){2-11}
 Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \\
 \midrule
-\multicolumn{12}{@{}l}{\emph{EasyCom (measured)}} \\
+\multicolumn{12}{@{}l}{\emph{EasyCom}} \\
 EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\
-Recent $\A$ & 79.69 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.51 \\
 Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\
-$\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\
-$\G+\A+$pooled-$\V$ & 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 65.40 \\
-Full-frame F3 & 82.38 & 76.42 & 70.78 & 66.35 & 63.93 & 61.87 & 60.41 & 59.11 & 57.61 & 56.53 & 65.54 \\
-Person-set F1 & 81.78 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 64.95 \\
+Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\
+Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
 \midrule
-\multicolumn{12}{@{}l}{\emph{Ego4D (transfer pending)}} \\
+\multicolumn{12}{@{}l}{\emph{Ego4D}} \\
+EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
 Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
-$\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
+Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
+Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
 \bottomrule
 \end{tabular*}
 \end{table*}
 
+\begin{table}[t]
+\caption{EasyCom visual-encoding screens (\%). Branch names are defined
+in Sec.~\ref{sec:method}. Available rows are exploratory and not
+capacity- or training-matched.}
+\label{tab:visual}
+\centering\ninept
+\renewcommand{\arraystretch}{0.92}
+\setlength{\tabcolsep}{4pt}
+\begin{tabular}{@{}lcc@{}}
+\toprule
+Visual branch on $\G+\A$ & mAP$_{.2}$ & Mean \\
+\midrule
+None & 81.72 & 65.11 \\
+Pool & 82.03 & 65.40 \\
+SVA & -- & -- \\
+TSVA & 82.38 & 65.54 \\
+\bottomrule
+\end{tabular}
+\end{table}
+
 \begin{table}[t]
 \caption{EasyCom privileged past-activity audit (\%). Three-seed mean
 $\pm$ sample SD. The extra activity cues are not deployable inputs; W is