feat: add causal frame set attention fusion

PROJECT_CONTEXT.md

@@ -19,6 +19,7 @@ recall, without hurting addressed responses or yield behavior.
 
 | Need | Authoritative file |
 |---|---|
+| Progressive feature-fusion implementation route | [docs/progressive_feature_fusion_route_2026-09-02.md](docs/progressive_feature_fusion_route_2026-09-02.md) |
 | Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) |
 | Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) |
 | Code/data map and current blocker | This file |
@@ -42,6 +43,13 @@ result.
 - Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
 - Timing-derived wearer floor-state and event diagnostics.
 
+### Implemented, awaiting remote runtime verification
+
+- F1 Frame-set Attention loader/head/train/infer path. It uses fused G+A as
+  one query over the latest half-open causal person set, retains direct A in
+  the head, and dynamically pads people per batch. Do not report it as a
+  measured method until the L20 smoke and full controls pass.
+
 ### Current blocker
 
 `evaluate_floor_state_changes.py` counts a wearer transition anywhere in a
@@ -60,10 +68,13 @@ semantic event ground truth.
 2. **P1:** probe aggregate activity and per-person active-speaker grounding;
    compare correct, wrong-person and time-shifted visual inputs.
 3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
-4. **P3:** train an anonymous person-wise causal side branch with matched
-   temporal-capacity controls.
+4. **P3:** begin the progressive person-fusion route with Frame-set Attention,
+   then add spatial or track-temporal structure only after its controls pass.
 5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.
 
+The detailed architecture, controls and stopping gates for P3 are frozen in
+[the progressive feature-fusion route](docs/progressive_feature_fusion_route_2026-09-02.md).
+
 ## Stable data contracts
 
 ### Time and labels

docs/README.md

@@ -7,10 +7,11 @@ implemented or validated.
 
 | Document | Role |
 |---|---|
+| [progressive_feature_fusion_route_2026-09-02.md](progressive_feature_fusion_route_2026-09-02.md) | Gated route from pooled local evidence to Frame-set Attention, temporal person memory and Thinker adapters |
 | [`omni_speaker_floor_experiment_plan_2026-08-30.md`](omni_speaker_floor_experiment_plan_2026-08-30.md) | Current hypotheses, experiment gates, main table and stopping rules |
 | [`multiparty_egocentric_floor_gap_audit_2026-08-30.md`](multiparty_egocentric_floor_gap_audit_2026-08-30.md) | Novelty boundary, closest work and restricted claims |
 
-These two files supersede earlier documents when discussing the paper's main
+These documents supersede earlier documents when discussing the paper's main
 question. They do not supersede verified implementation details in code and
 tests.
 

docs/local_feature_baseline.md

@@ -174,6 +174,27 @@ warmup_factor=0.3、dropout=0.2、LSTR-MCE、不加 transition objective。
 Loader 会检查 sample、时间、label_indices、维度、valid mask 和 encoder/pooling 配置。
 训练与评测必须用同一种 AP 协议作比较；不要混用 full-timeline 与 causal-valid 的表格。
 
+## 4. F1 Frame-set Attention
+
+已有完整局部 sidecar 的 persons 字段保存了逐帧、逐人的原生 2048D 特征，因此 F1
+不需要重新提取图片或音频。新增训练参数为：
+
+    --local-feature-root "$LOCAL_ROOT" \
+    --local-modalities av \
+    --local-fusion frame-set-attention \
+    --local-projection-dim 256 \
+    --person-attention-dim 256 \
+    --person-attention-heads 4
+
+F1 用融合后的 G 和直接局部音频 A 构造一个 query，读取当前最新半开因果帧中的可变人物
+集合。人物集合只在 batch 内动态 padding；NULL person 处理没有有效框的时刻。A 仍直接
+进入最终 head。第一版不使用 bbox、track ID、跨帧 GRU 或双向 AV attention。
+
+训练输出会打印 source frame 数、person observation 数、max_people 和 attention 配置。
+推理只需继续提供 local feature root；融合方式和 attention 参数从 checkpoint 恢复。
+正式比较需要重新训练同 split、seed 和优化设置的 pooled AV 对照，不能直接把不同开发
+协议的旧 checkpoint 当作唯一基线。
+
 ## Ego4D 接口
 
 完整命令见 [Ego4D 四组局部增强实验](ego4d_local_baselines.md)：新增 manifest 批量提取、

docs/progressive_feature_fusion_route_2026-09-02.md

@@ -0,0 +1,268 @@
+# 从 pooled local evidence 到逐人跨模态交融：渐进实验路线（2026-09-02）
+
+## 1. 目的和当前决策
+
+这份路线回答一个具体问题：
+
+> 在严格因果的 egocentric 多人对话中，如何逐步保留和利用可变数量人物的结构，使模型
+> 在相同 wearer 开口召回率下，减少 other-to-other 交谈期间的错误开口，同时不损害真正
+> wearer 接话和让话行为？
+
+当前实现把每帧人物 crop 的 Qwen visual encoder token 先平均为逐人向量，再对人物等权
+平均，最后对最近约 1 秒的有效帧平均。局部音频也被池化，之后与全局 Thinker state G
+拼接进入 MLP。这个基线计算简单，但会丢失：
+
+- 同一帧内有几个人以及哪一个人更相关；
+- 人与人之间的空间关系；
+- 同一匿名人物在多帧中的连续运动和说话状态；
+- 某个人与同期音频、Omni 历史之间的对应关系。
+
+因此，普通 pooling 保留为 F0 基线，不再把继续加深 pooled MLP 当作主方法。第一项新
+融合实验从 F1 Frame-set Attention 开始。后续模块只有在前一级通过预先定义的 gate 后
+才实施。
+
+实现状态（2026-09-02）：F1 的 loader、head、train/infer 路径和回归测试已经加入代码，
+但尚未在远程 PyTorch/L20 环境完成 runtime smoke，不能写成已验证结果。F2--F6 和 D1
+仍然只是计划。
+
+## 2. 不可改变的实验条件
+
+所有阶段共同遵守：
+
+- 目标与全局 state 均为 5 Hz；第 t 行只能使用截止到 (t+1)/5 秒的媒体。
+- 视频人物特征若为 2 FPS，在 5 Hz 预测点选择严格早于 cutoff 的最新人物集合，与已有
+  half-open local audio/visual 契约一致。
+- 不允许用未来帧插值；同一 2 FPS 人物集合可以供多个 5 Hz query 读取，但不能伪装成
+  新观测去重复更新时序状态。
+- EasyCom Session 4--12 为官方训练集，Session 1--3 为最终测试集。模型和阈值选择使用
+  训练集内部 calibration/dev；建议先固定 Session 12，训练使用 Session 4--11。
+- G 是可能已经融合音频、视频和语言上下文的 Thinker state，不能称为纯全局视觉。
+- 局部音频 A 始终直接进入最终预测头。即使它也参与 attention query，也不能只把 A
+  藏在 query 中，否则无法公平保留当前很强的 G+A 基线。
+- 人物数量用 mask 和 NULL person 处理，不通过截断后静默补零改变语义。
+- 所有新增模型与 G+A、pooled-V 以及参数量匹配的非结构化模型使用相同历史长度、
+  optimizer、epoch、seed 和有效样本。
+- 至少运行 3 个 seed，并保存逐事件或逐 session 输出，支持 paired bootstrap。
+
+当前事件 evaluator 尚缺 competing-risk censoring。正式训练选型前必须先完成 P0：
+
+- other offset 后的 response 窗口在下一次 other onset 处截止；
+- other onset 后的 yield 窗口在下一次 other offset 处截止。
+
+P0 修正前的 entry/yield 数值只能称为 timing-derived proxy，不得作为模块通过 gate 的
+最终依据。
+
+## 3. 总路线
+
+| 阶段 | 增量 | 核心问题 | 通过后才进入 |
+|---|---|---|---|
+| F0 | 当前 equal-person/equal-frame pool | 建立可复现起点 | F1 |
+| F1 | Frame-set Attention | 当前对话状态能否选择相关人物 | F2 |
+| F2 | Spatial person interaction | 人物相对位置是否提供额外多人线索 | F3 |
+| F3 | Track-temporal shared encoder | 匿名人物连续性是否必要 | F4 |
+| F4 | Person-conditioned AV history | 每个人读取音频和 Omni 历史是否改善 grounding | F5 |
+| F5 | 双向 person/audio interaction | 更深交互是否优于单向读取 | F6 |
+| F6 | Thinker person-memory adapter | 旁路证据能否安全接入 Omni 内部 | D1 |
+| D1 | 在线 detector + causal tracker | 部署管线与 provided-box 上限的差距 | 完整系统 |
+
+F1 是下一项实现。F2 及以后不并行堆叠，避免无法判断收益来自哪一层。
+
+## 4. F0：冻结现有 pooling 基线
+
+在同一训练协议下至少保留：
+
+| ID | 输入 | 作用 |
+|---|---|---|
+| B0 | G | 全局 Omni 表示 |
+| B1 | G + A | 当前强局部音频基线 |
+| B2 | G + pooled V | 普通视觉池化的边际作用 |
+| B3 | G + A + pooled V | 当前完整 pooling 基线 |
+| B4 | G + A + parameter-matched MLP | 排除收益仅来自更多参数 |
+
+还应保存以下视觉干预：
+
+- remove-V；
+- time-shift-V：只允许向过去错位，绝不能使用未来视觉；
+- wrong-clip-V 或 session 内无关时间的人物集合；
+- correct-V。
+
+如果 correct-V 与错位/错误视觉没有稳定差异，不能把 B3 的变化解释为视觉 grounding。
+
+## 5. F1：Frame-set Attention
+
+### 5.1 输入和数据结构
+
+预测时刻 t 使用：
+
+- G_t：5 Hz 全局 Thinker state；
+- A_t：截止到 t 的因果局部音频表示；
+- P_t = {p_t,i}：最新可用视频帧中 N_t 个逐人 visual encoder 特征；
+- M_t：人物有效 mask；
+- 一个学习的 NULL person，负责没有检测到人物或当前人物均不相关的情况。
+
+F1 不使用 track ID、不使用人物绝对身份 embedding，也不跨帧递归，因此对同一帧的人物
+排列天然不敏感。
+
+### 5.2 最小架构
+
+先把不同来源投影到共同维度 d，首轮建议 d=256、4 heads、单层 attention：
+
+    q_t = W_g LN(G_t) + W_a LN(A_t)
+    K_t = W_k LN([NULL; P_t])
+    V_t = W_v LN([NULL; P_t])
+    U_t = MHA(q_t, K_t, V_t, mask=M_t)
+    logits_t = Head([G_t, A_t, U_t, valid_bits])
+
+这里的语义是：G+A 描述当前 floor 和语境，作为 query 从当前可见人物集合中选择证据。
+人物是 key/value，而不是先被无条件平均。A 仍然直接进入 Head，保证对 B1 的公平比较。
+
+F1 暂不加入 bbox、track GRU、人物自注意力或 audio-to-person 反向 attention；这些都留给
+后续独立消融。
+
+### 5.3 空人物和可变人数
+
+- 始终保留 NULL person，因此全部人物无效时 attention 仍有确定输出。
+- 真实人物用 padding 加 mask 组成 batch；padding token 不得参与 softmax。
+- 不设固定身份 slot。若训练需要 max_people，只允许按确定、与标签无关的规则截断，并
+  报告截断比例。
+- 人物顺序改变后输出应在数值误差范围内不变，作为单元测试。
+
+### 5.4 严格因果映射
+
+对每个 5 Hz cutoff c_t：
+
+1. 找到 frame_time 严格小于 c_t 的最新 local frame；
+2. 读取该帧 frame_offsets 范围内全部逐人 feature；
+3. 如果尚无帧，使用仅含 NULL person 的集合；
+4. 不允许 nearest-neighbor 选到 c_t 之后的帧；
+5. 记录 chosen_frame_time，测试所有 chosen_frame_time 严格小于 cutoff。
+
+### 5.5 F1 必跑对照
+
+| ID | 模型或干预 | 回答的问题 |
+|---|---|---|
+| F1-a | G+A+Frame-set Attention | 主模型 |
+| F1-b | G+A+pooled V | 是否优于现有普通池化 |
+| F1-c | 参数匹配的 learned pooling/MLP | 是否只是参数或非线性增加 |
+| F1-d | correct set 换成 past-shifted set | 是否依赖正确时间 |
+| F1-e | correct set 换成 wrong-clip set | 是否依赖正确画面 |
+| F1-f | 随机排列同一 set | 验证 permutation invariance |
+| F1-g | remove person set，仅 NULL | 视觉是否真正被使用 |
+
+可额外训练逐人 active-speaker probe，但它只诊断人物和声音是否可对齐，不能替代 wearer
+floor-action 结果。
+
+### 5.6 F1 通过条件
+
+必须同时满足：
+
+1. correct F1 在训练内固定 dev 上稳定优于 B3 和参数匹配 F1-c；
+2. correct set 优于 past-shifted、wrong-clip 和 NULL 控制；
+3. 在相同 wearer onset recall 下，third-party false entry/min 下降；
+4. wearer 应接话时的 recall/latency 和 wearer yield 不出现有意义退化；
+5. 三个 seed 方向一致，paired session/event bootstrap 不显示收益完全由单一 session 驱动。
+
+如果只提高逐帧 mAP 而事件指标无变化，则记录为 representation gain，不进入复杂 spatial
+或 track 叙事。若 correct 与错误视觉没有差异，停止增加 attention 层，先审计 crop、
+时间对齐和 person feature。
+
+## 6. F2：加入空间人物交互
+
+只有 F1 通过后才加入：
+
+- 每人的归一化 bbox 几何：中心、宽高和面积；
+- 一层共享 person self-attention；
+- 可选相对几何 bias：dx、dy、宽高比的对数以及中心距离；
+- 再由同一个 G+A floor query 读取增强后的人物集合。
+
+关键对照为 F1、仅绝对 bbox、相对几何 bias 和错误 bbox。只有相对几何版本稳定优于这些
+对照，才可以主张模型利用了多人空间关系；否则保留更简单的 F1。
+
+## 7. F3：匿名 track-temporal 编码
+
+只有空间或 frame-set 证据成立后，才测试跨帧人物连续性：
+
+- 每条在线 track 维护共享参数的 causal GRU 或轻量 TCN；
+- track ID 只用于路由隐状态，不能成为可学习身份；
+- 只有新 2 FPS 帧到达时才更新一次，5 Hz 中间状态只读取最近缓存；
+- track 丢失超过固定 max_age 后重置，所有 reset 规则预先冻结；
+- 全局重命名 track ID 后预测必须不变。
+
+对照包括 frame-set、无 ID temporal pool、正确 track、打断连续性的 track 以及 past-shift。
+正确 track 必须优于无 ID 和打断 track，才能说明身份连续性有价值。
+
+## 8. F4：Person-conditioned audio/Omni history
+
+F4 让每个匿名人物分别读取已有的 5 Hz 因果历史：
+
+    C_A = MHA(H_person, A_history, A_history)
+    C_G = MHA(H_person, G_history, G_history)
+    H_prime = LN(H_person + tanh(alpha_A) C_A + tanh(alpha_G) C_G)
+
+alpha 从 0 初始化，使新分支开始时退化为已验证的人物表示。增强后的 H_prime 再由
+G+A floor query 读取。
+
+这里必须使用一段真实的 5 Hz token history，而不是对单个 pooled audio vector 做形式上
+的 attention。人物级 active-speaker 标签可以作为训练辅助监督，但人物声音标签永远不能
+作为推理输入。
+
+通过条件是 floor event 指标改善，而不只是 active-speaker probe 改善；同时 correct AV
+必须优于 audio-only、visual-only、错位 audio 和错位 visual。
+
+## 9. F5--F6：只在旁路证据充分后增加复杂度
+
+F5 可增加一次 audio query 到人物 memory 的反向读取，形成单块双向 interaction。必须和
+参数匹配 MLP、单向 F4、modality dropout 做比较。若收益不稳定，保留 F4。
+
+F6 才把验证过的人物 memory 通过零初始化 gate 接入 Thinker 上层，并比较：
+
+- 无人物 memory 的参数匹配 Thinker LoRA；
+- 冻结 Thinker 的 person-memory adapter；
+- adapter 加上层 LoRA。
+
+若 F6 不优于旁路，不为追求端到端表述牺牲可解释性，论文采用更简单的旁路模型。
+
+## 10. D1：实际在线检测与追踪
+
+方法验证阶段先使用数据集提供的 bbox/track，隔离融合能力。最终再替换为在线人脸或人物
+检测器加严格因果 tracker，并报告：
+
+- detection recall/coverage；
+- ID switch 和轨迹碎片；
+- max_people 截断比例；
+- P50/P95 单步延迟；
+- provided track 与 online track 的 floor 指标差距。
+
+论文可以把在线 detector/tracker 作为部署前端，但不能把 provided ID 的结果直接称为完整
+在线系统结果。
+
+## 11. 评价和记录规范
+
+P0 冻结后，主指标为：
+
+- third-party false entry/min at matched wearer onset recall，至少报告 R=0.7/0.8/0.9；
+- wearer onset event recall、precision 和 latency；
+- wearer yield event recall、precision 和 latency；
+- addressed-to-wearer recall/latency，仅用于有可靠 addressee 标注的 EasyCom 子集；
+- 0.2--2.0 秒逐帧 AP/mAP，作为 EgoSpeak 可比的次级指标。
+
+每阶段结果放在：
+
+    results/easycom_fusion_fN_<question>_<date>/
+      README.md
+      metrics.json or metrics.csv
+      run configuration and checkpoint references
+
+README 必须写明 split、seed、checkpoint、输入特征版本、评测协议和所有干预条件。最终
+测试集不能反向用于选择 attention 维度、heads、历史长度或 stopping gate。
+
+## 12. 最近执行顺序
+
+1. 完成并测试 P0 competing-risk censoring，人工打印至少一条 response 和 yield 时间轴。
+2. 扩展 local sidecar loader，使其在不破坏现有 pooled 加载路径的情况下返回逐帧人物
+   feature、offset 和 mask；测试成功、失败、空人物和越界路径。
+3. 实现 F1 最小 Frame-set Attention，不加入 bbox 或 track temporal。
+4. 单 session/短片 smoke：检查 shape、cutoff、chosen frame time、mask、空人物和排列
+   不变性。
+5. 在固定 train-internal dev 上运行 B1、B3、F1-a、F1-c、F1-d、F1-e 和 F1-g。
+6. F1 通过后才进入 F2；若未通过，优先审计人物 crop 和 AV 对齐，不继续堆模型。

easycom_head.py

@@ -5,13 +5,16 @@ from __future__ import annotations
 import csv
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Any, Literal, Sequence
+from typing import TYPE_CHECKING, Any, Literal, Sequence
 from uuid import UUID
 
 import numpy as np
 import torch
 from torch import nn
 
+if TYPE_CHECKING:
+    from local_turn_features import FramePersonFeatures
+
 
 OFFICIAL_TRAIN_SESSIONS = (
     "Session_7",
@@ -58,6 +61,7 @@ LAYER_FUSION_MODES = (
     "scalar-mixture",
     "concat-mlp",
 )
+LOCAL_FUSION_MODES = ("pooled", "frame-set-attention")
 DatasetName = Literal["easycom", "ego4d"]
 
 
@@ -77,6 +81,9 @@ class TurnPredictionHead(nn.Module):
         local_visual_dim: int = 0,
         local_audio_dim: int = 0,
         local_projection_dim: int = 256,
+        local_fusion: str = "pooled",
+        person_attention_dim: int = 256,
+        person_attention_heads: int = 4,
     ) -> None:
         super().__init__()
         if num_input_layers <= 0:
@@ -116,6 +123,24 @@ class TurnPredictionHead(nn.Module):
         self.local_visual_dim = local_visual_dim
         self.local_audio_dim = local_audio_dim
         self.local_projection_dim = local_projection_dim
+        if local_fusion not in LOCAL_FUSION_MODES:
+            raise ValueError(
+                f"Unknown local_fusion={local_fusion!r}; expected one of "
+                f"{LOCAL_FUSION_MODES}"
+            )
+        if person_attention_dim <= 0 or person_attention_heads <= 0:
+            raise ValueError("Person attention dimension/heads must be positive")
+        if person_attention_dim % person_attention_heads:
+            raise ValueError("Person attention dimension must be divisible by heads")
+        if local_fusion == "frame-set-attention" and (
+            local_visual_dim <= 0 or local_audio_dim <= 0
+        ):
+            raise ValueError(
+                "Frame-set Attention requires both local visual people and audio"
+            )
+        self.local_fusion = local_fusion
+        self.person_attention_dim = person_attention_dim
+        self.person_attention_heads = person_attention_heads
         if self.learnable_layer_mixture:
             # Zero initialization exactly reproduces a uniform layer mean at
             # step zero while allowing training to select useful query layers.
@@ -133,7 +158,12 @@ class TurnPredictionHead(nn.Module):
             self.layer_fusion_net = None
         self.local_projections = nn.ModuleDict()
         fused_size = hidden_size
-        for name, dimension in (("visual", local_visual_dim), ("audio", local_audio_dim)):
+        projected_modalities = (
+            (("audio", local_audio_dim),)
+            if local_fusion == "frame-set-attention"
+            else (("visual", local_visual_dim), ("audio", local_audio_dim))
+        )
+        for name, dimension in projected_modalities:
             if dimension:
                 self.local_projections[name] = (
                     nn.Sequential(
@@ -145,6 +175,36 @@ class TurnPredictionHead(nn.Module):
                     else nn.Identity()
                 )
                 fused_size += (local_projection_dim or dimension) + 1
+        self.person_input_projection: nn.Module | None = None
+        self.person_state_query: nn.Module | None = None
+        self.person_audio_query: nn.Module | None = None
+        self.person_attention: nn.MultiheadAttention | None = None
+        self.null_person: nn.Parameter | None = None
+        if local_fusion == "frame-set-attention":
+            self.person_input_projection = nn.Sequential(
+                nn.LayerNorm(local_visual_dim),
+                nn.Linear(local_visual_dim, person_attention_dim),
+                nn.GELU(),
+            )
+            self.person_state_query = nn.Sequential(
+                nn.LayerNorm(hidden_size),
+                nn.Linear(hidden_size, person_attention_dim),
+            )
+            self.person_audio_query = nn.Sequential(
+                nn.LayerNorm(local_audio_dim),
+                nn.Linear(local_audio_dim, person_attention_dim),
+            )
+            self.person_attention = nn.MultiheadAttention(
+                person_attention_dim,
+                person_attention_heads,
+                dropout=dropout,
+                batch_first=True,
+            )
+            self.null_person = nn.Parameter(
+                torch.zeros(1, 1, person_attention_dim)
+            )
+            nn.init.normal_(self.null_person, std=0.02)
+            fused_size += person_attention_dim + 1
         self.net = nn.Sequential(
             nn.LayerNorm(fused_size),
             nn.Linear(fused_size, hidden_size // 2),
@@ -190,17 +250,99 @@ class TurnPredictionHead(nn.Module):
 
     @property
     def local_input_size(self) -> int:
+        if self.local_fusion == "frame-set-attention":
+            return self.local_audio_dim + 1
         return sum(d + 1 for d in (self.local_visual_dim, self.local_audio_dim) if d)
 
+    def _frame_set_context(
+        self,
+        states: torch.Tensor,
+        local: torch.Tensor,
+        person_features: torch.Tensor | None,
+        person_mask: torch.Tensor | None,
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        if states.ndim != 2:
+            raise ValueError("Frame-set Attention expects batched [B,D] states")
+        if local.shape != (len(states), self.local_audio_dim + 1):
+            raise ValueError(
+                "Frame-set Attention requires aligned [audio,valid] local features"
+            )
+        if (
+            person_features is None
+            or person_features.ndim != 3
+            or person_features.shape[0] != len(states)
+            or person_features.shape[2] != self.local_visual_dim
+        ):
+            raise ValueError(
+                "Frame-set Attention requires person features [B,P,visual_dim]"
+            )
+        if (
+            person_mask is None
+            or person_mask.dtype != torch.bool
+            or person_mask.shape != person_features.shape[:2]
+        ):
+            raise ValueError("Frame-set Attention requires a boolean person mask [B,P]")
+        if not torch.isfinite(person_features).all():
+            raise ValueError("Person features contain NaN/Inf")
+        assert self.person_input_projection is not None
+        assert self.person_state_query is not None
+        assert self.person_audio_query is not None
+        assert self.person_attention is not None
+        assert self.null_person is not None
+
+        audio_values = local[:, : self.local_audio_dim]
+        audio_valid = local[:, self.local_audio_dim : self.local_audio_dim + 1]
+        audio_projected = self.local_projections["audio"](audio_values) * audio_valid
+        query = (
+            self.person_state_query(states)
+            + self.person_audio_query(audio_values) * audio_valid
+        )
+        people = self.person_input_projection(person_features)
+        null = self.null_person.expand(len(states), -1, -1)
+        memory = torch.cat((null, people), dim=1)
+        null_mask = torch.ones(
+            (len(states), 1),
+            dtype=torch.bool,
+            device=person_mask.device,
+        )
+        memory_valid = torch.cat((null_mask, person_mask), dim=1)
+        context, _ = self.person_attention(
+            query[:, None, :],
+            memory,
+            memory,
+            key_padding_mask=~memory_valid,
+            need_weights=False,
+        )
+        person_valid = person_mask.any(dim=1, keepdim=True).to(states.dtype)
+        return audio_projected, audio_valid, torch.cat(
+            (context[:, 0, :], person_valid),
+            dim=-1,
+        )
+
     def _shared_features(
         self,
         states: torch.Tensor,
         local: torch.Tensor | None = None,
+        person_features: torch.Tensor | None = None,
+        person_mask: torch.Tensor | None = None,
     ) -> torch.Tensor:
         """Return the activity MLP representation shared by both readouts."""
 
         states = self._fuse_layers(states)
-        if self.local_input_size:
+        if self.local_fusion == "frame-set-attention":
+            if local is None:
+                raise ValueError("Frame-set Attention requires local audio features")
+            audio, audio_valid, person_context = self._frame_set_context(
+                states,
+                local,
+                person_features,
+                person_mask,
+            )
+            states = torch.cat(
+                (states, audio, audio_valid, person_context),
+                dim=-1,
+            )
+        elif self.local_input_size:
             if local is None or local.shape != (*states.shape[:-1], self.local_input_size):
                 raise ValueError(
                     f"Head requires aligned local features of width {self.local_input_size}"
@@ -220,14 +362,25 @@ class TurnPredictionHead(nn.Module):
             states = torch.cat(pieces, dim=-1)
         elif local is not None and local.shape[-1] != 0:
             raise ValueError("Local features supplied to a baseline head")
+        if self.local_fusion != "frame-set-attention" and (
+            person_features is not None or person_mask is not None
+        ):
+            raise ValueError("Person sets supplied to a non-frame-set head")
         return self.net[:-1](states)
 
     def forward(
         self,
         states: torch.Tensor,
         local: torch.Tensor | None = None,
+        person_features: torch.Tensor | None = None,
+        person_mask: torch.Tensor | None = None,
     ) -> torch.Tensor:
-        features = self._shared_features(states, local)
+        features = self._shared_features(
+            states,
+            local,
+            person_features,
+            person_mask,
+        )
         logits = self.net[-1](features)
         return logits.view(
             *features.shape[:-1], self.num_horizons, self.num_classes
@@ -237,6 +390,8 @@ class TurnPredictionHead(nn.Module):
         self,
         states: torch.Tensor,
         local: torch.Tensor | None = None,
+        person_features: torch.Tensor | None = None,
+        person_mask: torch.Tensor | None = None,
     ) -> tuple[torch.Tensor, torch.Tensor]:
         """Return activity logits and binary transition logits."""
 
@@ -244,7 +399,12 @@ class TurnPredictionHead(nn.Module):
             raise RuntimeError(
                 "This head was created without transition_auxiliary=True"
             )
-        features = self._shared_features(states, local)
+        features = self._shared_features(
+            states,
+            local,
+            person_features,
+            person_mask,
+        )
         activity_logits = self.net[-1](features).view(
             *features.shape[:-1],
             self.num_horizons,
@@ -734,6 +894,7 @@ class SessionData:
     label_indices: torch.LongTensor
     feature_metadata: dict[str, Any]
     local_features: torch.Tensor | None = None
+    local_person_features: "FramePersonFeatures | None" = None
 
     @property
     def hidden_size(self) -> int:
@@ -768,6 +929,7 @@ def load_session_data(
     local_feature_root: str | Path | None = None,
     local_modalities: str = "av",
     local_contract: dict[str, Any] | None = None,
+    local_fusion: str = "pooled",
 ) -> SessionData:
     """Load and validate one saved Omni feature payload and target array."""
 
@@ -864,22 +1026,55 @@ def load_session_data(
         "".join(str(int(value)) for value in row.tolist()): int(count)
         for row, count in zip(unique_rows, row_counts)
     }
+    if local_fusion not in LOCAL_FUSION_MODES:
+        raise ValueError(f"Unknown local fusion: {local_fusion}")
     local = None
+    local_people = None
     if local_feature_root is not None:
-        from local_turn_features import load_local_features
-
-        local, contract = load_local_features(
-            resolve_feature_path(local_feature_root, name, dataset=dataset),
-            payload,
-            sample_id=name,
+        local_path = resolve_feature_path(
+            local_feature_root,
+            name,
             dataset=dataset,
-            modalities=local_modalities,
-            expected_contract=local_contract,
         )
+        if local_fusion == "frame-set-attention":
+            if local_modalities != "av":
+                raise ValueError(
+                    "Frame-set Attention requires --local-modalities av"
+                )
+            from local_turn_features import load_frame_set_features
+
+            local, local_people, contract = load_frame_set_features(
+                local_path,
+                payload,
+                sample_id=name,
+                dataset=dataset,
+                expected_contract=local_contract,
+            )
+        else:
+            from local_turn_features import load_local_features
+
+            local, contract = load_local_features(
+                local_path,
+                payload,
+                sample_id=name,
+                dataset=dataset,
+                modalities=local_modalities,
+                expected_contract=local_contract,
+            )
         metadata["local_contract"] = contract
     elif local_contract is not None:
         raise ValueError("--local-feature-root is required for this checkpoint")
-    return SessionData(name, states, targets, label_indices, metadata, local)
+    elif local_fusion != "pooled":
+        raise ValueError("Frame-set Attention requires --local-feature-root")
+    return SessionData(
+        name,
+        states,
+        targets,
+        label_indices,
+        metadata,
+        local,
+        local_people,
+    )
 
 
 def average_precision_metrics(
@@ -924,6 +1119,9 @@ def checkpoint_head_config(payload: dict[str, Any]) -> dict[str, Any]:
     config.setdefault("local_visual_dim", 0)
     config.setdefault("local_audio_dim", 0)
     config.setdefault("local_projection_dim", 256)
+    config.setdefault("local_fusion", "pooled")
+    config.setdefault("person_attention_dim", 256)
+    config.setdefault("person_attention_heads", 4)
     config.setdefault(
         "layer_fusion",
         (

infer_turn_head.py

@@ -30,6 +30,7 @@ from easycom_head import (
     output_space_for_loss,
     transition_targets_from_activity,
 )
+from local_turn_features import FramePersonFeatures
 
 
 def parse_args() -> argparse.Namespace:
@@ -118,6 +119,7 @@ def _predict_states(
     show_progress: bool,
     loss_name: str,
     local_features: torch.Tensor | None = None,
+    local_person_features: FramePersonFeatures | None = None,
 ) -> tuple[np.ndarray, np.ndarray | None]:
     chunks: list[torch.Tensor] = []
     transition_chunks: list[torch.Tensor] = []
@@ -139,11 +141,35 @@ def _predict_states(
             local_features[start : start + batch_size].to(device=device, dtype=torch.float32)
             if local_features is not None else None
         )
+        person_batch = None
+        person_mask = None
+        if local_person_features is not None:
+            rows = torch.arange(
+                start,
+                min(start + batch_size, len(states)),
+            )
+            person_batch, person_mask = local_person_features.padded(rows)
+            person_batch = person_batch.to(
+                device=device,
+                dtype=torch.float32,
+                non_blocking=True,
+            )
+            person_mask = person_mask.to(device=device, non_blocking=True)
         if head.transition_auxiliary:
-            activity_logits, transition_logits = head.forward_with_transition(batch, side)
+            activity_logits, transition_logits = head.forward_with_transition(
+                batch,
+                side,
+                person_batch,
+                person_mask,
+            )
             transition_chunks.append(transition_logits.sigmoid().cpu())
         else:
-            activity_logits = head(batch, side)
+            activity_logits = head(
+                batch,
+                side,
+                person_batch,
+                person_mask,
+            )
         chunks.append(
             marginal_probabilities_from_logits(activity_logits, loss_name).cpu()
         )
@@ -403,6 +429,7 @@ def main() -> None:
             local_feature_root=args.local_feature_root,
             local_modalities=local_contract["modalities"] if local_contract else "av",
             local_contract=local_contract,
+            local_fusion=head.local_fusion,
         )
         if data.hidden_size != int(head_config["hidden_size"]):
             raise ValueError(
@@ -423,6 +450,7 @@ def main() -> None:
             show_progress=not args.no_progress,
             loss_name=loss_name,
             local_features=data.local_features,
+            local_person_features=data.local_person_features,
         )
         predictions, valid = _align_to_egospeak_timeline(
             probabilities,
@@ -484,6 +512,7 @@ def main() -> None:
             "checkpoint_train_sessions": checkpoint_payload.get("train_sessions"),
             "checkpoint_dev_sessions": checkpoint_payload.get("dev_sessions"),
             "layer_fusion": head.layer_fusion,
+            "local_fusion": head.local_fusion,
             "local_contract": local_contract,
             "transition_auxiliary": head.transition_auxiliary,
             "transition_objective": checkpoint_payload.get(

local_turn_features.py

@@ -1,4 +1,4 @@
-"""Validated, causal local-feature sidecars for the pooled B1 baseline.
+"""Validated, causal pooled and frame-set local-feature sidecars.
 
 No model/decoder imports: these utilities are also used by CPU head training.
 Sidecars store encoder-native vectors, never a randomly initialized projection.
@@ -6,6 +6,7 @@ Sidecars store encoder-native vectors, never a randomly initialized projection.
 
 from __future__ import annotations
 
+from dataclasses import dataclass
 import math
 from pathlib import Path
 from typing import Any
@@ -16,6 +17,150 @@ LOCAL_FORMAT = "qwen-local-pooled-v1"
 MODALITIES = {"visual": ("visual",), "audio": ("audio",), "av": ("visual", "audio")}
 
 
+@dataclass(frozen=True)
+class FramePersonFeatures:
+    """Ragged person sets indexed by aligned 5 Hz rows without frame duplication."""
+
+    features: torch.Tensor
+    frame_offsets: torch.LongTensor
+    row_to_frame: torch.LongTensor
+    chosen_frame_times: torch.Tensor
+
+    def __post_init__(self) -> None:
+        if (
+            not isinstance(self.features, torch.Tensor)
+            or self.features.ndim != 2
+            or not self.features.is_floating_point()
+            or not torch.isfinite(self.features).all()
+            or self.features.shape[1] <= 0
+        ):
+            raise ValueError("Person features must be finite floating [N,D]")
+        if (
+            not isinstance(self.frame_offsets, torch.Tensor)
+            or self.frame_offsets.dtype != torch.long
+            or self.frame_offsets.ndim != 1
+            or not len(self.frame_offsets)
+            or int(self.frame_offsets[0]) != 0
+            or int(self.frame_offsets[-1]) != len(self.features)
+            or bool((self.frame_offsets[1:] < self.frame_offsets[:-1]).any())
+        ):
+            raise ValueError("Invalid person frame_offsets")
+        if (
+            not isinstance(self.row_to_frame, torch.Tensor)
+            or self.row_to_frame.dtype != torch.long
+            or self.row_to_frame.ndim != 1
+        ):
+            raise ValueError("row_to_frame must be int64 [T]")
+        num_frames = len(self.frame_offsets) - 1
+        if bool(((self.row_to_frame < -1) | (self.row_to_frame >= num_frames)).any()):
+            raise ValueError("row_to_frame contains an invalid frame index")
+        if (
+            not isinstance(self.chosen_frame_times, torch.Tensor)
+            or self.chosen_frame_times.dtype != torch.float64
+            or self.chosen_frame_times.shape != self.row_to_frame.shape
+        ):
+            raise ValueError("chosen_frame_times must be float64 [T]")
+        missing = self.row_to_frame < 0
+        if not torch.equal(torch.isnan(self.chosen_frame_times), missing):
+            raise ValueError("Only rows without an available frame may have NaN frame time")
+        if not torch.isfinite(self.chosen_frame_times[~missing]).all():
+            raise ValueError("Chosen person frame times must be finite")
+
+    def __len__(self) -> int:
+        return len(self.row_to_frame)
+
+    @property
+    def dimension(self) -> int:
+        return int(self.features.shape[1])
+
+    @property
+    def max_people(self) -> int:
+        counts = self.frame_offsets[1:] - self.frame_offsets[:-1]
+        return int(counts.max()) if len(counts) else 0
+
+    def select_rows(self, indices: torch.Tensor) -> "FramePersonFeatures":
+        indices = torch.as_tensor(indices)
+        if indices.dtype != torch.long or indices.ndim != 1:
+            raise ValueError("Person row selection must be int64 [N]")
+        if len(indices) and (int(indices.min()) < 0 or int(indices.max()) >= len(self)):
+            raise IndexError("Person row selection is out of range")
+        return FramePersonFeatures(
+            self.features,
+            self.frame_offsets,
+            self.row_to_frame[indices],
+            self.chosen_frame_times[indices],
+        )
+
+    def padded(
+        self,
+        row_indices: torch.Tensor | None = None,
+    ) -> tuple[torch.Tensor, torch.BoolTensor]:
+        """Materialize only one batch as [B,P,D] plus a True-for-real mask."""
+
+        if row_indices is None:
+            frames = self.row_to_frame
+        else:
+            row_indices = torch.as_tensor(row_indices)
+            if row_indices.dtype != torch.long or row_indices.ndim != 1:
+                raise ValueError("Batch row indices must be int64 [B]")
+            if len(row_indices) and (
+                int(row_indices.min()) < 0 or int(row_indices.max()) >= len(self)
+            ):
+                raise IndexError("Batch person row index is out of range")
+            frames = self.row_to_frame[row_indices]
+        valid_frames = frames >= 0
+        safe_frames = frames.clamp_min(0)
+        starts = torch.zeros_like(frames)
+        counts = torch.zeros_like(frames)
+        if bool(valid_frames.any()):
+            starts[valid_frames] = self.frame_offsets[safe_frames[valid_frames]]
+            counts[valid_frames] = (
+                self.frame_offsets[safe_frames[valid_frames] + 1]
+                - starts[valid_frames]
+            )
+        width = int(counts.max()) if len(counts) else 0
+        mask = torch.arange(width)[None, :] < counts[:, None]
+        if width == 0:
+            return self.features.new_zeros((len(frames), 0, self.dimension)), mask
+        person_indices = starts[:, None] + torch.arange(width)[None, :]
+        safe_indices = person_indices.masked_fill(~mask, 0)
+        values = self.features[safe_indices]
+        return values.masked_fill(~mask[..., None], 0), mask
+
+    @classmethod
+    def concatenate(
+        cls,
+        parts: list["FramePersonFeatures"],
+    ) -> "FramePersonFeatures":
+        if not parts:
+            raise ValueError("Cannot concatenate no person feature sets")
+        dimension = parts[0].dimension
+        if any(part.dimension != dimension for part in parts):
+            raise ValueError("Person feature dimensions differ across sessions")
+        feature_parts = []
+        offset_parts = [torch.zeros(1, dtype=torch.long)]
+        row_parts = []
+        time_parts = []
+        feature_base = 0
+        frame_base = 0
+        for part in parts:
+            feature_parts.append(part.features)
+            offset_parts.append(part.frame_offsets[1:] + feature_base)
+            rows = part.row_to_frame.clone()
+            valid = rows >= 0
+            rows[valid] += frame_base
+            row_parts.append(rows)
+            time_parts.append(part.chosen_frame_times)
+            feature_base += len(part.features)
+            frame_base += len(part.frame_offsets) - 1
+        return cls(
+            torch.cat(feature_parts, dim=0),
+            torch.cat(offset_parts, dim=0),
+            torch.cat(row_parts, dim=0),
+            torch.cat(time_parts, dim=0),
+        )
+
+
 def reference_alignment(payload: dict[str, Any]) -> tuple[torch.Tensor, torch.Tensor]:
     """Require the same full, zero-origin 5 Hz timeline used by head training."""
     states = payload.get("states")
@@ -34,16 +179,15 @@ def reference_alignment(payload: dict[str, Any]) -> tuple[torch.Tensor, torch.Te
     return times, indices
 
 
-def load_local_features(
+def _load_validated_payload(
     path: Path,
     reference: dict[str, Any],
     *,
     sample_id: str,
     dataset: str,
     modalities: str,
-    expected_contract: dict[str, Any] | None = None,
-) -> tuple[torch.Tensor, dict[str, Any]]:
-    """Pack each requested modality as [raw vector, validity bit]. Fail closed."""
+    expected_contract: dict[str, Any] | None,
+) -> tuple[torch.Tensor, torch.Tensor, dict[str, Any], dict[str, Any], dict[str, Any]]:
     if modalities not in MODALITIES:
         raise ValueError(f"Unknown local modalities: {modalities}")
     times, indices = reference_alignment(reference)
@@ -66,6 +210,27 @@ def load_local_features(
     contract = {"format": LOCAL_FORMAT, "modalities": modalities, "config": config}
     if expected_contract is not None and contract != expected_contract:
         raise ValueError(f"Local encoder/pooling contract differs across sessions/checkpoint: {path}")
+    return times, indices, payload, config, contract
+
+
+def load_local_features(
+    path: Path,
+    reference: dict[str, Any],
+    *,
+    sample_id: str,
+    dataset: str,
+    modalities: str,
+    expected_contract: dict[str, Any] | None = None,
+) -> tuple[torch.Tensor, dict[str, Any]]:
+    """Pack each requested modality as [raw vector, validity bit]. Fail closed."""
+    times, _, payload, config, contract = _load_validated_payload(
+        path,
+        reference,
+        sample_id=sample_id,
+        dataset=dataset,
+        modalities=modalities,
+        expected_contract=expected_contract,
+    )
     chunks = []
     for name in MODALITIES[modalities]:
         values = payload.get(name)
@@ -85,6 +250,109 @@ def load_local_features(
     return torch.cat(chunks, dim=-1), contract
 
 
+def load_frame_set_features(
+    path: Path,
+    reference: dict[str, Any],
+    *,
+    sample_id: str,
+    dataset: str,
+    expected_contract: dict[str, Any] | None = None,
+) -> tuple[torch.Tensor, FramePersonFeatures, dict[str, Any]]:
+    """Load direct audio and the latest causal per-frame person set per 5 Hz row."""
+
+    times, _, payload, config, contract = _load_validated_payload(
+        path,
+        reference,
+        sample_id=sample_id,
+        dataset=dataset,
+        modalities="av",
+        expected_contract=expected_contract,
+    )
+    audio = payload.get("audio")
+    audio_valid = payload.get("audio_valid")
+    audio_dim = config.get("audio_dim")
+    if not isinstance(audio_dim, int) or audio_dim <= 0:
+        raise ValueError(f"Missing audio_dim: {path}")
+    if not isinstance(audio, torch.Tensor) or audio.shape != (len(times), audio_dim):
+        raise ValueError(f"Invalid audio shape: {path}")
+    if not audio.is_floating_point() or not torch.isfinite(audio).all():
+        raise ValueError(f"Non-finite/non-floating audio features: {path}")
+    if (
+        not isinstance(audio_valid, torch.Tensor)
+        or audio_valid.dtype != torch.bool
+        or audio_valid.shape != (len(times),)
+    ):
+        raise ValueError(f"Invalid audio_valid mask: {path}")
+    if torch.count_nonzero(audio[~audio_valid]):
+        raise ValueError(f"Invalid audio rows must be stored as zeros: {path}")
+
+    visual_dim = config.get("visual_dim")
+    persons = payload.get("persons")
+    if not isinstance(visual_dim, int) or visual_dim <= 0:
+        raise ValueError(f"Missing visual_dim: {path}")
+    if not isinstance(persons, dict):
+        raise ValueError(f"Missing ragged person features: {path}")
+    features = persons.get("features")
+    frame_times = persons.get("frame_times")
+    frame_offsets = persons.get("frame_offsets")
+    records = persons.get("records")
+    if (
+        not isinstance(features, torch.Tensor)
+        or features.ndim != 2
+        or features.shape[1] != visual_dim
+        or not features.is_floating_point()
+        or not torch.isfinite(features).all()
+    ):
+        raise ValueError(f"Invalid person feature shape/content: {path}")
+    if (
+        not isinstance(frame_times, torch.Tensor)
+        or frame_times.ndim != 1
+        or not frame_times.is_floating_point()
+    ):
+        raise ValueError(f"Invalid person frame_times: {path}")
+    frame_times = frame_times.to(dtype=torch.float64)
+    if (
+        not torch.isfinite(frame_times).all()
+        or bool((frame_times < 0).any())
+        or bool((frame_times[1:] <= frame_times[:-1]).any())
+    ):
+        raise ValueError(f"Person frame_times must be finite, nonnegative and increasing: {path}")
+    if (
+        not isinstance(frame_offsets, torch.Tensor)
+        or frame_offsets.dtype != torch.long
+        or frame_offsets.shape != (len(frame_times) + 1,)
+        or not len(frame_offsets)
+        or int(frame_offsets[0]) != 0
+        or int(frame_offsets[-1]) != len(features)
+        or bool((frame_offsets[1:] < frame_offsets[:-1]).any())
+    ):
+        raise ValueError(f"Invalid person frame_offsets: {path}")
+    if not isinstance(records, list) or len(records) != len(features):
+        raise ValueError(f"Person records/features length mismatch: {path}")
+    if len(frame_times) and float(frame_times[-1]) > float(times[-1]) + 1e-8:
+        raise ValueError(f"Person frame timeline extends beyond the reference: {path}")
+
+    # Match the sidecar's half-open policy: a frame stamped exactly at the
+    # state cutoff is not yet part of that state's observed prefix.
+    row_to_frame = torch.searchsorted(frame_times, times, right=False) - 1
+    chosen = torch.full(times.shape, float("nan"), dtype=torch.float64)
+    has_frame = row_to_frame >= 0
+    chosen[has_frame] = frame_times[row_to_frame[has_frame]]
+    if bool((chosen[has_frame] >= times[has_frame]).any()):
+        raise ValueError(f"Person frame mapping selected future visual evidence: {path}")
+    frame_sets = FramePersonFeatures(
+        features,
+        frame_offsets,
+        row_to_frame.to(dtype=torch.long),
+        chosen,
+    )
+    direct_audio = torch.cat(
+        (audio.float(), audio_valid[:, None].float()),
+        dim=-1,
+    )
+    return direct_audio, frame_sets, contract
+
+
 def local_head_dimensions(contract: dict[str, Any] | None) -> dict[str, int]:
     if contract is None:
         return {"local_visual_dim": 0, "local_audio_dim": 0}

manuscript/main/README.md

@@ -28,3 +28,29 @@ main/
 The author/affiliation, final results, Method, Experimental Setup, Results and
 Conclusion are still explicit TODOs. Do not turn planned experiments into
 reported findings.
+
+The current paper thesis is model-agnostic and deliberately narrower than
+addressee prediction. It asks whether general-purpose omni representations
+expose the fine-grained evidence needed for 200 ms wearer floor-action
+projection, whether PEFT can recover it, and whether causal local audio/person
+evidence provides a reliable complement. The initial planned backbone set is:
+
+- Qwen2.5-Omni-3B;
+- Qwen2.5-Omni-7B;
+- MiniCPM-o 4.5.
+
+The backbone and final fusion method are **not frozen yet**. Current Qwen-3B
+numbers are exploratory motivation, not evidence that every omni model has the
+same limitation. Compare frozen, PEFT, and local-evidence protocols within each
+backbone; report common local encoders separately from model-native local
+encoders. Raw cross-model scores are secondary because the models expose
+different streaming/readout interfaces.
+
+Track-consistent person fusion remains one candidate after pooled and
+capacity-matched controls. If selected, dataset participant IDs route
+short-term causal state only; they are not semantic inputs. The publication
+gate is an event-level reduction in multi-other false entries at matched
+wearer-onset recall without degrading onset latency or wearer-yield behavior.
+A general omni claim requires a consistent result across at least two backbone
+families; otherwise the paper must state a model-specific finding. Selected
+examples are explanatory evidence only.

manuscript/main/REVIEWER_CHECKLIST.md

@@ -4,138 +4,157 @@ This is an internal execution checklist. It is not part of the submitted PDF.
 
 ## Reviewer verdict on the current draft
 
-The research question is potentially publishable: **at matched wearer-response
-recall, can correctly aligned person-level vision reduce false wearer entries
-during third-party conversation without degrading legitimate wearer responses?**
+The potentially publishable question is now model-agnostic: **do
+general-purpose omni representations expose the fine-grained evidence needed
+for 200 ms egocentric floor action, and can causal local evidence reduce false
+wearer entries at matched onset recall when frozen readout or PEFT is
+insufficient?**
 
 The contribution is not “the first audio-visual multiparty turn model.” VAP,
 EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
-gap is continuous, causal floor-entry projection for an **invisible camera
-wearer**, conditioned on addressee evidence from a variable participant set.
+gap is continuous, causal floor-action projection for an invisible camera
+wearer, combined with a controlled diagnosis of global omni states and a
+behavior-aligned multi-other event protocol.
 
-The current aggregate result (G+A+pooled-V minus G+A: +0.30 points at 0.2 s,
-+0.29 mean mAP, one seed) motivates the question but does not establish social
-grounding. Acceptance depends on the event protocol and identity controls.
+Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
+prove a limitation shared by omni models nor establish track-consistent fusion
+as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
+Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
 
-## P0 — Freeze the task before training a new model
+## P0 — Freeze the task before the backbone sweep
 
 - [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
   of the ten 0.2 s horizons.
-- [ ] Freeze addressed-response and third-party event definitions.
-- [ ] Add competing-risk rules: an addressed response ends at the first wearer
-  onset, unrelated-other onset, timeout, or invalid annotation boundary.
+- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
+- [ ] Add competing-risk rules: a candidate event ends at the first matched
+  wearer action, incompatible floor change, timeout, or invalid source boundary.
 - [ ] Specify overlap, minimum silence, maximum response delay, duplicate
-  detections, clip boundaries, group/none/multiple/unknown targets, and
-  offscreen speakers.
+  detections, source discontinuities, track visibility, offscreen speakers,
+  and whether target-of-speech strata are diagnostic only.
 - [ ] Produce train/test event-count tables, duration, class prevalence, and
   visibility coverage before examining model scores.
-- [ ] Mark all already viewed Sessions 1–3 results as exploratory. From this
-  point forward, lock the event protocol and select thresholds, checkpoints,
-  and all hyperparameters using a training-side fold or cross-fitting over
-  Sessions 4–12. Run the fixed final model on Sessions 1–3 only after that
-  lock; do not describe the historically viewed split as untouched.
+- [ ] Mark all already viewed Sessions 1–3 results as exploratory. Lock the
+  protocol and select thresholds, checkpoints, and hyperparameters using a
+  training-side fold or cross-fitting over Sessions 4–12.
 
 **Gate:** two independent code paths or a manual sample audit must agree on at
-least 100 randomly selected events. Do not start the main model sweep before
-this gate passes.
+least 100 randomly selected events. Publish the number of unique other IDs and
+multi-other opportunity minutes before the main sweep.
 
-## P1 — Establish matched baselines
+## P1 — Audit each backbone interface
 
-Run three seeds with identical optimizer, history, parameter budget, and event
-evaluator:
+For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:
 
-1. G: frozen streaming omni state.
-2. G+A: global state plus recent audio.
-3. G+A+V-pool: the current pooled visual baseline.
-4. Temporal control: G+A plus a parameter-matched non-person temporal branch.
+- [ ] exact model/code revision, processor, dtype, and hardware;
+- [ ] native audio/video sampling and streaming/chunk semantics;
+- [ ] exact causal hidden-state readout and dimensionality;
+- [ ] context limit, cache/reset behavior, and source-boundary handling;
+- [ ] a cutoff/impulse test proving no future frame or sample enters the state;
+- [ ] P50/P95 latency, throughput, and peak memory.
 
-Report original ten-horizon AP/mAP and the event metrics. Confirm that changing
-batching, cached features, or track padding does not change baseline outputs.
+Do not force an offline or future-conditioned feature into the common table. If
+a model cannot expose a comparable causal state, report that protocol as
+unsupported. Raw scores are not a pure architecture ranking; emphasize
+within-backbone deltas.
 
-**Gate:** baseline standard deviations are known, and the event evaluator
-reproduces identical scores from saved predictions.
+## P2 — Establish the compact representation matrix
 
-## P2 — Implement the minimum person-set model
+Run three seeds per supported backbone with the same data, physical cutoffs,
+target, prediction-head capacity, optimizer exposure, and evaluator:
 
-- [ ] Shared temporal encoder for every face track.
-- [ ] Each person representation cross-attends to the same recent mixed-audio
-  token sequence; no separated/close-mic audio at inference.
-- [ ] Masked permutation-equivariant set encoder.
-- [ ] Learned OFFSCREEN/NULL element.
-- [ ] Fixed floor/address/third-party role queries.
-- [ ] Zero-initialized gated residual into the frozen global state.
-- [ ] Unit tests for variable `N`, empty visible set, missing frames, track-slot
-  permutation, padding invariance, and strict causal cutoff.
+1. Frozen global causal state.
+2. Pre-registered PEFT/LoRA budget.
+3. Global state plus recent local audio.
+4. Global state plus pooled local audio and vision.
 
-**Gate:** with residual gate zero, logits are numerically identical to G+A; a
-random permutation of track slots leaves logits unchanged.
+Use a common local encoder as the primary cross-backbone control. If native
+local encoders are tested, report them in a separate within-model panel. Do not
+mix common-encoder and native-encoder deltas in one claim.
 
-## P3 — Train only the ablations that answer the claim
+**Gate:** baseline standard deviations are known; the evaluator reproduces
+identical scores from saved predictions; parameter counts and training budgets
+are reported. A single failed LoRA recipe is evidence about that recipe, not
+proof that the model cannot adapt.
 
-Use the same three seeds:
+## P3 — Select local fusion only after P2
 
-1. WearerSet with activity loss only.
-2. WearerSet + per-person active-speaker auxiliary loss.
-3. WearerSet + active-speaker + pure-target addressee auxiliary loss.
+If local evidence gives a reliable event-level gain, compare candidates in
+increasing complexity:
 
-Mask group/unknown/offscreen target annotations from addressee loss. They may
-remain in the activity loss. Do not use LLM pseudo-labels as primary gold.
+1. Pooled local audio/vision.
+2. Parameter-matched temporal branch.
+3. Per-frame person set with no cross-time identity.
+4. Shared track-wise temporal encoder.
+5. Person–audio cross-attention.
+6. Optional per-person active-speaker auxiliary loss, only after selecting a
+   person-aware activity-only model.
 
-Stop after these stages unless a clear event-level gain appears. Broad Thinker
-LoRA, 7B, and ROMA experiments are diagnostics, not the core method, because
-the existing runs show negative transfer or no consistent 0.2 s benefit.
+Keep the dense EgoSpeak wearer/others target as the primary task. Target of
+speech is analysis-only and never model input. Describe only the train-side
+winner as the final method; move rejected candidates to ablations.
 
-## P4 — Run causal visual counterfactuals
+For a track-aware candidate, temporary IDs route hidden state only: no ID
+embedding, cross-recording identity, or fixed seating slot. Unit-test variable
+person count, empty sets, padding, slot permutation, global ID renaming,
+source/gap resets, and strict causal cutoffs.
 
-For the frozen best checkpoint, evaluate:
+## P4 — Run visual counterfactuals conditionally
+
+Only if a track-aware candidate is selected, evaluate one frozen checkpoint
+with:
 
 - correct person tracks;
-- wrong-person tracks from the same scene;
+- clip-consistent global renaming, which must be invariant;
+- deliberately broken cross-time identity with unchanged per-frame crops;
 - time-shifted tracks with unchanged audio;
-- removed/zero visual features;
+- removed visual features;
+- fixed-rate ID switch, box dropout, and fragmentation;
 - visibility-conditioned subsets.
 
-All controls must reuse the same model weights. Retraining each control would
-mix representation and optimization effects.
+The separately trained no-ID model belongs in the main ablation table. Do not
+retrain test-time counterfactuals.
 
-**Acceptance gate:** at matched addressed-response recall, correct vision must
-reduce third-party false entries more than wrong/time-shifted/removed vision,
-without worse response latency or wearer-yield recall. Aggregate mAP alone is
-not sufficient.
+## P5 — Statistics and evidence
 
-## P5 — Statistics and analysis
+- [ ] Three seeds, mean ± sample standard deviation.
+- [ ] Paired cluster bootstrap by session/conversation block for 95% intervals.
+- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
+- [ ] False wearer entries per multi-other opportunity minute, onset latency,
+  wearer-yield recall/latency, and original ten-horizon AP/mAP.
+- [ ] Error strata: backbone, visibility, number of visible people, overlap,
+  offscreen activity, track age/fragmentation, and transition proxy.
+- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
+  same multi-other analysis.
+- [ ] Provided-track and online-tracker conditions named separately.
 
-- [ ] Three training seeds, mean ± sample standard deviation.
-- [ ] Paired cluster bootstrap by session/conversation block for 95% confidence
-  intervals; do not bootstrap adjacent 5 Hz frames independently.
-- [ ] Operating points at addressed recall 0.7, 0.8, and 0.9; headline at 0.8.
-- [ ] Error strata: target visibility, number of visible people, overlap,
-  backchannel/interruption proxy, and offscreen target.
-- [ ] P50/P95 decision latency, throughput, peak GPU memory, and trainable
-  parameters.
-- [ ] Ego4D only as secondary aggregate activity generalization; it cannot
-  validate addressee grounding without comparable target labels.
+Qualitative cases follow a frozen selection rule and include the largest paired
+improvement, a median case, a failure, and one preserved true wearer onset.
+Examples explain rates; they never replace them.
 
 ## Paper assembly order
 
-1. Finalize the event protocol and insert counts in Experimental Setup.
-2. Produce the main matched-recall table before polishing prose.
-3. Produce the correct/wrong/time-shift/remove counterfactual table.
-4. Draw one compact method figure with causal cutoffs and the variable set.
-5. Collapse the preliminary aggregate table into the final main table if page
-   pressure is high.
-6. Add one short failure analysis, not a catalog of all earlier experiments.
-7. Replace the abstract and conclusion TODOs with exact numbers and confidence
-   intervals.
-8. Remove every visible `\todo{}` marker and verify the official ICASSP 2027
-   template/page limit once released.
+1. Freeze and audit the event protocol.
+2. Implement and causally audit all backbone readouts.
+3. Complete the compact frozen/PEFT/local matrix.
+4. Decide whether the paper can support a cross-family or only model-specific
+   representation claim.
+5. Select a local fusion candidate using training-side event metrics.
+6. Run conditional visual/track counterfactuals.
+7. Produce the cross-backbone event table and paired intervals.
+8. Draw one compact final system figure; remove rejected methods from the main
+   narrative.
+9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
 
-## Stop/go decision
+## Stop/go decisions
 
-- **Go:** correct person-level vision provides a statistically credible
-  event-level advantage over G+A, pooled-V, temporal capacity control, and
-  wrong/time-shifted vision.
-- **Reframe:** all visual variants improve similarly. Claim better local
-  temporal context, not addressee grounding.
-- **Stop this method:** aggregate mAP rises but third-party false entries do
-  not fall at matched recall, or legitimate response latency/recall degrades.
+- **General claim:** at least two backbone families show the same frozen/PEFT
+  weakness and a reliable local-evidence improvement.
+- **Model-specific claim:** the effect appears only within Qwen; say so and do
+  not generalize to all omni models.
+- **Audio-local reframe:** local audio improves but all visual variants fail;
+  claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
+- **Track-aware method:** correct continuity beats pooled vision, temporal
+  capacity, no-ID/reset, broken continuity, and time-shifted controls at matched
+  recall without worse onset/yield behavior.
+- **Stop the adapter:** aggregate mAP rises but multi-other false entries do not
+  fall, or legitimate onset/yield behavior degrades.

manuscript/main/main.tex

@@ -7,11 +7,11 @@
 \usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
 
 \newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
-\newcommand{\method}{WearerSet}
 \newcommand{\vect}[1]{\boldsymbol{#1}}
+\setlength{\emergencystretch}{2em}
 
-\title{NOT EVERY OPEN FLOOR IS MINE: ADDRESSEE-CONDITIONED FLOOR-ENTRY
-PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
+\title{WHEN SHOULD AN EGOCENTRIC CONVERSATIONAL AGENT SPEAK? PROBING AND
+ADAPTING OMNI MODELS FOR MULTIPARTY FLOOR ACTION}
 
 % TODO: Insert the final author list and affiliations after the ICASSP 2027
 % review policy and author order are finalized.
@@ -23,55 +23,87 @@ PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
 \maketitle
 
 \begin{abstract}
-An egocentric conversational agent must decide not only when a speaking
-opportunity arises, but also whether that opportunity belongs to it. Existing
-speech-activity projection collapses all non-wearer speakers, allowing strong
-frame-level accuracy to coexist with intrusive entries during conversations
-between bystanders. We formulate addressee-conditioned floor-entry projection:
-dense causal forecasts of wearer and non-wearer activity at 200-ms resolution,
-conditioned on a variable-cardinality set of visible participants. Our model
-augments a frozen streaming omni-model with person-aligned audio--visual
-representations and permutation-invariant role queries for floor state,
-wearer address, and third-party interaction. We further introduce an EasyCom
-event protocol that separates addressed-to-wearer responses from third-party
-exchanges and reports false entries at matched response recall and decision
-latency. Correct-, wrong-person, time-shifted, and removed-video controls test
-whether a gain follows social grounding rather than model capacity.
-\todo{Replace this sentence with the main result: FP/min reduction at 80\%
-response recall, response latency/recall non-degradation, and 95\% confidence
-interval.}
+An egocentric conversational agent operating among several people must decide
+whether to speak or keep listening before it generates a response. EgoSpeak
+operationalizes this problem using recordings from human camera wearers:
+causal first-person audio--video forms the observation, and the wearer's future
+speaking behavior forms the action target. The wearer is thus a data-collection
+surrogate rather than the deployed system. EgoSpeak evaluates
+separately extracted audio--visual features with Transformer, GRU, and Mamba
+temporal models; whether end-to-end pretrained omni models that jointly process
+raw audio and video expose the
+fine-grained cues required for a decision every 200~ms remains unclear. We
+probe representative omni backbones using frozen readout,
+parameter-efficient tuning, and causal side branches that preserve recent
+audio and visible-person evidence. The study tests whether a global omni state
+underrepresents other-person activity and whether local evidence compensates
+without making the agent passive. An EasyCom event protocol measures false
+agent entries during multi-other exchanges at matched recall of wearer-derived
+onset targets. Exploratory Qwen2.5-Omni-3B results motivate this hypothesis but
+do not yet establish a cross-model limitation or determine the final adapter.
+\todo{After the train-side study is locked, name the selected backbones and
+adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction
+at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.}
 \end{abstract}
 
 \begin{keywords}
-turn-taking, speech initiation, addressee detection, egocentric video,
-audio-visual learning
+conversational agents, turn-taking, speech initiation, egocentric video,
+omni models
 \end{keywords}
 
 \section{Introduction}
 \label{sec:intro}
 
-Conversational agents must continuously decide whether to speak or keep
-listening. In a dyadic exchange, a pause or an apparent end of turn is often
-treated as an invitation to respond. This assumption fails in multiparty
-interaction: one participant may finish an utterance addressed to another
-participant, and an egocentric agent that reacts only to acoustic availability
-will enter a floor that was never offered to it. Such an error is socially more
-disruptive than a small framewise prediction error. A useful agent must
-therefore distinguish \emph{when the floor may change} from \emph{whether the
-change is directed to the camera wearer}.
+We consider a conversational agent equipped with an egocentric camera and
+microphone and deployed in a scene where several people may speak. Before
+response generation and speech synthesis, its causal floor controller must
+decide every 200~ms whether to continue listening, begin speaking, or yield. A
+pause in dyadic dialogue may invite a response; in multiparty interaction it
+may instead mark a handover between two other people. An agent that reacts only
+to acoustic availability can therefore interrupt a conversation that it
+should continue observing.
 
-Voice Activity Projection (VAP) models future joint speech activity and
-derives turn-taking events without manually labeling each event
-\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
-untrimmed egocentric video, forecasting wearer and aggregate non-wearer
-activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
-output is attractive for a wearable agent, but an ``others'' channel cannot
-distinguish an utterance addressed to the wearer from an exchange between
-bystanders. Conventional per-frame average precision can consequently be
-dominated by ordinary activity while hiding precisely the false-entry behavior
-that visual social cues should prevent. Event-oriented evaluation likewise
-emphasizes recall, false alarms, and latency rather than a single frame-level
-score \cite{jiang2026turnbench}.
+Existing datasets do not contain autonomous agents making these decisions at
+scale. Following EgoSpeak \cite{kim2025egospeak}, we use human egocentric
+recordings to construct supervision: the camera wearer defines the first-person
+sensor origin, and the wearer's observed speaking behavior defines the desired
+floor-action target. This is an imitation-learning proxy for a future agent
+under comparable observations. Throughout the paper, \emph{wearer} names this
+dataset role and target channel; it is neither the deployed agent nor a
+semantic model input. Response content and closed-loop effects of deploying an
+agent remain outside our scope.
+
+EgoSpeak already establishes real-time speech-initiation prediction in causal,
+untrimmed egocentric video. It combines separately pretrained ResNet-50 and
+wav2vec2 feature extractors with Transformer/LSTR, single-layer GRU, and Mamba
+temporal models, forecasting the camera wearer and aggregate other-person
+activity over the next two seconds \cite{kim2025egospeak}. Thus our novelty is
+not the task definition or the use of first-person data. The unresolved model
+question is whether an end-to-end pretrained audio--visual omni backbone,
+operating on causal raw media, exposes an adequate state for this dense
+floor-action controller. To our knowledge, such omni representations have not been
+systematically evaluated or adapted for the EgoSpeak task.
+
+Our working hypothesis is specific. An omni model may retain broad scene and
+speech semantics while its global causal state makes recent prosody, activity
+boundaries, and individual visible-speaker motion difficult for a shallow
+5-Hz head to recover. This would appear as inaccurate prediction of
+other-person activity and poorly timed agent entries, especially in
+multiparty exchanges. A causal local side branch may compensate by preserving
+recent audio and anonymous person-level evidence before fusion with the global
+state. This is a hypothesis to test, not an assumed omni-model deficiency:
+frozen, parameter-efficiently tuned, and locally augmented variants must be
+compared within each backbone.
+
+Voice Activity Projection (VAP) provides the broader future-activity view of
+turn taking \cite{ekstedt2022vap}. EgoSpeak's role-relative output preserves
+compatibility with that view, but its aggregate ``others'' channel discards
+whether the same or different visible people produced recent activity.
+Moreover, conventional per-frame average precision can hide the behavioral
+failure of entering during an exchange between others. We therefore retain the
+dense EgoSpeak target while adding event evaluation of agent entry and yield;
+event-oriented work likewise emphasizes recall, false alarms, and latency
+rather than a single frame-level score \cite{jiang2026turnbench}.
 
 Related work supplies complementary parts of the problem. Egocentric
 talking-to-me detection predicts whether a completed utterance addresses the
@@ -79,46 +111,48 @@ camera wearer and benefits from long-term social context
 \cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
 tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
 The former is generally utterance-level rather than continuous anticipation;
-the latter does not ask whether an invisible camera wearer should enter. Our
+the latter predicts visible participants rather than the action of an
+egocentric conversational agent represented by a wearer-derived target. Our
 gap is therefore narrower than ``audio--visual multiparty turn taking'': we
-study continuous, wearer-relative floor action when the candidate responder is
-behind the camera and the number of visible participants changes over time.
+study whether omni representations support direct, continuous agent
+floor-action prediction and whether causal local evidence helps when the
+visible participant set changes.
 
-We call this task \emph{addressee-conditioned egocentric floor-entry
-projection}. A global streaming audio--video representation supplies broad
-context, while a set branch preserves individual visual histories and their
-alignment to recent mixed audio. Fixed semantic role queries compress a
-variable participant set into wearer-address and third-party evidence without
-assuming a fixed seating order. EasyCom is suitable because it provides
-synchronized egocentric audio--video, per-person tracks, voice activity, and
-target-of-speech annotations \cite{donley2021easycom}. Target annotations are
-used only for training-side supervision and evaluation strata, never as input
-at test time.
+We call this task \emph{egocentric floor-action projection}. The study asks
+four questions: (RQ1) how accurately a frozen global omni state predicts the
+wearer-derived agent target and other-person activity; (RQ2) whether standard
+parameter-efficient tuning recovers missing performance; (RQ3) whether causal
+local audio and person-level visual evidence provide complementary
+information; and (RQ4) whether any aggregate gain reduces false agent entries
+without making the controller globally passive. EasyCom is suitable because
+it provides synchronized egocentric audio--video, per-person tracks, and voice
+activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
+analysis strata only and are never model input or required output.
 
-Our contributions are threefold. (1) We formulate continuous speech
-initiation as recipient-aware floor-entry projection for an invisible camera
-wearer. (2) We propose a causal set-based fusion module that grounds mixed
-audio in a variable number of visible participants and preserves the original
-dense EgoSpeak target. (3) We introduce an event evaluation that reports
-third-party false entries at matched addressed-response recall, accompanied by
-latency and visual counterfactual controls. This design makes the scientific
-claim falsifiable: visual fusion is useful only if \emph{correctly aligned}
-vision improves the operating trade-off without suppressing legitimate wearer
-responses.
+The intended contributions are therefore: a controlled cross-backbone study
+of pretrained omni representations for EgoSpeak-style floor action; a
+lightweight local-evidence adaptation selected from
+pre-registered candidates; and a behavior-aligned event evaluation that maps
+wearer demonstrations to agent entry and yield decisions. Current
+Qwen2.5-Omni observations are motivation, not a claim that all omni models
+intrinsically fail. Likewise, track-consistent cross-attention becomes the
+final method only if it beats pooled, no-ID, and capacity-matched controls.
 
 \begin{figure}[t]
 \centering
 \fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
-\todo{Draw the final method figure: causal omni state $\vect g_t$; recent
-mixed-audio tokens; a masked set of face-track histories; shared per-person
-audio--visual grounding; three role queries; zero-initialized residual; and
-the $10\times3$ activity head. Mark all causal cutoffs and masks.}}}
-\caption{Proposed wearer-relative set fusion. The number and ordering of
-visible participants may change, while the output remains fixed-size.}
+\todo{After model selection, draw the final system figure. For now it must show
+the shared experiment interface: raw causal AV, backbone-specific global
+readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous
+person histories, and the common $10\times3$ head. Mark causal cutoffs,
+trainable modules, parameter budgets, and source-boundary resets.}}}
+\caption{Planned omni-model study for egocentric conversational floor action.
+The local adapter remains a candidate family until the controlled train-side
+comparison is complete.}
 \label{fig:method}
 \end{figure}
 
-\section{Task and Method}
+\section{Methods}
 \label{sec:method}
 
 \subsection{Wearer-relative activity projection}
@@ -141,74 +175,109 @@ windows; elsewhere it remains a future-activity probability. This retains the
 EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
 signal.
 
-\subsection{Frozen global stream and local participant set}
+\subsection{Omni backbones and representation protocols}
 
-We obtain a causal global state $\vect{g}_t$ from the streaming Thinker of
-Qwen2.5-Omni \cite{xu2025qwenomni}. The backbone is frozen in the main
-experiment because preliminary LoRA adaptation caused negative transfer
-(Sec.~\ref{sec:results}). From the most recent $L_a$ seconds, a shared audio
-encoder produces token sequence $\vect{A}_t$. For each visible track
-$i\in\{1,\ldots,N_t\}$, a shared temporal visual encoder produces
-$\vect{v}_{t,i}$ from its recent face history. Person-aligned evidence is
+Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a
+backbone-specific causal global readout at the same physical cutoff $t$.
+Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family;
+Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5
+\cite{cui2026minicpmo45} form the planned initial comparison. These models have
+different native streaming APIs, temporal tokenization, hidden dimensions, and
+adaptation interfaces. We therefore emphasize within-backbone deltas rather
+than treating raw cross-model scores as a pure architecture ranking.
+
+Every supported backbone is evaluated under three representation protocols:
+(i) a frozen global readout with a dimension-matched prediction head;
+(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the
+same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak
+targets, context duration, training examples, head capacity, and evaluation
+are held fixed whenever the backbone permits it. A failure of one LoRA recipe
+is reported as that recipe's result, not as proof that the backbone cannot
+adapt.
+
+\todo{Implement and audit one adapter for each planned backbone. For every
+model, document model revision, processor, native audio/video rates, exact
+hidden-state readout, causal/streaming semantics, context limit, dtype,
+trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
+Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
+expose a comparable causal state, report it as unsupported rather than inventing
+a non-equivalent readout. Add future backbones only before the experiment
+protocol is locked.}
+
+\subsection{Candidate local-evidence adaptation}
+
+The current hypothesis is that global omni states preserve broad semantic
+context but may make recent prosody, activity boundaries, and individual
+visual motion difficult for a shallow floor head to recover. From the most
+recent $L_a$ seconds, a local encoder produces audio evidence $\vect A_t$ and,
+optionally, an anonymous visible-person set $\mathcal P_t$. A generic candidate
+adapter is
 \begin{equation}
- \vect{p}_{t,i}=\vect{v}_{t,i}+
- \mathrm{CA}(\vect{v}_{t,i},\vect{A}_t,\vect{A}_t),
- \label{eq:person}
+ \vect z^{(m)}_t=\vect g^{(m)}_t+
+ \gamma_t H_\phi(\vect g^{(m)}_t,\vect A_t,\mathcal P_t),
+ \qquad \gamma_t\vert_{\mathrm{init}}=0,
+ \label{eq:adapter}
 \end{equation}
-so every participant queries the same mixed audio rather than requiring
-unavailable separated microphones. A masked set encoder models interactions
-among $\mathcal{P}_t=\{\vect{p}_{t,i}\}_{i=1}^{N_t}$ and is permutation
-equivariant \cite{lee2019settransformer}. Missing detections use a learned
-\textsc{offscreen/null} element. In minibatches, sets are padded to
-$N_{\max}$ with an attention mask; no slot-ID embedding is used, and track
-slots are randomly permuted during training.
+followed by the common prediction head. Local evidence is derived from the
+same causal raw media, not close microphones, transcripts, speaker labels, or
+future frames.
 
-\todo{Implement and verify the shared track encoder, mixed-audio
-cross-attention, OFFSCREEN token, padding mask, and numerical invariance under
-track-slot permutation. Freeze $L_a$, visual crop history, $N_{\max}$, and
-face-track confidence filtering from train-side experiments only.}
+We evaluate candidates in increasing structural complexity: pooled recent
+audio/vision; a capacity-matched temporal branch; a per-frame person set with
+no cross-time identity; a shared track-wise temporal encoder; and person--audio
+cross-attention. Only a candidate that improves the pre-registered event metric
+over simpler and capacity-matched controls becomes the final method. If
+person continuity is selected, temporary IDs route hidden state only---there
+is no ID embedding, name recognition, or cross-recording identity---and the set
+operator is permutation equivariant \cite{lee2019settransformer}.
 
-\subsection{Role-query fusion}
-
-Mean pooling can erase which participant is speaking to whom. Instead, three
-learned semantic queries represent floor state, address-to-wearer evidence,
-and third-party interaction. Each query is conditioned on $\vect{g}_t$ and
-cross-attends to the participant set:
-\begin{align}
- \vect{Q}_t &= [\vect{q}_{\mathrm{floor}};
-               \vect{q}_{\mathrm{addr}};
-               \vect{q}_{\mathrm{third}}]+W_g\vect{g}_t,\\
- \vect{R}_t &= \mathrm{MHA}(\vect{Q}_t,\widetilde{\mathcal P}_t,
-                            \widetilde{\mathcal P}_t),\\
- \vect{z}_t &= \vect{g}_t+\tanh(\alpha)W_r\mathrm{vec}(\vect{R}_t),
- \label{eq:fusion}
-\end{align}
-where $\alpha$ is initialized to zero. The zero-initialized residual makes the
-starting model identical to the frozen global baseline and allows the visual
-branch to be removed exactly. A two-layer prediction head maps $\vect{z}_t$ to
-Eq.~\eqref{eq:task}.
+\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
+parameters and effective local history. Compare a shared local encoder across
+backbones (cross-model control) with each backbone's native local encoder
+(within-model control); do not mix these protocols in one claimed delta.
+For track candidates, specify visibility/age masks, gap/source resets,
+OFFSCREEN handling, and global-rename invariance. Describe only the selected
+architecture as the final method; move rejected candidates to ablations.}
 
 \subsection{Training objectives}
 
-The main loss is class-balanced binary cross entropy over all horizons and
-activity channels. When annotations are available, auxiliary heads predict
-per-track active-speaker state and whether the current source addresses the
-wearer:
+The common main loss is class-balanced binary cross entropy over all horizons
+and activity channels. This dense EgoSpeak target is the only supervision
+required by the task. If a person-aware candidate survives the activity-only
+comparison, an optional auxiliary head predicts current active-speaker state:
 \begin{equation}
  \mathcal{L}=\mathcal{L}_{\mathrm{act}}+
- \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}+
- \lambda_{\mathrm{addr}}\mathcal{L}_{\mathrm{addr}}.
+ \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}.
  \label{eq:loss}
 \end{equation}
-Ambiguous, group-addressed, and offscreen-target samples are masked from
-$\mathcal{L}_{\mathrm{addr}}$, but remain in activity training. No language
-model or pseudo-label is used to define primary test events.
+Participant ID, target-of-speech, transcript, close-microphone audio, and
+future frames are never model inputs. Target-of-speech may stratify qualitative
+analysis but is not a predicted variable or a requirement for the primary
+multi-other event definition.
 
-\todo{Run the staged ablation: activity only; +ASD; +addressee. Select
-$\lambda$ values and early stopping on a training-side calibration fold. Do
-not inspect Sessions 1--3 during selection.}
+\todo{First compare all representation/adaptation protocols with activity loss
+only. Then ablate +ASD for the selected person-aware candidate. Add an
+event-specific wearer onset/yield loss only as a separately reported extension.
+Select all coefficients and early stopping on a training-side calibration
+fold; do not inspect Sessions 1--3 during selection.}
 
-\section{Experimental Setup}
+\subsection{Track source and deployment contract}
+
+Person-aware candidates first use dataset-provided boxes and anonymous IDs to
+isolate the value of local visual structure. A deployable version instead
+obtains causal tracklets from a face/head detector followed by an online
+multi-object tracker; no future-frame smoothing or cross-session recognition
+is permitted. EasyCom IDs are continued only across genuinely adjacent source
+segments and reset across redacted gaps. Ego4D IDs are local to a clip.
+
+\todo{If a person-aware candidate is selected, state precisely whether face or
+head boxes are used. Report provided-track
+coverage, mean/maximum people, occlusion gaps, track fragments, and source-gap
+resets for both datasets. Add either one online detector/tracker experiment or
+controlled box jitter, dropout, fragmentation, and ID-switch curves; otherwise
+limit every deployment claim to ``given causal participant tracklets.''}
+
+\section{Experiments}
 \label{sec:setup}
 
 \textbf{Data and splits.}
@@ -216,45 +285,83 @@ We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
 Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
 and hyperparameter choices use training-side folds only. Inputs contain the
 camera stream and shared array audio available to the wearer; close-talking
-microphones and target-of-speech labels are supervision only. We also report aggregate
-activity forecasting on the EgoSpeak Ego4D split \cite{grauman2022ego4d}, but
-do not claim addressee grounding there because comparable target annotations
-are absent.
+microphones, per-person voice labels, and target-of-speech labels are not model
+inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D
+split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies
+clip-local anonymous participant tracks; its per-person voice coverage on the
+433-clip EgoSpeak split must be audited before reporting the same multi-other
+event analysis.
+
+\textbf{Backbone comparison.}
+The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
+MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
+$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence
+augmentation $G_m+L$. The primary diagnostic is the within-backbone change;
+cross-backbone absolute scores are secondary because native tokenization,
+pretraining, streaming interfaces, and hidden-state semantics differ. Common
+local encoders test whether adaptation transfers across global backbones,
+whereas native local encoders are a separate, model-specific protocol. All
+rows share physical input cutoffs, targets, data splits, and evaluation.
+Unsupported causal readouts are disclosed rather than replaced by offline or
+future-conditioned features.
+
+\todo{Implement the three backbone adapters and validate state timestamps with
+an impulse/cutoff test. Pre-register one matched PEFT budget and one common
+local branch. Report native-vs-common local encoders in separate table panels.
+Keep future model additions outside the confirmatory comparison once this
+matrix is locked.}
 
 \textbf{Event construction.}
-An \emph{addressed response} starts from a pure other-to-wearer utterance and
-requires a wearer onset before an unrelated other onset. A \emph{third-party
-exchange} starts from a pure other-to-other target while the wearer remains
-silent. Group, none, multiple, and unknown targets form a separate diagnostic
-stratum. Threshold $\tau_r$ is selected on the development split to obtain
-addressed-response recall $r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary
-metric is false wearer entries per minute of third-party opportunity at
-matched recall; the headline operating point is $r=0.8$. We also report
-addressed-response recall, signed detection latency, wearer yield recall, and
-the original ten-horizon per-frame mAP.
+We derive \emph{wearer onset} events directly from the wearer activity target.
+A \emph{multi-other exchange} is a wearer-silent opportunity interval in which
+at least two distinct non-wearer participant IDs contribute temporally ordered
+speech activity. It is an observable interaction proxy, not a claim about the
+speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the
+development split to obtain wearer-onset recall
+$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer
+entries per minute of multi-other opportunity at matched onset recall; the
+headline operating point is $r=0.8$. We also report signed onset latency,
+wearer-yield recall/latency, and the original ten-horizon per-frame mAP.
 
-\todo{Freeze the event pairing and competing-risk rules, including pre/post
-windows, minimum silence, maximum response delay, overlap handling, and one
-detection per event. Publish counts for train/test positive and negative
-events, visibility, duration, and excluded target types. Treat all previously
-viewed Sessions 1--3 scores as exploratory; lock the final protocol and model
-before the confirmatory three-seed run.}
+\todo{Freeze the event pairing and competing-risk rules, including the
+multi-other alternation window, minimum speech/silence, overlap, maximum onset
+delay, duplicate detections, one detection per event, track visibility, source
+boundaries, and group/offscreen cases. Publish train/test onset counts,
+multi-other events/minutes, number of distinct visible speakers, and exclusion
+counts. Manually audit at least 100 events. Treat all previously viewed
+Sessions 1--3 scores as exploratory; lock the protocol and model before the
+confirmatory three-seed run.}
 
 \textbf{Baselines and controls.}
-We compare: global omni state (G); G plus recent audio (G+A); G+A plus pooled
-visual context (G+A+V-pool); a parameter-matched temporal branch without
-person identity; and the proposed set branch. Required counterfactuals replace
-correct tracks with (i) wrong participants from the same scene, (ii)
-time-shifted tracks, and (iii) zeros. These controls share parameter count,
-audio history, and optimization budget.
+For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
+plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
+This compact matrix tests the representation hypothesis before investing in a
+complex fusion module. If local vision survives three seeds and the event
+gate, we then compare a parameter-matched temporal branch without person
+continuity, a separately trained per-frame/no-ID set branch, and a
+track-consistent branch. On a frozen track-aware checkpoint, a clip-consistent
+global renaming must leave outputs unchanged, whereas \emph{broken
+continuity} periodically remaps IDs while preserving every crop, timestamp,
+audio sample, and target. Time-shifted and removed vision provide additional
+alignment controls.
+
+\todo{Define the broken-continuity corruption before evaluation: remap only at
+pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
+call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
+frame counts, audio, labels, and visibility masks. Train the no-ID/reset
+baseline from scratch with matched parameters; evaluate all test-time
+counterfactuals using one frozen checkpoint and one frozen threshold.}
 
 \textbf{Implementation and statistics.}
-All models run at 5~Hz with an 8-s causal memory. The 3B Thinker is frozen and
-local audio/face histories are encoded causally. We report three seeds and
-mean$\pm$standard deviation. Confidence intervals for event metrics use paired
-cluster bootstrap over sessions/conversation blocks rather than treating
-adjacent frames as independent. Runtime includes P50/P95 decision latency,
-peak memory, and trainable parameter count.
+All prediction heads run at 5~Hz and use an 8-s causal observation window;
+model-specific deviations required by a native streaming interface are stated
+explicitly. In the frozen protocol the omni backbone does not update. In PEFT
+and local protocols, trainable parameters and optimizer exposure are reported
+separately. Every temporal state resets at non-contiguous source boundaries.
+We report three seeds and mean$\pm$standard deviation. Confidence intervals for
+event metrics use paired cluster bootstrap over sessions/conversation blocks
+rather than treating adjacent frames as independent. Runtime includes P50/P95
+decision latency, throughput, peak memory, and trainable parameter count.
 
 \todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
 $L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
@@ -264,95 +371,138 @@ count, hardware, and real-time latency after the architecture is frozen.}
 \label{sec:results}
 
 \begin{table}[t]
-\caption{Current aggregate evidence on EasyCom test Sessions 1--3. Values are
-percentages from one seed and are motivation, not the final main result.}
+\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
+Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
+confirmatory result.}
 \label{tab:preliminary}
 \centering
 \small
+\resizebox{\columnwidth}{!}{%
 \begin{tabular}{lccc}
 \hline
 Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
 \hline
-G+A             & 81.72 & 85.07 & 65.11\\
-G+A+V-pool      & 82.03 & 85.09 & 65.40\\
+Qwen-3B $G$+local A       & 81.72 & 85.07 & 65.11\\
+Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
 \hline
 $\Delta$ visual & +0.30 & +0.02 & +0.29\\
 \hline
-\end{tabular}
+\end{tabular}}
 \end{table}
 
 Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
 Pooled vision gives a small gain, while immediate wearer AP is nearly
 unchanged. This is compatible with either weak but useful social grounding or
 generic visual regularization; only matched-recall event evaluation and
-counterfactual identity controls can distinguish them.
+track-continuity controls can distinguish them.
+A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
+readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
+feature protocol, it is not combined numerically with
+Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
+study remains necessary.
 
 \begin{table}[t]
-\caption{Primary EasyCom event evaluation. Lower $\mathrm{FP}_{\mathrm{3p}}$
-is better; all rows use the same development-selected operating protocol.}
+\caption{Planned cross-backbone EasyCom event evaluation. Lower
+$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
+development-selected operating protocol. ``Local'' denotes the train-side
+selected common local-evidence candidate.}
 \label{tab:main}
 \centering
 \small
 \resizebox{\columnwidth}{!}{%
 \begin{tabular}{lccccc}
 \hline
-Model & $\mathrm{FP}_{\mathrm{3p}}$/min@$R=.8$ & $R_{\mathrm{addr}}$ &
-Latency & Yield R & Mean mAP\\
+Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R &
+Onset latency & Yield R & Mean mAP\\
 \hline
-G                 & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-G+A               & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-G+A+V-pool        & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-Temporal control  & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-\method{}         & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-\method{}+ASD     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
-\method{}+Addr    & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Qwen-3B frozen       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Qwen-3B PEFT         & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Qwen-3B + local      & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Qwen-7B frozen       & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+Qwen-7B PEFT/local   & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+MiniCPM-o frozen     & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
+MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
 \hline
 \end{tabular}}
 \end{table}
 
 \todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
-seeds, plus 95\% paired cluster-bootstrap confidence intervals for the
-proposed-vs-G+A difference. The paper's main claim is supported only if
-third-party FP/min decreases at matched recall without worse response latency
-or wearer-yield recall.}
+seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs
+PEFT and frozen vs local within each backbone. A general omni-representation
+claim requires a consistent pattern across at least two backbone families; an
+effect confined to Qwen must be stated as Qwen-specific. The behavior claim is
+supported only if multi-other FP/min decreases at matched wearer-onset recall
+without worse onset latency or wearer-yield recall. If the selected local
+candidate uses tracks, also report its paired intervals against no-ID and
+temporal-capacity controls.}
 
 \begin{table}[t]
-\caption{Visual grounding counterfactual on the frozen best model.}
+\caption{Conditional track-continuity counterfactual on the best track-aware
+model. This table is retained only if train-side selection chooses such a
+candidate; all rows reuse one frozen checkpoint.}
 \label{tab:counterfactual}
 \centering
 \small
+\resizebox{\columnwidth}{!}{%
 \begin{tabular}{lccc}
 \hline
-Visual evidence & $\mathrm{FP}_{\mathrm{3p}}$/min & Addr. recall & 0.2-s mAP\\
+Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\
 \hline
 Correct tracks    & \todo{--} & \todo{--} & \todo{--}\\
-Wrong person      & \todo{--} & \todo{--} & \todo{--}\\
+Global ID rename  & \todo{--} & \todo{--} & \todo{--}\\
+Broken continuity & \todo{--} & \todo{--} & \todo{--}\\
 Time shifted      & \todo{--} & \todo{--} & \todo{--}\\
 Removed           & \todo{--} & \todo{--} & \todo{--}\\
 \hline
-\end{tabular}
+\end{tabular}}
 \end{table}
 
-Table~\ref{tab:counterfactual} is the identity test. A true social-grounding
-effect should be largest for correct tracks and should collapse for wrong or
-time-shifted people. If every visual variant improves equally, the defensible
-conclusion is additional capacity/history, not addressee grounding.
+If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the
+continuity test. Global renaming must be
+numerically invariant because IDs carry no semantics. If the model exploits
+person history, correct tracks should outperform broken continuity even though
+both contain exactly the same per-frame people. If broken, time-shifted, and
+removed variants perform similarly to correct tracks, the defensible
+conclusion is additional capacity or generic visual context, not useful
+person-continuous reasoning.
 
-\todo{Add one compact error analysis split by target visibility, number of
-visible people, overlap, and backchannel/interruption proxy. Include event
-counts with every rate. Move negative LoRA/7B/ROMA diagnostics to supplementary
-material unless they directly explain the final method.}
+\todo{Add one compact error analysis split by model family, track visibility,
+number of visible people, overlap, track age/fragmentation, and
+backchannel/interruption proxy. Include event counts with every rate. Move
+rejected PEFT recipes and fusion candidates to supplementary material unless
+they directly explain the selected method.}
+
+\textbf{Qualitative evidence.}
+For a multi-other exchange, visualize per-person speech activity and track IDs,
+wearer ground truth, the frozen-global and selected-local wearer-entry scores,
+and the single frozen threshold on one physical time axis. If the selected
+candidate is track-aware, also show its broken-continuity score. A useful
+example shows the frozen baseline rising at an other-speaker boundary while
+the selected adapter remains below threshold; a companion positive example
+must show that a real wearer onset is not suppressed.
+
+\todo{Do not hand-pick only successful clips. Before viewing the final test
+plots, freeze a deterministic selection rule and show: largest paired
+improvement, near-median improvement, one failure, and one correctly retained
+wearer onset. Include full local timelines and source frame times, not cropped
+screenshots without context. State that examples illustrate the aggregate
+event result and cannot establish it alone.}
 
 \section{Conclusion}
 \label{sec:conclusion}
 
-We study when an egocentric agent should enter a multiparty floor, not merely
-whether someone will speak. The proposed formulation retains dense future
-activity prediction while grounding wearer entry in a variable set of visible
-participants. Its event protocol tests the intended behavior directly:
-reducing intrusions into third-party exchanges at matched legitimate-response
-recall. \todo{Replace with one quantitative conclusion and one limitation;
-do not claim success from aggregate mAP alone.}
+We study when an egocentric agent should enter or yield a multiparty floor, not
+merely whether someone will speak. The central question is model-agnostic:
+whether frozen or parameter-efficiently adapted omni representations expose
+the fine-grained floor cues required by a 5-Hz controller, and whether causal
+local evidence closes any gap. Dense activity forecasting supplies a common
+target, while matched-recall event evaluation tests whether an apparent gain
+actually prevents intrusions during multi-other exchanges. Current Qwen-3B
+results are exploratory motivation only. \todo{After the controlled study,
+state which backbone/adapter was selected, whether the representation pattern
+generalized beyond Qwen, the quantitative false-entry and onset/yield results,
+and the main deployment limitation. Do not infer success from aggregate mAP or
+selected examples alone.}
 
 \bibliographystyle{IEEEbib}
 \bibliography{references}

manuscript/main/references.bib

@@ -48,6 +48,13 @@
   year    = {2025}
 }
 
+@article{cui2026minicpmo45,
+  author  = {Junbo Cui and Bokai Xu and Chongyi Wang and Tianyu Yu and Weiyue Sun and Yingjing Xu and Tianran Wang and Zhihui He and Wenshuo Ma and Tianchi Cai and others},
+  title   = {{MiniCPM-o 4.5}: Towards Real-Time Full-Duplex Omni-Modal Interaction},
+  journal = {arXiv preprint arXiv:2604.27393},
+  year    = {2026}
+}
+
 @inproceedings{lee2019settransformer,
   author    = {Juho Lee and Yoonho Lee and Jungtaek Kim and Adam Kosiorek and Seungjin Choi and Yee Whye Teh},
   title     = {Set Transformer: A Framework for Attention-Based Permutation-Invariant Neural Networks},

test_local_turn_features.py

@@ -28,8 +28,9 @@ from extract_turn_local_features import (
     main as extract_main,
 )
 from local_turn_features import (
-    LOCAL_FORMAT, causal_frame_ticks, crop_box, easycom_boxes, load_local_features,
-    local_head_dimensions, pooled_recent_visual, reference_alignment,
+    LOCAL_FORMAT, FramePersonFeatures, causal_frame_ticks, crop_box, easycom_boxes,
+    load_frame_set_features, load_local_features, local_head_dimensions,
+    pooled_recent_visual, reference_alignment,
 )
 from train_turn_head import _load_examples
 
@@ -47,6 +48,17 @@ def fixture(n=20, name="Session_4", dataset="easycom"):
                        "causal_policy": "half-open-trailing-window", "uses_speech_labels": False}}
     side["visual_valid"][0] = False
     side["visual"][0] = 0
+    frame_times = torch.arange(max(1, (n + 2) // 3), dtype=torch.float64) * .5
+    counts = [2 if index % 3 == 0 else (0 if index % 3 == 1 else 1)
+              for index in range(len(frame_times))]
+    offsets = torch.tensor([0, *np.cumsum(counts).tolist()], dtype=torch.long)
+    person_features = torch.randn(int(offsets[-1]), 6)
+    side["persons"] = {
+        "features": person_features,
+        "frame_times": frame_times,
+        "frame_offsets": offsets,
+        "records": [{"id": str(index)} for index in range(len(person_features))],
+    }
     labels = np.tile(np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 1]], dtype=np.float32), (n // 4 + 1, 1))[:n]
     return reference, side, labels
 
@@ -280,6 +292,148 @@ class LocalTests(unittest.TestCase):
             with self.assertRaisesRegex(ValueError, "times"):
                 reference_alignment({**reference, "times": reference["times"] + .2})
 
+    def test_frame_set_loader_is_causal_ragged_and_fail_closed(self):
+        reference, side, _ = fixture()
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / "Session_4.pt"
+            torch.save(side, path)
+            audio, people, contract = load_frame_set_features(
+                path,
+                reference,
+                sample_id="Session_4",
+                dataset="easycom",
+            )
+            self.assertEqual(audio.shape, (20, 6))
+            self.assertEqual(contract["modalities"], "av")
+            self.assertEqual(people.dimension, 6)
+            self.assertTrue(
+                torch.all(
+                    people.chosen_frame_times[
+                        torch.isfinite(people.chosen_frame_times)
+                    ]
+                    < reference["times"][
+                        torch.isfinite(people.chosen_frame_times)
+                    ]
+                )
+            )
+            # At 0.2/0.4s, both rows reuse the 0.0s frame with two people.
+            values, mask = people.padded(torch.tensor([0, 1, 2]))
+            self.assertEqual(values.shape, (3, 2, 6))
+            self.assertEqual(mask.tolist(), [[True, True], [True, True], [False, False]])
+            self.assertTrue(torch.equal(values[0], values[1]))
+            self.assertEqual(torch.count_nonzero(values[2]), 0)
+            # The frame stamped exactly 1.0s is excluded at the 1.0s cutoff.
+            self.assertEqual(people.chosen_frame_times[4].item(), .5)
+
+            broken_cases = []
+            broken = copy.deepcopy(side)
+            broken["persons"]["frame_offsets"][-1] -= 1
+            broken_cases.append(broken)
+            broken = copy.deepcopy(side)
+            broken["persons"]["frame_times"][1] = broken["persons"]["frame_times"][0]
+            broken_cases.append(broken)
+            broken = copy.deepcopy(side)
+            broken["persons"]["records"] = []
+            broken_cases.append(broken)
+            broken = copy.deepcopy(side)
+            broken["persons"]["features"][0, 0] = float("nan")
+            broken_cases.append(broken)
+            for index, broken in enumerate(broken_cases):
+                torch.save(broken, path)
+                with self.subTest(index=index), self.assertRaises(ValueError):
+                    load_frame_set_features(
+                        path,
+                        reference,
+                        sample_id="Session_4",
+                        dataset="easycom",
+                    )
+
+    def test_frame_person_concatenation_selection_and_empty_padding(self):
+        features = torch.tensor([[1., 2.], [3., 4.]])
+        a = FramePersonFeatures(
+            features,
+            torch.tensor([0, 2], dtype=torch.long),
+            torch.tensor([0, -1], dtype=torch.long),
+            torch.tensor([0., float("nan")], dtype=torch.float64),
+        )
+        b = FramePersonFeatures(
+            torch.empty(0, 2),
+            torch.tensor([0, 0], dtype=torch.long),
+            torch.tensor([0], dtype=torch.long),
+            torch.tensor([.5], dtype=torch.float64),
+        )
+        combined = FramePersonFeatures.concatenate([a, b])
+        self.assertEqual(len(combined), 3)
+        self.assertEqual(combined.row_to_frame.tolist(), [0, -1, 1])
+        selected = combined.select_rows(torch.tensor([2, 0]))
+        values, mask = selected.padded()
+        self.assertEqual(mask.tolist(), [[False, False], [True, True]])
+        self.assertEqual(torch.count_nonzero(values[0]), 0)
+        self.assertTrue(torch.equal(values[1], features))
+
+    def test_frame_set_head_is_permutation_invariant_and_masks_padding(self):
+        torch.manual_seed(7)
+        head = TurnPredictionHead(
+            hidden_size=8,
+            num_input_layers=4,
+            layer_fusion="concat-mlp",
+            local_visual_dim=6,
+            local_audio_dim=5,
+            local_projection_dim=3,
+            local_fusion="frame-set-attention",
+            person_attention_dim=4,
+            person_attention_heads=2,
+            transition_auxiliary=True,
+            dropout=0,
+        ).eval()
+        states = torch.randn(3, 4, 8)
+        local = torch.randn(3, 6)
+        local[:, -1] = 1
+        people = torch.randn(3, 3, 6)
+        mask = torch.tensor(
+            [[True, True, False], [True, False, False], [False, False, False]]
+        )
+        people[~mask] = 0
+        logits, transitions = head.forward_with_transition(
+            states,
+            local,
+            people,
+            mask,
+        )
+        self.assertEqual(logits.shape, (3, 10, 3))
+        self.assertEqual(transitions.shape, (3, 10))
+        (logits.square().mean() + transitions.square().mean()).backward()
+        self.assertIsNotNone(head.person_attention.in_proj_weight.grad)
+
+        permutation = torch.tensor([1, 0, 2])
+        permuted = head(states, local, people[:, permutation], mask[:, permutation])
+        self.assertTrue(torch.allclose(logits, permuted, atol=1e-6, rtol=1e-6))
+        changed = people.clone()
+        changed[~mask] = 1000
+        self.assertTrue(torch.allclose(logits, head(states, local, changed, mask)))
+        with self.assertRaisesRegex(ValueError, "person features"):
+            head(states, local)
+
+        config = {
+            "hidden_size": 8,
+            "num_input_layers": 4,
+            "layer_fusion": "concat-mlp",
+            "local_visual_dim": 6,
+            "local_audio_dim": 5,
+            "local_projection_dim": 3,
+            "local_fusion": "frame-set-attention",
+            "person_attention_dim": 4,
+            "person_attention_heads": 2,
+            "transition_auxiliary": True,
+            "dropout": 0,
+        }
+        restored = TurnPredictionHead(
+            **checkpoint_head_config({"head_config": config})
+        )
+        restored.load_state_dict(head.state_dict())
+        restored.eval()
+        self.assertTrue(torch.equal(restored(states, local, people, mask), logits))
+
     def test_head_shapes_gradients_missing_and_roundtrip(self):
         states = torch.randn(3, 4, 8)
         for v, a in ((0, 0), (6, 0), (0, 5), (6, 5)):
@@ -338,7 +492,8 @@ class LocalTests(unittest.TestCase):
             np.save(root / "targets/Session_4.npy", targets)
             data = _load_examples(root / "features", root / "targets", ["Session_4"], "open-stream-replay",
                                   "easycom", {"Session_4": {1, 4}}, root / "local", "audio")
-            self.assertTrue(torch.equal(data[-1][:, :5], side["audio"][[1, 4]]))
+            self.assertTrue(torch.equal(data[-2][:, :5], side["audio"][[1, 4]]))
+            self.assertIsNone(data[-1])
             self.assertTrue(torch.equal(data[0], reference["states"][[1, 4]]))
             self.assertTrue(torch.equal(data[1][:, 0], torch.from_numpy(targets[[2, 5]])))
 
@@ -562,7 +717,7 @@ class RoundTripTests(unittest.TestCase):
                 np.save(root / "targets" / f"{name}.npy", targets)
             common = ["--feature-root", str(root / "features"), "--target-root", str(root / "targets"),
                       "--device", "cpu", "--no-progress", "--expected-mode", "open-stream-replay"]
-            for mode in ("baseline", "audio", "visual", "av", "ego4d"):
+            for mode in ("baseline", "audio", "visual", "av", "frame-set", "ego4d"):
                 checkpoint = root / f"{mode}.pth"
                 train = [sys.executable, str(repo / "train_turn_head.py"), *common,
                          "--epochs", "1", "--warmup-epochs", "0", "--batch-size", "4",
@@ -579,8 +734,15 @@ class RoundTripTests(unittest.TestCase):
                     train += ["--dev-sessions", "5", "--local-projection-dim", "0"]
                 if mode == "av":
                     train += ["--transition-objective", "r1-supervised"]
+                if mode == "frame-set":
+                    train += [
+                        "--local-fusion", "frame-set-attention",
+                        "--person-attention-dim", "4",
+                        "--person-attention-heads", "2",
+                    ]
                 if mode != "baseline":
-                    train += ["--local-feature-root", str(root / "local"), "--local-modalities", "av" if mode == "ego4d" else mode]
+                    modality = "av" if mode in ("ego4d", "frame-set") else mode
+                    train += ["--local-feature-root", str(root / "local"), "--local-modalities", modality]
                     infer += ["--local-feature-root", str(root / "local")]
                 for command in (train, infer):
                     completed = subprocess.run(command, cwd=repo, env=env, capture_output=True, text=True, timeout=60)
@@ -590,6 +752,8 @@ class RoundTripTests(unittest.TestCase):
                 self.assertEqual(bool(metrics["local_contract"]), mode != "baseline")
                 if mode == "av":
                     self.assertTrue(metrics["transition_auxiliary"])
+                if mode == "frame-set":
+                    self.assertEqual(metrics["local_fusion"], "frame-set-attention")
             # The reporting command uses the same metric keys as inference.
             completed = subprocess.run(
                 [sys.executable, str(repo / "summarize_local_baselines.py"), "--root", str(root)],

train_turn_head.py

@@ -24,6 +24,7 @@ from easycom_head import (
     CLASS_NAMES,
     DATASET_NAMES,
     LAYER_FUSION_MODES,
+    LOCAL_FUSION_MODES,
     LOSS_NAMES,
     NUM_HORIZONS,
     OFFICIAL_TRAIN_SESSIONS,
@@ -46,6 +47,7 @@ from easycom_head import (
     transition_targets_from_activity,
     validate_disjoint_sessions,
 )
+from local_turn_features import FramePersonFeatures
 
 
 def parse_args() -> argparse.Namespace:
@@ -58,10 +60,18 @@ def parse_args() -> argparse.Namespace:
         help="Optional frozen local-feature sidecars; omit for the original baseline.",
     )
     parser.add_argument("--local-modalities", choices=("visual", "audio", "av"), default="av")
+    parser.add_argument(
+        "--local-fusion",
+        choices=LOCAL_FUSION_MODES,
+        default="pooled",
+        help="pooled keeps the existing baseline; frame-set-attention queries the latest causal person set.",
+    )
     parser.add_argument(
         "--local-projection-dim", type=int, default=256,
         help="Trainable projection per local modality; 0 retains encoder dimensions.",
     )
+    parser.add_argument("--person-attention-dim", type=int, default=256)
+    parser.add_argument("--person-attention-heads", type=int, default=4)
     parser.add_argument(
         "--dataset",
         choices=DATASET_NAMES,
@@ -165,6 +175,17 @@ def parse_args() -> argparse.Namespace:
 def _validate_args(args: argparse.Namespace) -> None:
     if args.local_projection_dim < 0:
         raise ValueError("--local-projection-dim must be non-negative")
+    if args.person_attention_dim <= 0 or args.person_attention_heads <= 0:
+        raise ValueError("Person attention dimension/heads must be positive")
+    if args.person_attention_dim % args.person_attention_heads:
+        raise ValueError("--person-attention-dim must be divisible by --person-attention-heads")
+    if args.local_fusion == "frame-set-attention" and (
+        args.local_feature_root is None or args.local_modalities != "av"
+    ):
+        raise ValueError(
+            "--local-fusion frame-set-attention requires --local-feature-root "
+            "and --local-modalities av"
+        )
     if args.epochs <= 0:
         raise ValueError("--epochs must be positive")
     if args.batch_size <= 0:
@@ -225,6 +246,7 @@ def _load_examples(
     local_modalities: str = "av",
     local_contract: dict[str, Any] | None = None,
     compute_transition_targets: bool = True,
+    local_fusion: str = "pooled",
 ) -> tuple[
     torch.Tensor,
     torch.Tensor,
@@ -233,11 +255,13 @@ def _load_examples(
     int,
     list[dict[str, Any]],
     torch.Tensor,
+    FramePersonFeatures | None,
 ]:
     all_states: list[torch.Tensor] = []
     all_targets: list[torch.Tensor] = []
     all_transition_targets: list[torch.Tensor] = []
     all_local: list[torch.Tensor] = []
+    all_people: list[FramePersonFeatures] = []
     metadata: list[dict[str, Any]] = []
     hidden_size: int | None = None
     num_input_layers: int | None = None
@@ -251,6 +275,7 @@ def _load_examples(
             local_feature_root=local_feature_root,
             local_modalities=local_modalities,
             local_contract=local_contract,
+            local_fusion=local_fusion,
         )
         local_contract = data.feature_metadata.get("local_contract")
         states, targets, future_indices = data.future_examples(NUM_HORIZONS)
@@ -260,6 +285,11 @@ def _load_examples(
             if data.local_features is not None
             else torch.empty(len(states), 0)
         )
+        people = (
+            data.local_person_features.select_rows(current_indices)
+            if data.local_person_features is not None
+            else None
+        )
         if compute_transition_targets:
             activity_sequence = torch.cat(
                 (data.targets[current_indices, None, :], targets),
@@ -286,6 +316,8 @@ def _load_examples(
             targets = targets[keep]
             transition_targets = transition_targets[keep]
             local = local[keep]
+            if people is not None:
+                people = people.select_rows(torch.nonzero(keep).flatten())
             if not len(states):
                 raise ValueError(f"Sparse anchor selection kept no examples for {name}")
         if hidden_size is None:
@@ -306,6 +338,8 @@ def _load_examples(
         all_targets.append(targets)
         all_transition_targets.append(transition_targets)
         all_local.append(local)
+        if people is not None:
+            all_people.append(people)
         metadata.append(
             {
                 "session": name,
@@ -332,6 +366,8 @@ def _load_examples(
         )
     if hidden_size is None or num_input_layers is None:
         raise ValueError("No sessions selected")
+    if all_people and len(all_people) != len(all_states):
+        raise ValueError("Person sets are missing for some sessions")
     return (
         torch.cat(all_states, dim=0),
         torch.cat(all_targets, dim=0),
@@ -340,6 +376,7 @@ def _load_examples(
         num_input_layers,
         metadata,
         torch.cat(all_local, dim=0),
+        FramePersonFeatures.concatenate(all_people) if all_people else None,
     )
 
 
@@ -407,6 +444,7 @@ def _evaluate_valid_horizons(
     batch_size: int,
     loss_name: str,
     local: torch.Tensor | None = None,
+    people: FramePersonFeatures | None = None,
 ) -> dict[str, Any]:
     head.eval()
     chunks: list[torch.Tensor] = []
@@ -420,7 +458,19 @@ def _evaluate_valid_horizons(
             local[start : start + batch_size].to(device=device, dtype=torch.float32)
             if local is not None else None
         )
-        chunks.append(marginal_probabilities_from_logits(head(x, side), loss_name).cpu())
+        person_batch = None
+        person_mask = None
+        if people is not None:
+            rows = torch.arange(start, min(start + batch_size, len(states)))
+            person_batch, person_mask = people.padded(rows)
+            person_batch = person_batch.to(device=device, dtype=torch.float32)
+            person_mask = person_mask.to(device=device)
+        chunks.append(
+            marginal_probabilities_from_logits(
+                head(x, side, person_batch, person_mask),
+                loss_name,
+            ).cpu()
+        )
     probabilities = torch.cat(chunks, dim=0).numpy()
     target_array = targets.numpy()
 
@@ -455,7 +505,7 @@ def _checkpoint(
     history: list[dict[str, Any]],
 ) -> dict[str, Any]:
     return {
-        "format_version": 4,
+        "format_version": 5,
         "head": head.state_dict(),
         "head_config": {
             "hidden_size": hidden_size,
@@ -469,6 +519,9 @@ def _checkpoint(
             "local_visual_dim": head.local_visual_dim,
             "local_audio_dim": head.local_audio_dim,
             "local_projection_dim": head.local_projection_dim,
+            "local_fusion": head.local_fusion,
+            "person_attention_dim": head.person_attention_dim,
+            "person_attention_heads": head.person_attention_heads,
         },
         "local_contract": feature_metadata[0].get("local_contract"),
         "alignment": {
@@ -513,7 +566,10 @@ def _checkpoint(
             "layer_fusion": args.layer_fusion,
             "local_feature_root": str(args.local_feature_root) if args.local_feature_root else None,
             "local_modalities": args.local_modalities if args.local_feature_root else None,
+            "local_fusion": args.local_fusion,
             "local_projection_dim": args.local_projection_dim,
+            "person_attention_dim": args.person_attention_dim,
+            "person_attention_heads": args.person_attention_heads,
             "learnable_layer_mixture": args.layer_fusion == "scalar-mixture",
             "supervision_policy": (
                 "transition-sparse-manifest"
@@ -605,6 +661,7 @@ def main() -> None:
         num_input_layers,
         feature_metadata,
         local_features,
+        person_features,
     ) = _load_examples(
         args.feature_root,
         args.target_root,
@@ -615,6 +672,7 @@ def main() -> None:
         args.local_feature_root,
         args.local_modalities,
         compute_transition_targets=args.transition_objective != "none",
+        local_fusion=args.local_fusion,
     )
     local_contract = feature_metadata[0].get("local_contract")
     if not len(states):
@@ -645,6 +703,7 @@ def main() -> None:
     dev_states: torch.Tensor | None = None
     dev_targets: torch.Tensor | None = None
     dev_local: torch.Tensor | None = None
+    dev_people: FramePersonFeatures | None = None
     if dev_sessions:
         (
             dev_states,
@@ -654,6 +713,7 @@ def main() -> None:
             dev_num_input_layers,
             _,
             dev_local,
+            dev_people,
         ) = _load_examples(
             args.feature_root,
             args.target_root,
@@ -665,6 +725,7 @@ def main() -> None:
             args.local_modalities,
             local_contract,
             compute_transition_targets=args.transition_objective != "none",
+            local_fusion=args.local_fusion,
         )
         if dev_hidden_size != hidden_size:
             raise ValueError(
@@ -678,7 +739,13 @@ def main() -> None:
             )
         print(f"development examples: {len(dev_states)}")
 
-    dataset = TensorDataset(states, targets, transition_targets, local_features)
+    dataset = TensorDataset(
+        states,
+        targets,
+        transition_targets,
+        local_features,
+        torch.arange(len(states), dtype=torch.long),
+    )
     generator = torch.Generator().manual_seed(args.seed)
     loader = DataLoader(
         dataset,
@@ -702,13 +769,26 @@ def main() -> None:
         layer_fusion=args.layer_fusion,
         transition_auxiliary=args.transition_objective != "none",
         local_projection_dim=args.local_projection_dim,
+        local_fusion=args.local_fusion,
+        person_attention_dim=args.person_attention_dim,
+        person_attention_heads=args.person_attention_heads,
         **local_head_dimensions(local_contract),
     ).to(device)
     if local_contract is not None:
         print(
-            f"local fusion: {args.local_modalities}; raw V={head.local_visual_dim}, "
-            f"A={head.local_audio_dim}; projection={args.local_projection_dim or 'identity'}"
+            f"local fusion: {args.local_fusion}/{args.local_modalities}; "
+            f"raw V={head.local_visual_dim}, A={head.local_audio_dim}; "
+            f"projection={args.local_projection_dim or 'identity'}"
         )
+        if person_features is not None:
+            print(
+                "frame sets: "
+                f"{len(person_features.frame_offsets) - 1} source frames, "
+                f"{len(person_features.features)} person observations, "
+                f"max_people={person_features.max_people}, "
+                f"attention={args.person_attention_dim}D/"
+                f"{args.person_attention_heads} heads"
+            )
     trainable_parameters = sum(
         parameter.numel() for parameter in head.parameters() if parameter.requires_grad
     )
@@ -782,7 +862,13 @@ def main() -> None:
             unit="batch",
             disable=args.no_progress,
         )
-        for state_batch, target_batch, transition_target_batch, local_batch in progress:
+        for (
+            state_batch,
+            target_batch,
+            transition_target_batch,
+            local_batch,
+            example_index_batch,
+        ) in progress:
             local_batch = local_batch.to(
                 device=device, dtype=torch.float32, non_blocking=True,
             )
@@ -801,15 +887,37 @@ def main() -> None:
                 dtype=torch.float32,
                 non_blocking=True,
             )
+            person_batch = None
+            person_mask = None
+            if person_features is not None:
+                person_batch, person_mask = person_features.padded(
+                    example_index_batch
+                )
+                person_batch = person_batch.to(
+                    device=device,
+                    dtype=torch.float32,
+                    non_blocking=True,
+                )
+                person_mask = person_mask.to(device=device, non_blocking=True)
             optimizer.zero_grad(set_to_none=True)
             if args.transition_objective == "none":
-                logits = head(state_batch, local_batch)
+                logits = head(
+                    state_batch,
+                    local_batch,
+                    person_batch,
+                    person_mask,
+                )
                 activity_loss = loss_function(logits, target_batch)
                 loss = activity_loss
                 transition_loss = activity_loss.detach().new_zeros(())
                 expected_reward = activity_loss.detach().new_zeros(())
             else:
-                logits, transition_logits = head.forward_with_transition(state_batch, local_batch)
+                logits, transition_logits = head.forward_with_transition(
+                    state_batch,
+                    local_batch,
+                    person_batch,
+                    person_mask,
+                )
                 activity_loss = loss_function(logits, target_batch)
                 components = transition_aware_objective(
                     activity_loss,
@@ -890,6 +998,7 @@ def main() -> None:
                 args.batch_size,
                 args.loss,
                 dev_local,
+                dev_people,
             )
             dev_map = float(dev_metrics["mean_map"])
             record["dev_valid_mean_map"] = dev_map