Add pooled local AV baselines and research notes
This commit is contained in:
Xin Wang
@@ -7,6 +7,10 @@ EgoSpeak** for egocentric turn-taking anticipation.
|
||||
The persistent paper index and Chinese summaries are maintained in
|
||||
[literature_notes](literature_notes/README.md).
|
||||
|
||||
The pooled local-feature B1 experiment (frozen Qwen face crops + recent audio,
|
||||
native 2048D sidecars, optional trainable projection, four-epoch ablations) is
|
||||
documented in [docs/local_feature_baseline.md](docs/local_feature_baseline.md).
|
||||
|
||||
The main experiment is deliberately simple:
|
||||
|
||||
```text
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
# B1：局部视觉＋近期混合音频，普通池化融合
|
||||
|
||||
这是用于判断“原 Omni readout 是否丢掉有用的局部 AV 线索”的基线，
|
||||
不是已证明的改进，也不是 multi-speaker attention 模型。
|
||||
保留原 `[T,4,2048]` 特征；不重新提取全局 Thinker，不改标签和 AP 协议。
|
||||
|
||||
## 固定实验定义
|
||||
|
||||
| 分支 | 冻结特征 | 更新／池化 | 训练时处理 |
|
||||
|---|---|---|---|
|
||||
| 原全局特征 | Thinker last4，`[T,4,2048]` | 原来的 8s memory、5Hz | 原 concat-MLP，8192→2048 |
|
||||
| 局部视觉 V | 每个人脸裁剪，Qwen vision encoder + merger，2048D | 原视频固定 2FPS；先空间 mean，再逐人等权 mean，再最近 1s 有效帧等权 mean | 可训练 LN→Linear(2048,256)→GELU |
|
||||
| 近期音频 A | Qwen audio encoder + 原生 output projection，2048D | 每 200ms 重编码 `[max(0,t-1),t)` 混合音频；仅池化有效输出 tokens | 同上 |
|
||||
|
||||
Qwen 3B 的 **2048 是编码器原生输出维度，256 只是新增 head 的可选投影宽度**。
|
||||
特征文件永远保存原始维度，`--local-projection-dim 0` 可不降维。
|
||||
正常 AV 组合为 `[2048 global,256 V,256 A,2 validity]`，共 2562D，
|
||||
接 LN→Linear(2562,1024)→GELU→Dropout→Linear(1024,30)，输出仍为 `[10,3]`。
|
||||
单模态只附加对应一路与一个有效标志；不传 `--local-feature-root` 时保留原 head 参数结构。
|
||||
|
||||
局部视觉没有运动编码、ASD 分数、speaker-aware attention、身份 embedding 或图网络。
|
||||
参与者 ID 仅保存供审计，不输入 head。人数多的帧不会因 token 多而得到更大权重。
|
||||
没有有效框时视觉为零、valid=False；音频静音仍 valid=True。
|
||||
缺少音轨、损坏的媒体或缺少应有的标注文件会报错，不会悄悄冒充“空输入”。
|
||||
|
||||
时间约定是 `[t-1,t)`：例如 t=1.0s，视觉取 0.0、0.5s，排除 1.0s 的帧。
|
||||
音频先截断再做 mel 和编码，不对完整视频的音频编码后切片。
|
||||
这保证相对于 readout 时刻的输入因果性，但不保证编码器在窗口内部逐 token 因果。
|
||||
|
||||
实现依据：[Qwen 3B 配置](https://huggingface.co/Qwen/Qwen2.5-Omni-3B/blob/main/config.json)、
|
||||
[Transformers 4.52.3 编码器 API](https://github.com/huggingface/transformers/blob/v4.52.3/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py)。
|
||||
脚本加载 Thinker checkpoint 后释放语言解码器，仅将冻结的视觉／音频塔移到 GPU；没有 Thinker forward。
|
||||
|
||||
## 1. EasyCom：先检查框，再跑一个 4s smoke test
|
||||
|
||||
在已有项目依赖的 GPU 机器运行，以下路径沿用之前实验；raw 数据根按实际挂载修改。
|
||||
不需要先重新编码 2FPS 视频，直接读取原 `Main/Video_Compressed`。
|
||||
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B
|
||||
DATA_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
|
||||
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
|
||||
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
||||
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s
|
||||
|
||||
LOCAL_ARGS=(
|
||||
--dataset easycom --data-root "$DATA_ROOT"
|
||||
--feature-root "$FEATURE_ROOT" --target-root "$TARGET_ROOT"
|
||||
--model-id "$MODEL" --device cuda:0 --dtype bfloat16 --flash-attention-2
|
||||
--video-fps 2 --visual-window-seconds 1 --audio-window-seconds 1
|
||||
--box-kind face --box-coordinate-space camera
|
||||
)
|
||||
|
||||
# 不加载模型；输出四张带框图片，请先检查框是否真在人脸上。
|
||||
python extract_turn_local_features.py "${LOCAL_ARGS[@]}" \
|
||||
--samples 1 --max-steps 20 --preview-only --output-root features/local_preview
|
||||
|
||||
# 4s / 20 states；此文件明确标记 partial，不能误拿去训练整段。
|
||||
python extract_turn_local_features.py "${LOCAL_ARGS[@]}" \
|
||||
--samples 1 --max-steps 20 --output-root features/local_smoke
|
||||
```
|
||||
|
||||
EasyCom face 标注首帧是 1，head 首帧是 0；按**原始解码帧号**取框。
|
||||
视频裁去左侧身份 banner，保留右侧 1920 像素；默认框坐标属于裁后的相机图像。
|
||||
如果你的导出文件的框坐标含 banner，需要改成 `--box-coordinate-space encoded`。
|
||||
必须通过 preview 检查这个约定，不能只看最终特征维度。
|
||||
官方帧号与 banner 说明见 [EasyCom 数据说明](https://github.com/facebookresearch/EasyComDataset#notes-and-known-issues)。
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import torch
|
||||
p = torch.load('features/local_smoke/Session_1.pt', map_location='cpu', weights_only=False)
|
||||
print('partial:', p['partial'])
|
||||
for key in ('visual', 'audio'):
|
||||
x = p[key]
|
||||
print(key, tuple(x.shape), x.dtype, 'finite:', bool(torch.isfinite(x).all()),
|
||||
'valid:', p[key + '_valid'].float().mean().item())
|
||||
print('person features:', tuple(p['persons']['features'].shape))
|
||||
print('times:', p['times'][0].item(), p['times'][-1].item())
|
||||
PY
|
||||
```
|
||||
|
||||
3B 预期 V/A 各 `[20,2048]`，时间 0.2–4.0s；没有人脸的时刻允许 V 无效。
|
||||
本机已验证 CPU 对齐／融合／合成数据训练推理，真实 checkpoint + GPU 提取仍需此 smoke test。
|
||||
另有不下载权重的小型随机 Qwen 编码器 API 测试。可在已有依赖的环境运行：
|
||||
|
||||
```bash
|
||||
python -m unittest -v test_easycom_head_objectives test_local_turn_features
|
||||
```
|
||||
|
||||
## 2. 提取全部 12 sessions（只做一次）
|
||||
|
||||
```bash
|
||||
python extract_turn_local_features.py "${LOCAL_ARGS[@]}" \
|
||||
--samples 1 2 3 4 5 6 7 8 9 10 11 12 --output-root "$LOCAL_ROOT"
|
||||
```
|
||||
|
||||
默认前台顺序执行，显示视觉帧进度和音频 state 进度；模型只加载一次。
|
||||
已有文件会报错，恢复时在 `--samples` 中只列尚未完成的 sessions。
|
||||
`--overwrite` 是明确重算，不会重用不兼容文件。不要覆盖原始 feature root。
|
||||
`persons` 还保留每帧变长的逐人特征、frame_offsets、框、帧号和时间，供后续 B2 复用。
|
||||
若在不同挂载路径提取同一 checkpoint，使用相同 `--encoder-id`；不同权重不可共用这个 ID。
|
||||
|
||||
## 3. 四组对照：4 epochs、无 dev、相同种子和 loss
|
||||
|
||||
| 名称 | 输入 |
|
||||
|---|---|
|
||||
| baseline | 原 Omni last4 |
|
||||
| audio | 原 Omni + 近期音频 |
|
||||
| visual | 原 Omni + 局部视觉 |
|
||||
| av | 原 Omni + 两路局部特征（B1） |
|
||||
|
||||
```bash
|
||||
# 先做 seed0 四组；默认前台跑完训练后立即评测 Session 3/2/1。
|
||||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_local_baselines.sh all 0
|
||||
|
||||
# 再补 seed1/2；不会覆盖 seed0。
|
||||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_local_baselines.sh all 1 2
|
||||
|
||||
# 也可把互不重复的组合分配到不同卡，不要复用相同 variant/seed 输出路径。
|
||||
# CUDA_VISIBLE_DEVICES=1 bash run_easycom_local_baselines.sh audio 0
|
||||
|
||||
python summarize_local_baselines.py --protocol causal_valid_only
|
||||
python summarize_local_baselines.py --protocol egospeak_full_timeline
|
||||
```
|
||||
|
||||
脚本使用训练 Session 4–12、epoch=4、batch=512、lr=7e-5、wd=5e-5、warmup=1.6 epochs、
|
||||
warmup_factor=0.3、dropout=0.2、LSTR-MCE、不加 transition objective。
|
||||
`FEATURE_ROOT/TARGET_ROOT/LOCAL_ROOT` 用上面 export 的值，也可自行覆盖。
|
||||
结果分别在 `checkpoints/easycom_local_b1` 和 `results/easycom_local_b1`。
|
||||
如要不降维，设置 `LOCAL_PROJECTION_DIM=0`，并换 `CHECKPOINT_ROOT/RESULT_ROOT`。
|
||||
|
||||
手动训练只需在原命令增加：
|
||||
|
||||
```bash
|
||||
--local-feature-root "$LOCAL_ROOT" --local-modalities av --local-projection-dim 256
|
||||
```
|
||||
|
||||
手动推理增加 `--local-feature-root "$LOCAL_ROOT"`；模态组合从 checkpoint 读取。
|
||||
Loader 会检查 sample、时间、label_indices、维度、valid mask 和 encoder/pooling 配置。
|
||||
训练与评测必须用同一种 AP 协议作比较；不要混用 full-timeline 与 causal-valid 的表格。
|
||||
|
||||
## Ego4D 接口
|
||||
|
||||
提取脚本同样支持 `--dataset ego4d`：输入根为 `/data/wangx/Datasets/ego4d`，
|
||||
读取 `v1/clips/<uid>.mp4`、`v1/annotations/av_train.json` 与 `av_val.json`，
|
||||
`--samples` 传已有 Omni `.pt` 的 clip UID。`--target-root` 指向 `ego4d_target_frame`。
|
||||
可用 `--ego4d-annotations` 指定其他导出路径。先用 `--preview-only --max-steps 20` 检查框。
|
||||
读取的是与 clip_uid 对应的可见人物 tracks；不使用 wearer 框、voice_segments、讲话标签，
|
||||
不对缺失帧用未来框插值。载入大 JSON 需要相应主机内存。
|
||||
train/infer 使用原 `--dataset ego4d --manifest ...` 加上述 local 参数；
|
||||
EasyCom 的运行 shell 只管理 EasyCom，不能用于 Ego4D 的 split。
|
||||
|
||||
## 如何解读实验
|
||||
|
||||
先看每种子 paired 的 0.2s wearer AP 与 10-horizon mean mAP，不能提前承诺提升。
|
||||
若只有 A 有效，先支持“近期声音补充”的解释；V/AV 的额外收益才支持局部视觉线索有价值。
|
||||
这几组 head 参数量不同，结果不是纯架构公平性证明；下一步应做同参数量全局头／打乱局部输入控制。
|
||||
|
||||
这一版使用提供的框（部分身份标注经人工修正），不是在线 detector 的端到端测试。
|
||||
论文必须披露这个协议，后续补在线检测框实验；不得把 close-mic、讲话标签或测试集调优引入输入。
|
||||
提供框读取即使只取当前帧，也不代表框生成器本身是在线因果的。
|
||||
2FPS 静态脸特征不能代表完整唇动建模；若 V 无提升，不宜直接断言多人视觉无用。
|
||||
+84
-7
@@ -74,6 +74,9 @@ class TurnPredictionHead(nn.Module):
|
||||
learnable_layer_mixture: bool = False,
|
||||
layer_fusion: str | None = None,
|
||||
transition_auxiliary: bool = False,
|
||||
local_visual_dim: int = 0,
|
||||
local_audio_dim: int = 0,
|
||||
local_projection_dim: int = 256,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
if num_input_layers <= 0:
|
||||
@@ -108,6 +111,11 @@ class TurnPredictionHead(nn.Module):
|
||||
self.layer_fusion = layer_fusion
|
||||
self.learnable_layer_mixture = layer_fusion == "scalar-mixture"
|
||||
self.transition_auxiliary = transition_auxiliary
|
||||
if min(local_visual_dim, local_audio_dim, local_projection_dim) < 0:
|
||||
raise ValueError("Local dimensions must be non-negative")
|
||||
self.local_visual_dim = local_visual_dim
|
||||
self.local_audio_dim = local_audio_dim
|
||||
self.local_projection_dim = local_projection_dim
|
||||
if self.learnable_layer_mixture:
|
||||
# Zero initialization exactly reproduces a uniform layer mean at
|
||||
# step zero while allowing training to select useful query layers.
|
||||
@@ -123,9 +131,23 @@ class TurnPredictionHead(nn.Module):
|
||||
)
|
||||
else:
|
||||
self.layer_fusion_net = None
|
||||
self.local_projections = nn.ModuleDict()
|
||||
fused_size = hidden_size
|
||||
for name, dimension in (("visual", local_visual_dim), ("audio", local_audio_dim)):
|
||||
if dimension:
|
||||
self.local_projections[name] = (
|
||||
nn.Sequential(
|
||||
nn.LayerNorm(dimension),
|
||||
nn.Linear(dimension, local_projection_dim),
|
||||
nn.GELU(),
|
||||
)
|
||||
if local_projection_dim
|
||||
else nn.Identity()
|
||||
)
|
||||
fused_size += (local_projection_dim or dimension) + 1
|
||||
self.net = nn.Sequential(
|
||||
nn.LayerNorm(hidden_size),
|
||||
nn.Linear(hidden_size, hidden_size // 2),
|
||||
nn.LayerNorm(fused_size),
|
||||
nn.Linear(fused_size, hidden_size // 2),
|
||||
nn.GELU(),
|
||||
nn.Dropout(dropout),
|
||||
nn.Linear(hidden_size // 2, num_horizons * num_classes),
|
||||
@@ -166,14 +188,46 @@ class TurnPredictionHead(nn.Module):
|
||||
states = self.layer_fusion_net(states)
|
||||
return states
|
||||
|
||||
def _shared_features(self, states: torch.Tensor) -> torch.Tensor:
|
||||
@property
|
||||
def local_input_size(self) -> int:
|
||||
return sum(d + 1 for d in (self.local_visual_dim, self.local_audio_dim) if d)
|
||||
|
||||
def _shared_features(
|
||||
self,
|
||||
states: torch.Tensor,
|
||||
local: torch.Tensor | None = None,
|
||||
) -> torch.Tensor:
|
||||
"""Return the activity MLP representation shared by both readouts."""
|
||||
|
||||
states = self._fuse_layers(states)
|
||||
if self.local_input_size:
|
||||
if local is None or local.shape != (*states.shape[:-1], self.local_input_size):
|
||||
raise ValueError(
|
||||
f"Head requires aligned local features of width {self.local_input_size}"
|
||||
)
|
||||
pieces = [states]
|
||||
offset = 0
|
||||
for name, dimension in (
|
||||
("visual", self.local_visual_dim),
|
||||
("audio", self.local_audio_dim),
|
||||
):
|
||||
if dimension:
|
||||
values = local[..., offset : offset + dimension]
|
||||
valid = local[..., offset + dimension : offset + dimension + 1]
|
||||
# Mask AFTER projection too: a learned bias is not an observation.
|
||||
pieces.extend((self.local_projections[name](values) * valid, valid))
|
||||
offset += dimension + 1
|
||||
states = torch.cat(pieces, dim=-1)
|
||||
elif local is not None and local.shape[-1] != 0:
|
||||
raise ValueError("Local features supplied to a baseline head")
|
||||
return self.net[:-1](states)
|
||||
|
||||
def forward(self, states: torch.Tensor) -> torch.Tensor:
|
||||
features = self._shared_features(states)
|
||||
def forward(
|
||||
self,
|
||||
states: torch.Tensor,
|
||||
local: torch.Tensor | None = None,
|
||||
) -> torch.Tensor:
|
||||
features = self._shared_features(states, local)
|
||||
logits = self.net[-1](features)
|
||||
return logits.view(
|
||||
*features.shape[:-1], self.num_horizons, self.num_classes
|
||||
@@ -182,6 +236,7 @@ class TurnPredictionHead(nn.Module):
|
||||
def forward_with_transition(
|
||||
self,
|
||||
states: torch.Tensor,
|
||||
local: torch.Tensor | None = None,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Return activity logits and binary transition logits."""
|
||||
|
||||
@@ -189,7 +244,7 @@ class TurnPredictionHead(nn.Module):
|
||||
raise RuntimeError(
|
||||
"This head was created without transition_auxiliary=True"
|
||||
)
|
||||
features = self._shared_features(states)
|
||||
features = self._shared_features(states, local)
|
||||
activity_logits = self.net[-1](features).view(
|
||||
*features.shape[:-1],
|
||||
self.num_horizons,
|
||||
@@ -631,6 +686,7 @@ class SessionData:
|
||||
targets: torch.Tensor
|
||||
label_indices: torch.LongTensor
|
||||
feature_metadata: dict[str, Any]
|
||||
local_features: torch.Tensor | None = None
|
||||
|
||||
@property
|
||||
def hidden_size(self) -> int:
|
||||
@@ -662,6 +718,9 @@ def load_session_data(
|
||||
*,
|
||||
expected_mode: str | None = "causal-prefix",
|
||||
dataset: DatasetName = "easycom",
|
||||
local_feature_root: str | Path | None = None,
|
||||
local_modalities: str = "av",
|
||||
local_contract: dict[str, Any] | None = None,
|
||||
) -> SessionData:
|
||||
"""Load and validate one saved Omni feature payload and target array."""
|
||||
|
||||
@@ -738,7 +797,22 @@ def load_session_data(
|
||||
"video_sampling_grid",
|
||||
)
|
||||
}
|
||||
return SessionData(name, states, targets, label_indices, metadata)
|
||||
local = None
|
||||
if local_feature_root is not None:
|
||||
from local_turn_features import load_local_features
|
||||
|
||||
local, contract = load_local_features(
|
||||
Path(local_feature_root) / f"{name}.pt",
|
||||
payload,
|
||||
sample_id=name,
|
||||
dataset=dataset,
|
||||
modalities=local_modalities,
|
||||
expected_contract=local_contract,
|
||||
)
|
||||
metadata["local_contract"] = contract
|
||||
elif local_contract is not None:
|
||||
raise ValueError("--local-feature-root is required for this checkpoint")
|
||||
return SessionData(name, states, targets, label_indices, metadata, local)
|
||||
|
||||
|
||||
def average_precision_metrics(
|
||||
@@ -780,6 +854,9 @@ def checkpoint_head_config(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
config.setdefault("num_input_layers", 1)
|
||||
config.setdefault("learnable_layer_mixture", False)
|
||||
config.setdefault("transition_auxiliary", False)
|
||||
config.setdefault("local_visual_dim", 0)
|
||||
config.setdefault("local_audio_dim", 0)
|
||||
config.setdefault("local_projection_dim", 256)
|
||||
config.setdefault(
|
||||
"layer_fusion",
|
||||
(
|
||||
|
||||
@@ -0,0 +1,407 @@
|
||||
"""Extract B1 frozen Qwen local crops + recent mixed audio, aligned to Omni .pt.
|
||||
|
||||
Uses provided boxes (not speaking labels), raw source media and the existing
|
||||
dense-timeline reader. No Thinker forward, cross-person attention, or learned
|
||||
256D projection is performed here. See docs/local_feature_baseline.md.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from tqdm.auto import tqdm
|
||||
|
||||
from easycom_head import normalize_sample_name
|
||||
from local_turn_features import (
|
||||
LOCAL_FORMAT, causal_frame_ticks, crop_box, easycom_boxes,
|
||||
pooled_recent_visual, reference_alignment,
|
||||
)
|
||||
|
||||
|
||||
def model_config_digest(config: dict[str, Any]) -> str:
|
||||
"""Ignore location fields so an explicitly identified checkpoint can move."""
|
||||
def stable(value):
|
||||
if isinstance(value, dict):
|
||||
return {key: stable(item) for key, item in value.items() if key != "_name_or_path"}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [stable(item) for item in value]
|
||||
return value
|
||||
|
||||
return hashlib.sha256(json.dumps(stable(config), sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
class FrozenQwenLocalEncoder:
|
||||
"""Keep only frozen vision+merger and audio+projection on the GPU."""
|
||||
|
||||
def __init__(self, args: argparse.Namespace) -> None:
|
||||
import transformers
|
||||
from transformers import Qwen2_5OmniProcessor, Qwen2_5OmniThinkerForConditionalGeneration
|
||||
|
||||
if transformers.__version__ != "4.52.3":
|
||||
raise RuntimeError("Local encoder API is tested against transformers==4.52.3; use requirements.txt")
|
||||
self.device = torch.device(args.device)
|
||||
if self.device.type == "cuda" and not torch.cuda.is_available():
|
||||
raise RuntimeError(f"CUDA requested but unavailable: {args.device}")
|
||||
kwargs = {"torch_dtype": getattr(torch, args.dtype), "device_map": {"": "cpu"},
|
||||
"output_loading_info": True}
|
||||
if args.flash_attention_2:
|
||||
kwargs["attn_implementation"] = "flash_attention_2"
|
||||
model, info = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained(args.model_id, **kwargs)
|
||||
bad = [key for key in info.get("missing_keys", [])
|
||||
if key.startswith(("visual.", "audio_tower."))]
|
||||
if bad or info.get("mismatched_keys") or info.get("error_msgs"):
|
||||
raise RuntimeError(f"Cannot extract frozen features from incompletely loaded weights: {info}")
|
||||
model.requires_grad_(False).eval()
|
||||
self.visual_dim = int(model.config.vision_config.out_hidden_size)
|
||||
self.audio_dim = int(model.config.audio_config.output_dim)
|
||||
self.model_config_sha256 = model_config_digest(model.config.to_dict())
|
||||
# These modules are never called. Releasing them avoids keeping the
|
||||
# multi-billion-parameter language decoder on the extraction GPU.
|
||||
model.model = None
|
||||
model.lm_head = None
|
||||
model.visual.to(self.device)
|
||||
model.audio_tower.to(self.device)
|
||||
self.model = model
|
||||
self.processor = Qwen2_5OmniProcessor.from_pretrained(args.model_id)
|
||||
self.max_pixels = args.max_pixels
|
||||
self.image_batch_size = args.image_batch_size
|
||||
|
||||
@torch.inference_mode()
|
||||
def images(self, crops: list[Any]) -> torch.Tensor:
|
||||
pieces = []
|
||||
for start in range(0, len(crops), self.image_batch_size):
|
||||
batch = self.processor.image_processor(
|
||||
images=crops[start:start + self.image_batch_size],
|
||||
min_pixels=4 * 28 * 28, max_pixels=self.max_pixels, return_tensors="pt",
|
||||
)
|
||||
grid = batch["image_grid_thw"].to(self.device)
|
||||
pixels = batch["pixel_values"].to(self.device, dtype=self.model.visual.dtype)
|
||||
tokens = self.model.get_image_features(pixels, grid)
|
||||
merge = int(self.model.config.vision_config.spatial_merge_size)
|
||||
counts = (grid.prod(-1) // (merge * merge)).tolist()
|
||||
if tokens.ndim != 2 or tokens.shape != (sum(counts), self.visual_dim):
|
||||
raise RuntimeError(f"Unexpected image encoder shape: {tuple(tokens.shape)}")
|
||||
# Split BEFORE spatial mean: each person has one vote regardless of
|
||||
# box size / number of visual tokens.
|
||||
pieces.extend(part.float().mean(0).cpu() for part in tokens.split(counts))
|
||||
return torch.stack(pieces) if pieces else torch.empty(0, self.visual_dim)
|
||||
|
||||
@torch.inference_mode()
|
||||
def audio(self, waveform: np.ndarray) -> torch.Tensor:
|
||||
if waveform.ndim != 1 or not len(waveform) or not np.isfinite(waveform).all():
|
||||
raise ValueError("Audio must be finite nonempty mono samples")
|
||||
# The waveform has ALREADY been cut at the current time. Computing its
|
||||
# mel transform here cannot read future samples of the source file.
|
||||
batch = self.processor.feature_extractor(
|
||||
waveform, sampling_rate=16000, padding="longest",
|
||||
return_attention_mask=True, return_tensors="pt",
|
||||
)
|
||||
features = batch["input_features"].to(self.device, dtype=self.model.audio_tower.dtype)
|
||||
mask = batch["attention_mask"].to(self.device)
|
||||
tokens = self.model.get_audio_features(features, feature_attention_mask=mask)
|
||||
_, lengths = self.model.audio_tower._get_feat_extract_output_lengths(mask.sum(-1))
|
||||
if tokens.ndim != 2 or tokens.shape != (int(lengths.sum()), self.audio_dim) or not len(tokens):
|
||||
raise RuntimeError(f"Unexpected valid audio encoder shape: {tuple(tokens.shape)}")
|
||||
# get_audio_features removes padding and verifies the output length.
|
||||
return tokens.float().mean(0).cpu()
|
||||
|
||||
|
||||
def ego4d_clip_boxes(clip: dict[str, Any]) -> dict[int, list[dict[str, Any]]]:
|
||||
"""AVD tracks indexed by clip-local decoded frame; never use voice labels."""
|
||||
result: dict[int, list[dict[str, Any]]] = {}
|
||||
for person in clip.get("persons", []):
|
||||
if person.get("camera_wearer", False):
|
||||
continue
|
||||
seen = set()
|
||||
for track in person.get("tracking_paths", []):
|
||||
for box in track.get("track", []):
|
||||
frame = box.get("clip_frame")
|
||||
if frame is None:
|
||||
frame = box.get("frame")
|
||||
if frame is None:
|
||||
frame = int(box["video_frame"]) - int(clip["video_start_frame"])
|
||||
frame = int(frame)
|
||||
if frame < 0:
|
||||
raise ValueError("Negative clip-relative Ego4D box frame")
|
||||
# Some exports duplicate a person's boundary box in two tracks.
|
||||
if frame in seen:
|
||||
continue
|
||||
seen.add(frame)
|
||||
x, y, w, h = (float(box[k]) for k in ("x", "y", "width", "height"))
|
||||
result.setdefault(frame, []).append(
|
||||
{"id": str(person["person_id"]), "xyxy": [x, y, x + w, y + h]}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def load_ego4d_boxes(paths: list[Path], names: list[str]) -> dict[str, Any]:
|
||||
result = {}
|
||||
for path in paths:
|
||||
print(f"Reading provided tracks: {path}", flush=True)
|
||||
with path.open() as handle:
|
||||
payload = json.load(handle)
|
||||
for video in payload["videos"]:
|
||||
for clip in video["clips"]:
|
||||
uid = clip["clip_uid"]
|
||||
if uid in names:
|
||||
if uid in result:
|
||||
raise ValueError(f"Duplicate Ego4D annotation UID: {uid}")
|
||||
result[uid] = ego4d_clip_boxes(clip)
|
||||
del payload
|
||||
missing = set(names) - result.keys()
|
||||
if missing:
|
||||
raise ValueError(f"No provided tracks for: {sorted(missing)}")
|
||||
return result
|
||||
|
||||
|
||||
def build_media_and_boxes(args: argparse.Namespace, name: str, end: float, ego_boxes: dict[str, Any]):
|
||||
# Lazy imports allow --help and pure tests without Transformers/decoders.
|
||||
from streaming_turn_model_directly import CausalSourceAV, SourceClipSpec, _easycom_directory_specs
|
||||
|
||||
if args.dataset == "easycom":
|
||||
specs = _easycom_directory_specs(args.data_root / "Main" / "Video_Compressed" / name)
|
||||
else:
|
||||
specs = [SourceClipSpec(args.data_root / "v1" / "clips" / f"{name}.mp4")]
|
||||
media = CausalSourceAV(specs, output_fps=args.video_fps, required_duration=end,
|
||||
decoder_backend=args.decoder_backend, decoder_threads=args.decoder_threads,
|
||||
decoder_gpu_id=args.decoder_gpu_id)
|
||||
if args.dataset == "easycom":
|
||||
folder = "Face_Bounding_Boxes" if args.box_kind == "face" else "Head_Bounding_Boxes"
|
||||
annotations = []
|
||||
for clip in media.clips:
|
||||
path = args.data_root / "Main" / folder / name / f"{clip.path.stem}.json"
|
||||
with path.open() as handle:
|
||||
rows = json.load(handle)
|
||||
mapping = easycom_boxes(rows, args.box_kind)
|
||||
# Head annotations have a documented extra terminal record; it is
|
||||
# not a decoded frame and is never sampled.
|
||||
allowed = clip.num_frames + (1 if args.box_kind == "head" else 0)
|
||||
if set(mapping) != set(range(allowed)):
|
||||
raise ValueError(f"Box/video frame inventory mismatch: {path}")
|
||||
annotations.append(mapping)
|
||||
else:
|
||||
annotations = [ego_boxes[name]]
|
||||
if annotations[0] and max(annotations[0]) >= media.clips[0].num_frames:
|
||||
raise ValueError(f"Ego4D annotation frame exceeds source clip: {name}")
|
||||
return media, annotations
|
||||
|
||||
|
||||
def frame_crops(media, annotations, time: float, args: argparse.Namespace):
|
||||
clip_index, frame_index, source_pts = media._source_frame_for_time(time)
|
||||
clip = media.clips[clip_index]
|
||||
image = media.frame_at_or_before(time)
|
||||
# Dataset coordinates normally refer to the 1920px camera image (after
|
||||
# removing EasyCom's left banner). Explicit override for alternate exports.
|
||||
offset_x, offset_y = (clip.crop[:2] if clip.crop and args.box_coordinate_space == "encoded" else (0, 0))
|
||||
crops, records = [], []
|
||||
for person in annotations[clip_index].get(frame_index, []):
|
||||
box = crop_box(person["xyxy"], *image.size, offset_x=offset_x, offset_y=offset_y,
|
||||
expansion=args.crop_expansion)
|
||||
if box is not None:
|
||||
crops.append(image.crop(box))
|
||||
records.append({"id": person["id"], "box": box, "source_clip_index": clip_index,
|
||||
"source_frame_index": frame_index,
|
||||
"source_time": clip.dense_start + source_pts})
|
||||
return image, crops, records
|
||||
|
||||
|
||||
def validate_source_timeline(reference: dict[str, Any], media) -> None:
|
||||
"""Check source order/frames/offsets when the Omni file has direct metadata."""
|
||||
saved = (reference.get("direct_media") or {}).get("source_clips")
|
||||
if saved is None:
|
||||
print("NOTE: reference has no direct-media inventory; verify its dense source timeline manually.")
|
||||
return
|
||||
if len(saved) < len(media.clips):
|
||||
raise ValueError("Local source has more clips than the reference inventory")
|
||||
for old, new in zip(saved, media.metadata()["source_clips"]):
|
||||
if (Path(old["path"]).name != Path(new["path"]).name
|
||||
or old["source_frames"] != new["source_frames"]
|
||||
or abs(old["dense_start"] - new["dense_start"]) > 1e-5
|
||||
or abs(old["duration"] - new["duration"]) > 1e-5):
|
||||
raise ValueError(f"Local/reference source timeline differs at {new['path']}")
|
||||
|
||||
|
||||
def preview_boxes(media, annotations, args: argparse.Namespace, name: str, end: float) -> None:
|
||||
from PIL import ImageDraw
|
||||
|
||||
folder = args.output_root / f"{name}_preview"
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
ticks = list(causal_frame_ticks(end, args.video_fps))
|
||||
for index in np.linspace(0, len(ticks) - 1, min(4, len(ticks)), dtype=int):
|
||||
time = ticks[index] / args.video_fps
|
||||
image, _, records = frame_crops(media, annotations, time, args)
|
||||
draw = ImageDraw.Draw(image)
|
||||
for record in records:
|
||||
draw.rectangle(record["box"], outline="lime", width=4)
|
||||
draw.text(record["box"][:2], record["id"], fill="red")
|
||||
image.save(folder / f"frame_{ticks[index]:06d}.jpg")
|
||||
print(f"Box previews: {folder} (inspect before extracting all sessions)")
|
||||
|
||||
|
||||
def extract_local(media, annotations, encoder, times: torch.Tensor, args: argparse.Namespace) -> dict[str, Any]:
|
||||
# Canonical grid, not float32-serialized rounding near a tick boundary.
|
||||
ends = (torch.arange(len(times), dtype=torch.float64) + 1) / 5
|
||||
ticks = list(causal_frame_ticks(float(ends[-1]), args.video_fps))
|
||||
frame_times = torch.tensor(ticks, dtype=torch.float64) / args.video_fps
|
||||
frame_vectors = torch.zeros(len(ticks), encoder.visual_dim)
|
||||
frame_valid = torch.zeros(len(ticks), dtype=torch.bool)
|
||||
people, records, offsets = [], [], [0]
|
||||
for row, tick in enumerate(tqdm(ticks, desc="local vision", unit="frame", disable=args.no_progress)):
|
||||
_, crops, observations = frame_crops(media, annotations, tick / args.video_fps, args)
|
||||
vectors = encoder.images(crops)
|
||||
if vectors.shape != (len(crops), encoder.visual_dim) or not torch.isfinite(vectors).all():
|
||||
raise RuntimeError("Invalid person features")
|
||||
if len(vectors):
|
||||
stored = vectors.to(torch.bfloat16)
|
||||
frame_vectors[row] = stored.float().mean(0)
|
||||
frame_valid[row] = True
|
||||
people.append(stored)
|
||||
records.extend(observations)
|
||||
offsets.append(offsets[-1] + len(vectors))
|
||||
visual = torch.zeros(len(times), encoder.visual_dim)
|
||||
audio = torch.zeros(len(times), encoder.audio_dim)
|
||||
visual_valid = torch.zeros(len(times), dtype=torch.bool)
|
||||
audio_valid = torch.zeros(len(times), dtype=torch.bool)
|
||||
for row, end in enumerate(tqdm(ends.tolist(), desc="local audio/pool", unit="state", disable=args.no_progress)):
|
||||
visual[row], present = pooled_recent_visual(
|
||||
frame_times, frame_vectors, frame_valid, end, args.visual_window_seconds,
|
||||
)
|
||||
visual_valid[row] = present
|
||||
waveform = media.audio_slice(max(0., end - args.audio_window_seconds), end)
|
||||
audio[row] = encoder.audio(waveform)
|
||||
audio_valid[row] = True # Silence is an observation, not missing audio.
|
||||
if not torch.isfinite(audio).all() or not torch.isfinite(visual).all():
|
||||
raise RuntimeError("Non-finite local encoder outputs")
|
||||
return {
|
||||
"visual": visual.to(torch.bfloat16), "audio": audio.to(torch.bfloat16),
|
||||
"visual_valid": visual_valid, "audio_valid": audio_valid,
|
||||
"persons": {"features": torch.cat(people) if people else torch.empty(0, encoder.visual_dim, dtype=torch.bfloat16),
|
||||
"frame_times": frame_times, "frame_offsets": torch.tensor(offsets, dtype=torch.long),
|
||||
"records": records},
|
||||
}
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("--dataset", choices=("easycom", "ego4d"), default="easycom")
|
||||
p.add_argument("--data-root", required=True, type=Path)
|
||||
p.add_argument("--feature-root", required=True, type=Path)
|
||||
p.add_argument("--target-root", required=True, type=Path)
|
||||
p.add_argument("--output-root", required=True, type=Path)
|
||||
p.add_argument("--samples", required=True, nargs="+", help="EasyCom session numbers or Ego4D clip UIDs")
|
||||
p.add_argument("--ego4d-annotations", type=Path, nargs="+")
|
||||
p.add_argument("--model-id", default="Qwen/Qwen2.5-Omni-3B")
|
||||
p.add_argument("--encoder-id", help="Stable checkpoint identity for relocated copies; default: model-id")
|
||||
p.add_argument("--device", default="cuda:0")
|
||||
p.add_argument("--dtype", choices=("float32", "float16", "bfloat16"), default="bfloat16")
|
||||
p.add_argument("--flash-attention-2", action="store_true")
|
||||
p.add_argument("--video-fps", type=float, default=2.)
|
||||
p.add_argument("--audio-window-seconds", type=float, default=1.)
|
||||
p.add_argument("--visual-window-seconds", type=float, default=1.)
|
||||
p.add_argument("--box-kind", choices=("face", "head"), default="face")
|
||||
p.add_argument("--box-coordinate-space", choices=("camera", "encoded"), default="camera")
|
||||
p.add_argument("--crop-expansion", type=float, default=1.)
|
||||
p.add_argument("--max-pixels", type=int, default=224 * 224)
|
||||
p.add_argument("--image-batch-size", type=int, default=8)
|
||||
p.add_argument("--decoder-backend", choices=("decord", "torchcodec", "pynvcodec"), default="decord")
|
||||
p.add_argument("--decoder-threads", type=int, default=1)
|
||||
p.add_argument("--decoder-gpu-id", type=int, default=0, help="Visible-device index, not physical GPU number")
|
||||
p.add_argument("--max-steps", type=int, help="Smoke test only; partial sidecars are rejected by training")
|
||||
p.add_argument("--preview-only", action="store_true", help="Save box overlays without loading the model")
|
||||
p.add_argument("--overwrite", action="store_true")
|
||||
p.add_argument("--no-progress", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
for name in ("video_fps", "audio_window_seconds", "visual_window_seconds", "crop_expansion"):
|
||||
if not np.isfinite(getattr(args, name)) or getattr(args, name) <= 0:
|
||||
p.error(f"--{name.replace('_', '-')} must be finite and positive")
|
||||
if args.audio_window_seconds > 30:
|
||||
p.error("--audio-window-seconds must not exceed the audio processor's 30s limit")
|
||||
if args.max_pixels < 4 * 28 * 28 or args.image_batch_size <= 0:
|
||||
p.error("max-pixels must be >=3136; image-batch-size must be positive")
|
||||
if args.max_steps is not None and args.max_steps <= 0:
|
||||
p.error("--max-steps must be positive")
|
||||
if args.output_root.resolve() == args.feature_root.resolve():
|
||||
p.error("Local outputs must not overwrite original Omni features")
|
||||
return args
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
names = [normalize_sample_name(value, args.dataset) for value in args.samples]
|
||||
if len(set(names)) != len(names):
|
||||
raise ValueError("Duplicate samples")
|
||||
for name in names:
|
||||
for path in (args.feature_root / f"{name}.pt", args.target_root / f"{name}.npy"):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
if (args.output_root / f"{name}.pt").exists() and not args.overwrite and not args.preview_only:
|
||||
raise FileExistsError(f"{name}.pt exists; use a new output root or --overwrite")
|
||||
ego_boxes = {}
|
||||
if args.dataset == "ego4d":
|
||||
paths = args.ego4d_annotations or [args.data_root / "v1" / "annotations" / f"av_{split}.json"
|
||||
for split in ("train", "val")]
|
||||
ego_boxes = load_ego4d_boxes(paths, names)
|
||||
args.output_root.mkdir(parents=True, exist_ok=True)
|
||||
encoder = None
|
||||
for name in names:
|
||||
path = args.feature_root / f"{name}.pt"
|
||||
reference = torch.load(path, map_location="cpu", weights_only=False)
|
||||
times, indices = reference_alignment(reference)
|
||||
target = np.load(args.target_root / f"{name}.npy", allow_pickle=False, mmap_mode="r")
|
||||
if target.shape != (len(times), 3):
|
||||
raise ValueError(f"Reference/target mismatch: {name}, {target.shape} vs {len(times)} states")
|
||||
full_steps = len(times)
|
||||
if args.max_steps is not None:
|
||||
times, indices = times[:args.max_steps], indices[:args.max_steps]
|
||||
media, annotations = build_media_and_boxes(args, name, len(times) / 5, ego_boxes)
|
||||
validate_source_timeline(reference, media)
|
||||
if args.preview_only:
|
||||
preview_boxes(media, annotations, args, name, len(times) / 5)
|
||||
continue
|
||||
if encoder is None:
|
||||
encoder = FrozenQwenLocalEncoder(args)
|
||||
print(f"{name}: {len(times)} states; native visual={encoder.visual_dim}D audio={encoder.audio_dim}D", flush=True)
|
||||
output = extract_local(media, annotations, encoder, times, args)
|
||||
output.update({
|
||||
"format": LOCAL_FORMAT, "dataset": args.dataset, "sample_id": name,
|
||||
"times": times, "label_indices": indices, "partial": len(times) != full_steps,
|
||||
"reference_features": str(path.resolve()), "direct_media": media.metadata(),
|
||||
"config": {
|
||||
"encoder_id": args.encoder_id or args.model_id,
|
||||
"model_config_sha256": encoder.model_config_sha256,
|
||||
"transformers": "4.52.3", "dtype": args.dtype,
|
||||
"attention": "flash_attention_2" if args.flash_attention_2 else "default",
|
||||
"visual_dim": encoder.visual_dim, "audio_dim": encoder.audio_dim,
|
||||
"visual_fps": args.video_fps, "visual_window_seconds": args.visual_window_seconds,
|
||||
"audio_window_seconds": args.audio_window_seconds,
|
||||
"visual_pooling": "spatial-mean_then_equal-person-mean_then_equal-valid-frame-mean",
|
||||
"audio_pooling": "valid-output-token-mean", "audio_source": "video-mixed-mono-16khz",
|
||||
"box_kind": args.box_kind if args.dataset == "easycom" else "ego4d-provided-tracks",
|
||||
"box_coordinate_space": args.box_coordinate_space, "crop_expansion": args.crop_expansion,
|
||||
"max_pixels": args.max_pixels, "uses_speech_labels": False,
|
||||
"causal_policy": "half-open-trailing-window", "box_protocol": "provided-boxes-not-online-detector",
|
||||
},
|
||||
})
|
||||
destination = args.output_root / f"{name}.pt"
|
||||
# Atomic replacement: a failed extraction never looks like a ready file.
|
||||
with tempfile.NamedTemporaryFile(dir=args.output_root, prefix=f".{name}.", suffix=".tmp", delete=False) as handle:
|
||||
temporary = Path(handle.name)
|
||||
try:
|
||||
torch.save(output, temporary)
|
||||
os.replace(temporary, destination)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
print(f"Saved {destination}; V-valid={output['visual_valid'].float().mean():.1%}, "
|
||||
f"A-valid={output['audio_valid'].float().mean():.1%}, partial={output['partial']}", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+23
-2
@@ -38,6 +38,10 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument("--target-root", type=Path, required=True)
|
||||
parser.add_argument("--checkpoint", type=Path, required=True)
|
||||
parser.add_argument("--output-dir", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--local-feature-root", type=Path,
|
||||
help="Required for a local-fusion checkpoint; modalities come from the checkpoint.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dataset",
|
||||
choices=DATASET_NAMES,
|
||||
@@ -113,6 +117,7 @@ def _predict_states(
|
||||
description: str,
|
||||
show_progress: bool,
|
||||
loss_name: str,
|
||||
local_features: torch.Tensor | None = None,
|
||||
) -> tuple[np.ndarray, np.ndarray | None]:
|
||||
chunks: list[torch.Tensor] = []
|
||||
transition_chunks: list[torch.Tensor] = []
|
||||
@@ -130,11 +135,15 @@ def _predict_states(
|
||||
dtype=torch.float32,
|
||||
non_blocking=True,
|
||||
)
|
||||
side = (
|
||||
local_features[start : start + batch_size].to(device=device, dtype=torch.float32)
|
||||
if local_features is not None else None
|
||||
)
|
||||
if head.transition_auxiliary:
|
||||
activity_logits, transition_logits = head.forward_with_transition(batch)
|
||||
activity_logits, transition_logits = head.forward_with_transition(batch, side)
|
||||
transition_chunks.append(transition_logits.sigmoid().cpu())
|
||||
else:
|
||||
activity_logits = head(batch)
|
||||
activity_logits = head(batch, side)
|
||||
chunks.append(
|
||||
marginal_probabilities_from_logits(activity_logits, loss_name).cpu()
|
||||
)
|
||||
@@ -357,6 +366,13 @@ def main() -> None:
|
||||
)
|
||||
head_config = checkpoint_head_config(checkpoint_payload)
|
||||
loss_name = checkpoint_loss_name(checkpoint_payload)
|
||||
local_contract = checkpoint_payload.get("local_contract")
|
||||
if bool(head.local_input_size) != bool(local_contract):
|
||||
raise ValueError("Checkpoint local dimensions/contract disagree")
|
||||
if bool(head.local_input_size) != (args.local_feature_root is not None):
|
||||
raise ValueError(
|
||||
"--local-feature-root must be supplied exactly when the checkpoint uses local fusion"
|
||||
)
|
||||
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
prediction_dir = args.output_dir / "pred"
|
||||
@@ -384,6 +400,9 @@ def main() -> None:
|
||||
name,
|
||||
expected_mode=expected_mode,
|
||||
dataset=args.dataset,
|
||||
local_feature_root=args.local_feature_root,
|
||||
local_modalities=local_contract["modalities"] if local_contract else "av",
|
||||
local_contract=local_contract,
|
||||
)
|
||||
if data.hidden_size != int(head_config["hidden_size"]):
|
||||
raise ValueError(
|
||||
@@ -403,6 +422,7 @@ def main() -> None:
|
||||
description=f"infer {name}",
|
||||
show_progress=not args.no_progress,
|
||||
loss_name=loss_name,
|
||||
local_features=data.local_features,
|
||||
)
|
||||
predictions, valid = _align_to_egospeak_timeline(
|
||||
probabilities,
|
||||
@@ -464,6 +484,7 @@ def main() -> None:
|
||||
"checkpoint_train_sessions": checkpoint_payload.get("train_sessions"),
|
||||
"checkpoint_dev_sessions": checkpoint_payload.get("dev_sessions"),
|
||||
"layer_fusion": head.layer_fusion,
|
||||
"local_contract": local_contract,
|
||||
"transition_auxiliary": head.transition_auxiliary,
|
||||
"transition_objective": checkpoint_payload.get(
|
||||
"training_config",
|
||||
|
||||
@@ -10,5 +10,10 @@ Qwen2.5-Omni 是端到端全模态模型，统一处理文本、图像、音频
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
当前特征提取应优先保持 Thinker 的原生流式输入语义：按固定 AV 块提交、保持同一开放媒体 turn、让已提交块进入持久 KV。因果前缀反复重算可作为严格离线参照，但不是原作的在线执行方式。预先重编码成 2 FPS 便于构造严格网格，却可能改变原视频末尾不足完整块时的抽帧语义，因此直接解码方案必须显式验证时间戳、最终帧和目标长度。
|
||||
当前特征提取借鉴其分块编码和 AV 时间对齐：本项目按固定块提交，在开放媒体 turn 内保留已提交块的 KV，并处理当前部分块。因果前缀反复重算是另一个连续窗口参照，两种输入组织不必数值等价。预先重编码成 2 FPS 便于构造严格网格，但直接解码或重编码都需显式验证时间戳、末尾处理和目标对齐。
|
||||
|
||||
## 2026-08-27 补充：适用边界
|
||||
|
||||
因果窗口参照可以离线提取，但不指可以看未来的双向教师。论文的块式编码/TMRoPE、官方处理器的 AV 交织参数，以及本项目的开放 turn、部分块重算和滚动 KV 策略应分开描述，不能把本项目策略直接写成官方已提供的完整在线 API。
|
||||
|
||||
视频采样率、编码/交织块长和预测更新频率是三个不同量。2 FPS 输入可以支持 5 Hz 决策，但并非每个决策点都有新视频帧。短期高频分支是本项目的适配假设，不是 Qwen 原作要求；缩短块长也不保证与连续窗口逐数值等价。
|
||||
|
||||
@@ -1,15 +1,19 @@
|
||||
# ROMA: A Real-Time Omni-Multimodal Assistant
|
||||
# ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
|
||||
|
||||
- 年份：2026
|
||||
- 发表状态：本轮按 arXiv 预印本记录，未核验会议录用
|
||||
- 阅读深度：深入阅读
|
||||
- 官方链接：https://arxiv.org/abs/2601.10323
|
||||
- 本地参考实现：/Users/wx44wx/Code/ROMA
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
ROMA 面向实时全模态助手，把连续音视频整理成同步的流式单元，通过分块时间位置编码和持久 KV 维持在线上下文。模型把内容理解与“是否说话”的时机判断分离，设置独立 Speak Head，并对最后若干层隐藏状态做可学习聚合。训练包含流式模板对齐和时间感知决策阶段，目标是同时支持主动响应、被动倾听和打断处理。
|
||||
ROMA 面向实时全模态助手，把连续音视频整理成同步流式单元，通过分块时间位置编码和持久 KV 维持上下文。独立 Speak Head 将响应时机与内容生成分开，并聚合末层隐藏状态。两阶段训练先适配流式格式，再学习主动响应；论文覆盖被动问答、主动事件提醒和实时描述，而非专门评测多人接话或语音打断。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
最值得复用的不是照搬其 1 秒块，而是三个思想：最后四层可学习融合、表示头与时机头解耦、对临界转换区域增加决策监督。ROMA 的时间分辨率和任务输出与当前 5 Hz EgoSpeak 预测不同，不能直接把 1 秒决策粒度当作替代方案。本地实现还应结合论文逐项核查，不能默认所有描述组件都已完整落地。
|
||||
最值得复用的不是照搬其 1 秒块，而是最后四层融合、时机与生成解耦，以及流式格式适配。ROMA 的任务输出与当前 5 Hz EgoSpeak 预测不同，不能直接把 1 秒决策粒度当作替代方案。对 transition 区域额外加权属于我们的迁移设想，不应归为 ROMA 原作结论。
|
||||
|
||||
## 2026-08-27 补充核对
|
||||
|
||||
本地 `src/llamafactory/data/mm_plugin.py` 可见 `chunk_duration=1.0` 的音频切分及按秒组织 AV token 的路径。因而把它直接改成 0.2 秒会改变输入组织，需要同步核对音频编码、视觉时序聚合、位置编码、训练分布和端到端延迟；不能只改一个参数就宣称保持原作语义。保留其块式上下文、另加高频决策分支，是待验证的另一条路线。
|
||||
|
||||
@@ -6,9 +6,10 @@
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
该工作在面对面人机对话中联合研究轮次转换和 backchannel，融合语言、声学和视觉信号，并报告多模态模型相对既有方法的改进。其任务把正常接管、简短反馈和持续倾听视为可区分的互动事件。
|
||||
该工作构建超过 210 小时的 MM-F2F 对话数据，研究联合语言、声学和视觉线索的轮次转换及 backchannel 预测。框架强调模态间关系，并支持不同输入模态组合，论文报告其在两个任务上的改进。这里按摘要级证据记录，不把这些结果直接外推到第一视角多人在线预测。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
它支持把笼统的 wearer/others 活动进一步拆成 transition 子类型进行诊断。即使训练标签暂时不变，也可以从目标序列派生 onset、offset、overlap onset 和短促 backchannel-like 片段，分别报告性能，判断模型真正改善了哪种动态。
|
||||
|
||||
补充边界：短促语音不必然是 backchannel；仅凭三路活动标签不能得到可靠的互动功能标签。若要报告真正的 backchannel 或说话对象识别，需要相应标注和协议，不能把活动时长规则当作语义真值。
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# 多人 AV、多速率流式处理与四条实验路线
|
||||
|
||||
记录日期：2026-08-27。以下是文献启发下的本项目假设与实验建议，不是已完成的实现、已验证的增益或已确立的论文新颖性结论。
|
||||
|
||||
## 本轮相关阅读
|
||||
|
||||
- 块式多模态基础：[Qwen2.5-Omni](01_qwen2_5_omni.md)、[ROMA](05_roma.md)。ROMA 本轮按预印本记录，不与已核验会议论文混称。
|
||||
- 流式输入与分层记忆：[VideoLLM-online](21_videollm_online.md)、[ProVideLLM](22_providellm.md)。
|
||||
- 多人对应与第一视角感知：[Multi-speaker Attention Alignment](23_multi_speaker_attention_alignment.md)、[EgoAVU](24_egoavu.md)、[AV Speaker Localization](25_egocentric_av_speaker_localization.md)、[Auditory Attention Localization](26_egocentric_auditory_attention.md)。
|
||||
- 任务及评测：[EgoSpeak](02_egospeak.md)、[Multimodal Turn-taking and Backchannel](16_multimodal_turn_taking_backchannel.md)、[OmniMMI](27_omnimmi.md)。
|
||||
|
||||
每篇笔记分别保存中文转述摘要、官方来源、阅读深度与适用边界；新编号为 21–27。
|
||||
|
||||
## 现有证据与不能推断的结论
|
||||
|
||||
- [open-stream 与 causal-prefix](../results/easycom_open_stream_vs_causal_prefix_2026-08-19/README.md)的单次比较中，后者 mean mAP 高 0.514，0.2 秒高 2.393。差异集中于短 horizon，支持检验分块/近期信息假设，但窗口保留、AV 排列、前端边界等也可能参与，不能认定 chunk 是唯一原因。
|
||||
- [最后四层读取](../results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/README.md)的记录相对旧单层基线约增 2.015 mean mAP；训练配置尚需匹配，权重近均匀，尚不能证明 learned mix 优于固定 mean。
|
||||
- [四状态/BCE 消融](../results/easycom_loss_structure_4ep_2026-08-26/README.md)与[R1/R2](../results/easycom_transition_auxiliary_4ep_2026-08-27/README.md)没有建立明显的活动预测提升。四状态是对 overlap 的表达改进，不是多个人身份建模；R2 是离线标签上的精确期望奖励控制，不是交互 RL。
|
||||
- [稀疏 LoRA 试验](../results/easycom_sparse_supervision_vs_lora_2026-08-24/README.md)退化，但这是特定训练配方的诊断，不能据此认定 LoRA 普遍无效。
|
||||
|
||||
## 四条路线：展开说明
|
||||
|
||||
**路线一：多速率近期/历史适配器。** 当前 open-stream 已经每 200 ms 产生状态，所以要改善的是状态中的局部证据，而不是补一个已有的更新频率。保留块式 Omni 历史及 last-4 语义表征，另设每 200 ms 更新的短期因果窗口，保留最近 1–2 秒音频动态和轻量人脸/头部运动特征，再用门控或交叉注意力融合。200 ms 是更新步长，不要求音频编码器每次只看孤立的 200 ms。5–10 FPS 局部视频是待测配置，必须从原始视频读取而非对 2 FPS 重复插帧；没有新帧时保留旧特征并记录时间戳/新鲜度。先冻结骨干，比较原头、近期分支单独使用、普通拼接和双分支融合，并控制参数量及媒体预算；只有超过这些对照，才能把增益归因于多速率组织。
|
||||
|
||||
**路线二：说话人感知 AV 对齐。** 将可见参与者的区域特征按因果 track 组织成集合，在混合音频条件下学习哪些人及其动作与当前互动相关；另留不可见佩戴者和画外/未定位声源表示，不能假设所有声音都有对应人脸。可用训练集的逐人语音活动作辅助监督，而主预测仍输出 silence/wearer-only/others-only/overlap，以兼容三路评测。多人并发时不能简单相加各人的发声概率当作 others 概率；可直接从集合特征训练全局活动头。真值框、身份和说话对象只能用于明确声明的监督或 oracle 分析，正式在线系统要计入检测/跟踪误差和成本。若增益尤其体现在多参与者、重叠和交接子集，才支持“多人 AV 对齐”的机制解释。
|
||||
|
||||
**路线三：分块稳定性与严格因果蒸馏。** 先按块内十个 200 ms 相位统计误差，控制事件比例，并对相同时间点采用不同分块边界比较，确认是否存在边界依赖。之后分别测试相位输入、随机分块起点、以及 causal-prefix 教师的软预测蒸馏，不一次加入所有模块。教师和学生必须只看截至 t 的真实媒体，并匹配上下文预算、标签及训练划分；教师可以离线计算，但不能看未来。目标是提高预测正确性与对分块选择的鲁棒性，而不是逼不同输入语义的 hidden state 达到 cosine=1。也不能强迫相邻时刻的预测相同，否则会抹掉真正的快速语音转换；应比较同一 t 的不同实现/分块视图。
|
||||
|
||||
**路线四：选择性 AV 适配。** 在冻结骨干的分支/头上先获得可重复收益，再分别试实际存在的模态连接层、少量上层 LoRA，使用较小学习率并与冻结对照匹配训练样本和更新步数；不要同时解冻全部模块。训练可保留活动预测，并将逐人活动/跨模态对应作为独立可消融的监督；是否加 dropout 或教师约束也要单独验证。权重更新后必须重新计算受影响的媒体特征或 KV，不能把旧骨干缓存当作新骨干表示。超参与 checkpoint 应在训练 session 内部留出数据选择；若继续不设 dev，则事先固定配置和训练时长，不根据正式测试集择优。该路线用于提升任务适配能力，但不是“八卡就保证有收益”的承诺。
|
||||
|
||||
## 共同验证要求与论文定位
|
||||
|
||||
1. 先固定主协议、输入条件和活动 loss；四状态/BCE 保留为对照，不把微小差异包装成主要贡献。
|
||||
2. 比较 A-only、V-only、AV；在有可靠标注时分层报告参与人数、wearer onset/offset、overlap 与交接。`others` 内部 A→B 的转换不能仅凭三路标签恢复。
|
||||
3. 除总体及 wearer AP，还报告分块相位、按 session 的差异、匹配多 seed，以及前端+融合+骨干的真实延迟、显存和吞吐。200 ms 更新周期不等于实测延迟达标。
|
||||
4. 八卡可以并行实验，但新局部/说话人分支需要先提取原始高频媒体和因果 tracks，不能直接把现有 `[T,4,2048]` 冻结状态当作它们的输入替代。
|
||||
5. 路线一解决时间尺度，路线二对应多人特性，路线三是边界控制，路线四是后续适配。它们是可以逐项证伪的研究方向，不是需要全部叠加的四个模块；论文贡献应由最终有效机制决定。
|
||||
|
||||
候选定位：在第一视角多人环境中，以说话人感知的多速率 AV 适配提高因果活动/接话预测。当前三/四状态评测不等同于全参与者的 next-speaker 分配；若后续声称身份级多人轮次能力，需另设逐人标签与评测。
|
||||
@@ -0,0 +1,19 @@
|
||||
# VideoLLM-online: Online Video Large Language Model for Streaming Video
|
||||
|
||||
- 作者：Joya Chen 等
|
||||
- 会议：CVPR 2024
|
||||
- 阅读深度：摘要级筛选（官方会议页面）
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[CVF 论文页](https://openaccess.thecvf.com/content/CVPR2024/html/Chen_VideoLLM-online_Online_Video_Large_Language_Model_for_Streaming_Video_CVPR_2024_paper.html)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
该工作指出，面向预先切好的视频片段训练的视觉语言模型不自然地支持连续视频。LIVE 框架联合设计流式语言建模目标、离线时序标注到在线对话的数据转换，以及高效推理流程，让视频输入与响应时刻对齐。VideoLLM-online 用于连续视频中的在线交互，并评估多类视频任务。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
它提示“流式适配”不仅是复用 KV，还包括训练和推理时间契约一致。可借鉴因果样本构造与逐步决策设计，但原作不是混合音频中的多人接话预测，也没有证明任意 200 ms AV 切块有效。
|
||||
|
||||
## 本项目的待验证迁移
|
||||
|
||||
比较同样观察截止时间、同样训练标签下的原始 open-stream 与专门流式适配头；所有教师和学生输入都只能使用当前时刻以前的媒体。
|
||||
@@ -0,0 +1,20 @@
|
||||
# Streaming VideoLLMs for Real-Time Procedural Video Understanding
|
||||
|
||||
- 模型简称：ProVideLLM
|
||||
- 作者：Dibyadip Chatterjee 等
|
||||
- 会议：ICCV 2025
|
||||
- 阅读深度：摘要及引言筛选（缓存设计概述，未复现）
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[CVF 论文页](https://openaccess.thecvf.com/content/ICCV2025/html/Chatterjee_Streaming_VideoLLMs_for_Real-Time_Procedural_Video_Understanding_ICCV_2025_paper.html)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
ProVideLLM 面向长时程、实时的程序性视频理解，将长期观察压缩成文本 token，同时用 DETR-QFormer 产生的视觉 token 保留近期细节。两类 token 组成多模态缓存，在保留任务历史的同时降低长视频的存储与计算开销；论文在多个程序性任务和数据集上评估这一设计。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
可借鉴“近期细节与长期上下文采用不同表示”的思想，而非将整段对话压入单个最终状态。但其模态缓存主要是视觉与文本，不是已验证的音视频接话模型；文本化长期历史也可能丢掉声学韵律。
|
||||
|
||||
## 本项目的待验证迁移
|
||||
|
||||
先测试冻结 Omni 历史状态与短期 AV 分支的融合，并与短期分支单独使用、普通拼接及参数量匹配头对照。论文速度数字不能直接用作本项目硬件和任务的速度预期。
|
||||
@@ -0,0 +1,19 @@
|
||||
# Multi-speaker Attention Alignment for Multimodal Social Interaction
|
||||
|
||||
- 作者：Liangyang Ouyang 等
|
||||
- 会议：CVPR 2026；预印本初发于 2025 年
|
||||
- 阅读深度：重点章节阅读（问题分析、方法及输入要求）
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[CVF 论文页](https://openaccess.thecvf.com/content/CVPR2026/html/Ouyang_Multi-speaker_Attention_Alignment_for_Multimodal_Social_Interaction_CVPR_2026_paper.html)、[arXiv 正文](https://arxiv.org/pdf/2511.17952)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
该工作分析多人社交视频中视觉 token 与说话人话语的对齐不足，指出加入图像并不总能改善社交推理。方法动态选择相关注意力头，再依照说话人位置施加自适应注意力偏置，增强同一说话人的视觉区域与话语联系，无需新增可训练参数。作者在多个视觉语言模型和社交任务上验证该干预。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
它支持检验按参与者组织局部视觉证据的价值，而不是仅增加全图 token。**原作使用视频、带身份的话语转录和说话人框，主要研究视觉—文本对齐，不是从单路混合音频直接学到 AV 说话人对应。** 原作结果不能直接作为本项目 200 ms 接话预测的证据。
|
||||
|
||||
## 本项目的待验证迁移
|
||||
|
||||
用当前可见 track 的区域特征构成参与者集合，训练音频条件下的融合和逐人活动辅助任务。身份标签可作训练监督；推理若使用真值框或真值说话人归属，必须单列为 oracle 条件，不能与仅原始 AV 的结果混报。还需保留佩戴者和画外/未定位说话人的表示。
|
||||
@@ -0,0 +1,19 @@
|
||||
# EgoAVU: Egocentric Audio-Visual Understanding
|
||||
|
||||
- 作者：Ashish Seth 等
|
||||
- 会议：CVPR 2026
|
||||
- 阅读深度：摘要及官方实现说明筛选
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[arXiv 论文](https://arxiv.org/abs/2602.06139)、[官方实现](https://github.com/facebookresearch/EgoAVU)、[CVPR 会议目录](https://openaccess.thecvf.com/CVPR2026?day=2026-06-06)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
EgoAVU 用数据生成与筛选流程，将第一视角视频中的音频和视觉关联补充到叙述、问题与答案中，构建训练集及人工核验的评测集。评测发现部分多模态模型偏重视觉、忽略音频或难以对应声音来源。围绕联合 AV 信息进行微调能改善相应理解能力，并在其他相关评测上体现迁移收益。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
“能输入 AV”不等于“有效利用 AV 对应关系”。因此应做 audio-only、video-only、AV 和错配/遮挡诊断，再决定是否训练模态连接层或加入对齐监督。
|
||||
|
||||
## 适用边界
|
||||
|
||||
其任务主要是理解与问答，并非多人轮次预测；大规模指令数据上的微调收益不保证迁移到小规模 EasyCom。由此提出的局部 LoRA、模态 dropout 等是我们的候选控制，而非本次阅读已验证的原作训练结论。
|
||||
@@ -0,0 +1,19 @@
|
||||
# Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization
|
||||
|
||||
- 作者：Hao Jiang、Calvin Murdock、Vamsi Krishna Ithapu
|
||||
- 会议：CVPR 2022
|
||||
- 阅读深度：摘要及引言筛选
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[CVF 论文页](https://openaccess.thecvf.com/content/CVPR2022/html/Jiang_Egocentric_Deep_Multi-Channel_Audio-Visual_Active_Speaker_Localization_CVPR_2022_paper.html)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
该工作联合第一视角视频与多通道麦克风阵列音频，检测佩戴者及周围人的语音活动并定位活跃说话人。模型考虑头部运动、遮挡、噪声与画外声源，在整个空间方向上定位讲话者，同时预测佩戴者自己的活动，面向真实增强现实设备的实时感知。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
它表明多人 AV 建模不能只关注画面里的一张脸，应区分佩戴者、可见参与者与画外声源。其任务是当前活动检测/定位，不是预测未来接话。
|
||||
|
||||
## 适用边界
|
||||
|
||||
原作利用多通道空间信息。我们的 mono/混合音轨不能凭空恢复同等阵列线索；EasyCom 多通道扩展需与当前输入设定分开报告。人脸分支应保留不可见声源通道，不能把“没有可见人讲话”直接判为 silence。
|
||||
@@ -0,0 +1,19 @@
|
||||
# Egocentric Auditory Attention Localization in Conversations
|
||||
|
||||
- 作者：Fiona Ryan、Hao Jiang、Abhinav Shukla、James M. Rehg、Vamsi Krishna Ithapu
|
||||
- 会议：CVPR 2023
|
||||
- 阅读深度：摘要及引言筛选
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[CVF 论文页](https://openaccess.thecvf.com/content/CVPR2023/html/Ryan_Egocentric_Auditory_Attention_Localization_in_Conversations_CVPR_2023_paper.html)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
该工作研究多人噪声对话中佩戴者在听谁，而不只检测谁在发声。方法结合第一视角视频、多通道音频、时空 AV 特征与场景推理，预测佩戴者听觉注意目标的热图，并在多人对话数据上与基线比较。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
说话人的存在、当前发声和佩戴者关注对象是不同变量。可把参与者区域、头部朝向和互动关系视为接话预测的候选线索，但不能默认视线或头朝向等于真实听觉注意。
|
||||
|
||||
## 本项目的待验证迁移
|
||||
|
||||
以参与者集合而非全图平均特征做融合，并在有可靠标注时分析互动对象相关的错误。不要仅凭 wearer/others 活动标签生成“在听谁”的真值，也不要把原作检测结果当作未来轮次预测成绩。
|
||||
@@ -0,0 +1,19 @@
|
||||
# OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
|
||||
|
||||
- 作者：Yuxuan Wang 等
|
||||
- 会议：CVPR 2025
|
||||
- 阅读深度：摘要级筛选
|
||||
- 记录日期：2026-08-27
|
||||
- 官方来源：[CVF 论文页](https://openaccess.thecvf.com/content/CVPR2025/html/Wang_OmniMMI_A_Comprehensive_Multi-modal_Interaction_Benchmark_in_Streaming_Video_Contexts_CVPR_2025_paper.html)
|
||||
|
||||
## 摘要（中文转述）
|
||||
|
||||
OmniMMI 面向持续输入下的多模态交互，评估流式理解与主动推理等能力，并提出多模态复用框架 M4，尝试让模型在生成时继续接收视觉和听觉信息。工作强调完整视频后的问答准确率不足以衡量在线交互，需要考虑响应时刻与连续输入。
|
||||
|
||||
## 与当前项目的关系
|
||||
|
||||
可借鉴“判断内容是否正确”和“何时做出判断”分开评估的思想。我们的 headline 仍应保持 EgoSpeak 的因果预测协议，再补充事件级时机指标、漏报/误触发和延迟。
|
||||
|
||||
## 适用边界
|
||||
|
||||
主动提醒与人类自然接话不是同一个任务。不能直接用其回答准确率或响应阈值替代 EasyCom/Ego4D 的活动 AP，也不能只报每秒输出五次就宣称端到端实时。
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
本目录持久化记录我们在本项目讨论中实际阅读、分析或筛选过的论文。这里的“摘要”均为面向当前课题的中文转述，不是原文摘要的复制。
|
||||
|
||||
最后更新：2026-08-26
|
||||
最后更新：2026-08-27
|
||||
|
||||
本轮专题：[多人 AV、多速率流式处理与四条实验路线](2026-08-27_multirate_speaker_aware_routes.md)。新增 21–27 七篇笔记，并补充 01、05、16 的适用边界；摘要级筛选不等同于全文复现。
|
||||
|
||||
## 阅读深度
|
||||
|
||||
@@ -34,19 +36,29 @@
|
||||
| 18 | [LLM-Enhanced Dialogue Management](18_llm_enhanced_dialogue_management.md) | 2025/2026 | 摘要级筛选 | 模块化流式对话控制状态 |
|
||||
| 19 | [Aligning Spoken Dialogue Models from User Interactions](19_aligning_spoken_dialogue_from_interactions.md) | 2025 | 摘要级筛选 | 从用户交互构造偏好数据 |
|
||||
| 20 | [Optimizing Conversational Quality with RLAIF](20_spoken_dialogue_rlaif.md) | 2026 | 摘要级筛选 | 块级语音生成与多奖励偏好优化 |
|
||||
| 21 | [VideoLLM-online](21_videollm_online.md) | 2024 | 摘要级筛选 | 流式训练目标、时间对齐与在线推理 |
|
||||
| 22 | [ProVideLLM](22_providellm.md) | 2025 | 摘要及引言筛选 | 短期细粒度视觉与长期压缩记忆 |
|
||||
| 23 | [Multi-speaker Attention Alignment](23_multi_speaker_attention_alignment.md) | 2026 | 重点章节阅读 | 说话人区域与话语对齐；原工作是视觉—文本设定 |
|
||||
| 24 | [EgoAVU](24_egoavu.md) | 2026 | 摘要及官方实现说明筛选 | 第一视角 AV 对应关系、模态偏置与任务适配 |
|
||||
| 25 | [Egocentric Deep Multi-Channel AV Active Speaker Localization](25_egocentric_av_speaker_localization.md) | 2022 | 摘要及引言筛选 | 佩戴者、可见及画外说话人的活动与位置 |
|
||||
| 26 | [Egocentric Auditory Attention Localization in Conversations](26_egocentric_auditory_attention.md) | 2023 | 摘要及引言筛选 | 区分谁在说话与佩戴者在听谁 |
|
||||
| 27 | [OmniMMI](27_omnimmi.md) | 2025 | 摘要级筛选 | 主动流式交互评测与多模态复用 |
|
||||
|
||||
## 当前综合判断
|
||||
|
||||
对本项目最直接的主线不是立即把三分类预测改成通用 RL，而是：
|
||||
截至 2026-08-27，四状态/BCE 和 R1/R2 的活动预测增益较小；R1/R2 仅有两个已报告 seed，不能视作最终结论。最后四层读取有较大单次增益，但仍需匹配训练配置的多 seed 对照。参见[损失消融](../results/easycom_loss_structure_4ep_2026-08-26/README.md)、[transition 消融](../results/easycom_transition_auxiliary_4ep_2026-08-27/README.md)和[last-4 结果](../results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/README.md)。
|
||||
|
||||
1. 以 Qwen2.5-Omni 原生流式语义和 EgoSpeak 协议为任务基础。
|
||||
2. 从 ROMA 借鉴末层聚合、时机与内容解耦以及独立决策头。
|
||||
3. 先用 SoulX-Duplug、Action Sensitivity Learning 一类监督式状态或边界强化建立可靠增益。
|
||||
4. 若转向交互式闭环训练，再借鉴 DuplexPO 的动态关键窗口和 ASPIRin 的低维动作空间，避免对语义表示进行不受约束的 RL。
|
||||
当前建议把资源优先投入以下四条待验证路线，而不是继续放大 transition loss 或直接转向通用 RL：
|
||||
|
||||
1. 多速率近期/历史表征：保留块式 Omni 上下文，同时显式保留每 200 ms 更新的近期 AV 线索。
|
||||
2. 说话人感知对齐：从合并的 others 场景描述，走向按参与者组织的音视频证据。
|
||||
3. 分块稳定性：先测 chunk phase，再检验相位输入、分块偏移增强和严格因果教师蒸馏。
|
||||
4. 选择性适配：冻结骨干建立有效分支，再有限度调整连接层或上层 LoRA。
|
||||
|
||||
这些是本项目的研究假设，不是所引论文已在 EasyCom/Ego4D 接话预测上验证的结论。现有 open-stream 已经输出 5 Hz 状态；需要检验的是局部信息质量，而不是把已有 200 ms 更新误写为不存在。四状态表达 overlap，但不区分多个他人的身份。
|
||||
|
||||
## 维护约定
|
||||
|
||||
- 新增论文时，创建独立 Markdown 文件，并在本索引登记阅读深度。
|
||||
- 若后来深入阅读过一篇摘要级论文，更新其阅读深度、方法细节和实验结论。
|
||||
- 实验设计引用具体数值前，应再次核对论文正文、版本和官方实现。
|
||||
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
"""Validated, causal local-feature sidecars for the pooled B1 baseline.
|
||||
|
||||
No model/decoder imports: these utilities are also used by CPU head training.
|
||||
Sidecars store encoder-native vectors, never a randomly initialized projection.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
LOCAL_FORMAT = "qwen-local-pooled-v1"
|
||||
MODALITIES = {"visual": ("visual",), "audio": ("audio",), "av": ("visual", "audio")}
|
||||
|
||||
|
||||
def reference_alignment(payload: dict[str, Any]) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Require the same full, zero-origin 5 Hz timeline used by head training."""
|
||||
states = payload.get("states")
|
||||
if not isinstance(states, torch.Tensor) or states.ndim not in (2, 3) or not len(states):
|
||||
raise ValueError("Reference must contain nonempty [T,D] or [T,L,D] states")
|
||||
if float(payload.get("start", 0)) != 0:
|
||||
raise ValueError("Local features require a zero-origin reference (start=0)")
|
||||
indices = torch.as_tensor(payload.get("label_indices", []))
|
||||
if indices.dtype != torch.long or not torch.equal(indices, torch.arange(len(states))):
|
||||
raise ValueError("Reference label_indices must be dense int64 [0,...,T-1]")
|
||||
times = torch.as_tensor(payload.get("times", []), dtype=torch.float64)
|
||||
expected = (torch.arange(len(states), dtype=torch.float64) + 1) / 5
|
||||
# Existing feature files may have serialized times as float32.
|
||||
if times.shape != expected.shape or not torch.allclose(times, expected, atol=5e-4, rtol=0):
|
||||
raise ValueError("Reference times must be [0.2,0.4,...] seconds at 5 Hz")
|
||||
return times, indices
|
||||
|
||||
|
||||
def load_local_features(
|
||||
path: Path,
|
||||
reference: dict[str, Any],
|
||||
*,
|
||||
sample_id: str,
|
||||
dataset: str,
|
||||
modalities: str,
|
||||
expected_contract: dict[str, Any] | None = None,
|
||||
) -> tuple[torch.Tensor, dict[str, Any]]:
|
||||
"""Pack each requested modality as [raw vector, validity bit]. Fail closed."""
|
||||
if modalities not in MODALITIES:
|
||||
raise ValueError(f"Unknown local modalities: {modalities}")
|
||||
times, indices = reference_alignment(reference)
|
||||
payload = torch.load(path, map_location="cpu", weights_only=False)
|
||||
if not isinstance(payload, dict) or payload.get("format") != LOCAL_FORMAT:
|
||||
raise ValueError(f"Invalid local feature format: {path}")
|
||||
if payload.get("sample_id") != sample_id or payload.get("dataset") != dataset:
|
||||
raise ValueError(f"Local sample/dataset mismatch: {path}")
|
||||
if payload.get("partial", False):
|
||||
raise ValueError(f"Partial smoke-test features cannot be used for training: {path}")
|
||||
local_indices = torch.as_tensor(payload.get("label_indices", []))
|
||||
local_times = torch.as_tensor(payload.get("times", []), dtype=torch.float64)
|
||||
if local_indices.dtype != torch.long or not torch.equal(indices, local_indices):
|
||||
raise ValueError(f"Local label_indices mismatch: {path}")
|
||||
if local_times.shape != times.shape or not torch.allclose(local_times, times, atol=1e-7, rtol=0):
|
||||
raise ValueError(f"Local times mismatch: {path}")
|
||||
config = payload.get("config", {})
|
||||
if config.get("causal_policy") != "half-open-trailing-window" or config.get("uses_speech_labels") is not False:
|
||||
raise ValueError(f"Local features lack the causal/no-speech-label contract: {path}")
|
||||
contract = {"format": LOCAL_FORMAT, "modalities": modalities, "config": config}
|
||||
if expected_contract is not None and contract != expected_contract:
|
||||
raise ValueError(f"Local encoder/pooling contract differs across sessions/checkpoint: {path}")
|
||||
chunks = []
|
||||
for name in MODALITIES[modalities]:
|
||||
values = payload.get(name)
|
||||
valid = payload.get(f"{name}_valid")
|
||||
dimension = config.get(f"{name}_dim")
|
||||
if not isinstance(dimension, int) or dimension <= 0:
|
||||
raise ValueError(f"Missing {name}_dim: {path}")
|
||||
if not isinstance(values, torch.Tensor) or values.shape != (len(times), dimension):
|
||||
raise ValueError(f"Invalid {name} shape: {path}")
|
||||
if not values.is_floating_point() or not torch.isfinite(values).all():
|
||||
raise ValueError(f"Non-finite/non-floating {name} features: {path}")
|
||||
if not isinstance(valid, torch.Tensor) or valid.dtype != torch.bool or valid.shape != (len(times),):
|
||||
raise ValueError(f"Invalid {name}_valid mask: {path}")
|
||||
if torch.count_nonzero(values[~valid]):
|
||||
raise ValueError(f"Invalid {name} rows must be stored as zeros: {path}")
|
||||
chunks.extend((values.float(), valid[:, None].float()))
|
||||
return torch.cat(chunks, dim=-1), contract
|
||||
|
||||
|
||||
def local_head_dimensions(contract: dict[str, Any] | None) -> dict[str, int]:
|
||||
if contract is None:
|
||||
return {"local_visual_dim": 0, "local_audio_dim": 0}
|
||||
selected = MODALITIES[contract["modalities"]]
|
||||
return {
|
||||
f"local_{name}_dim": int(contract["config"][f"{name}_dim"]) if name in selected else 0
|
||||
for name in ("visual", "audio")
|
||||
}
|
||||
|
||||
|
||||
def causal_frame_ticks(end: float, fps: float) -> range:
|
||||
"""Global fixed lattice [0,end), excluding the frame exactly at end."""
|
||||
if end <= 0 or fps <= 0:
|
||||
raise ValueError("end and fps must be positive")
|
||||
return range(max(0, math.ceil(end * fps - 1e-8)))
|
||||
|
||||
|
||||
def pooled_recent_visual(
|
||||
frame_times: torch.Tensor,
|
||||
frame_vectors: torch.Tensor,
|
||||
frame_valid: torch.Tensor,
|
||||
end: float,
|
||||
window_seconds: float,
|
||||
) -> tuple[torch.Tensor, bool]:
|
||||
"""Equal frame weighting after equal person weighting within each frame."""
|
||||
keep = ((frame_times >= max(0., end - window_seconds) - 1e-8)
|
||||
& (frame_times < end - 1e-8) & frame_valid)
|
||||
if not bool(keep.any()):
|
||||
return frame_vectors.new_zeros(frame_vectors.shape[-1]), False
|
||||
return frame_vectors[keep].float().mean(0), True
|
||||
|
||||
|
||||
def easycom_boxes(rows: list[dict[str, Any]], kind: str) -> dict[int, list[dict[str, Any]]]:
|
||||
"""Map EasyCom face (1-based) / head (0-based) records to decoded indices."""
|
||||
if kind not in ("face", "head"):
|
||||
raise ValueError(f"Unknown box kind: {kind}")
|
||||
result = {}
|
||||
for row in rows:
|
||||
frame = int(row["Frame_Number"]) - (1 if kind == "face" else 0)
|
||||
people = row.get("Participants", [])
|
||||
if isinstance(people, dict):
|
||||
people = [people]
|
||||
if frame < 0 or frame in result or not isinstance(people, list):
|
||||
raise ValueError("Invalid/duplicate EasyCom box frame")
|
||||
result[frame] = [
|
||||
{"id": str(p.get("Participant_ID", -1)),
|
||||
"xyxy": [float(p[k]) for k in ("x1", "y1", "x2", "y2")]}
|
||||
for p in people
|
||||
]
|
||||
return result
|
||||
|
||||
|
||||
def crop_box(
|
||||
xyxy: list[float], width: int, height: int, *,
|
||||
offset_x: int = 0, offset_y: int = 0, expansion: float = 1.,
|
||||
) -> tuple[int, int, int, int] | None:
|
||||
"""Transform to camera coordinates, expand, clip; reject degenerate boxes."""
|
||||
if len(xyxy) != 4 or not all(math.isfinite(v) for v in xyxy):
|
||||
return None
|
||||
x1, y1, x2, y2 = xyxy
|
||||
if x2 <= x1 or y2 <= y1 or expansion <= 0:
|
||||
return None
|
||||
cx, cy = (x1 + x2) / 2 - offset_x, (y1 + y2) / 2 - offset_y
|
||||
half_w, half_h = (x2 - x1) * expansion / 2, (y2 - y1) * expansion / 2
|
||||
box = (max(0, math.floor(cx - half_w)), max(0, math.floor(cy - half_h)),
|
||||
min(width, math.ceil(cx + half_w)), min(height, math.ceil(cy + half_h)))
|
||||
w, h = box[2] - box[0], box[3] - box[1]
|
||||
# Qwen's smart_resize rejects aspect ratios above 200 (e.g. a sliver of
|
||||
# an offscreen box). Such a crop is not a usable person observation.
|
||||
return box if min(w, h) >= 2 and max(w, h) / min(w, h) <= 200 else None
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env bash
|
||||
# Foreground only. Usage: bash run_easycom_local_baselines.sh [all|baseline|audio|visual|av] [seed ...]
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
FEATURE_ROOT=${FEATURE_ROOT:-/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack}
|
||||
TARGET_ROOT=${TARGET_ROOT:-/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame}
|
||||
LOCAL_ROOT=${LOCAL_ROOT:-/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s}
|
||||
CHECKPOINT_ROOT=${CHECKPOINT_ROOT:-checkpoints/easycom_local_b1}
|
||||
RESULT_ROOT=${RESULT_ROOT:-results/easycom_local_b1}
|
||||
LOCAL_PROJECTION_DIM=${LOCAL_PROJECTION_DIM:-256}
|
||||
export OMP_NUM_THREADS=${OMP_NUM_THREADS:-4}
|
||||
export MKL_NUM_THREADS=${MKL_NUM_THREADS:-4}
|
||||
export OPENBLAS_NUM_THREADS=${OPENBLAS_NUM_THREADS:-4}
|
||||
|
||||
variant=${1:-all}
|
||||
if (( $# )); then shift; fi
|
||||
if (( $# )); then seeds=("$@"); else seeds=(0 1 2); fi
|
||||
case "$variant" in
|
||||
all) variants=(baseline audio visual av) ;;
|
||||
baseline|audio|visual|av) variants=("$variant") ;;
|
||||
*) echo "Expected all, baseline, audio, visual, or av" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
for variant in "${variants[@]}"; do
|
||||
train_local=()
|
||||
infer_local=()
|
||||
if [[ "$variant" != baseline ]]; then
|
||||
train_local=(--local-feature-root "$LOCAL_ROOT" --local-modalities "$variant"
|
||||
--local-projection-dim "$LOCAL_PROJECTION_DIM")
|
||||
infer_local=(--local-feature-root "$LOCAL_ROOT")
|
||||
fi
|
||||
for seed in "${seeds[@]}"; do
|
||||
checkpoint="$CHECKPOINT_ROOT/${variant}_seed${seed}.pth"
|
||||
result="$RESULT_ROOT/${variant}_seed${seed}"
|
||||
if [[ -e "$checkpoint" || -e "$result/metrics.json" ]]; then
|
||||
echo "Refusing to overwrite $checkpoint or $result; select new output roots." >&2
|
||||
exit 1
|
||||
fi
|
||||
python train_turn_head.py \
|
||||
--feature-root "$FEATURE_ROOT" --target-root "$TARGET_ROOT" \
|
||||
--output "$checkpoint" --sessions 4 5 6 7 8 9 10 11 12 \
|
||||
--epochs 4 --batch-size 512 --learning-rate 7e-5 --weight-decay 5e-5 \
|
||||
--warmup-epochs 1.6 --warmup-factor 0.3 --dropout 0.2 \
|
||||
--loss lstr-mce --transition-objective none --layer-fusion concat-mlp \
|
||||
--expected-mode open-stream-replay --seed "$seed" --device cuda:0 \
|
||||
"${train_local[@]}"
|
||||
python infer_turn_head.py \
|
||||
--feature-root "$FEATURE_ROOT" --target-root "$TARGET_ROOT" \
|
||||
--checkpoint "$checkpoint" --output-dir "$result" --sessions 3 2 1 \
|
||||
--expected-mode open-stream-replay --device cuda:0 "${infer_local[@]}"
|
||||
done
|
||||
done
|
||||
@@ -2351,6 +2351,11 @@ def _load_head(
|
||||
payload = torch.load(checkpoint, map_location="cpu", weights_only=True)
|
||||
if isinstance(payload, dict) and "head" in payload:
|
||||
config = checkpoint_head_config(payload)
|
||||
if config["local_visual_dim"] or config["local_audio_dim"]:
|
||||
raise ValueError(
|
||||
"Local-fusion checkpoints require local sidecars; use infer_turn_head.py "
|
||||
"with --local-feature-root, not --head-checkpoint during Omni extraction"
|
||||
)
|
||||
loss_name = checkpoint_loss_name(payload)
|
||||
if int(config["hidden_size"]) != hidden_size:
|
||||
raise ValueError(
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Print B0/B1 activity results in percent, with one explicit AP protocol."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import statistics
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path("results/easycom_local_b1"))
|
||||
parser.add_argument("--protocol", choices=("causal_valid_only", "egospeak_full_timeline"), default="causal_valid_only")
|
||||
args = parser.parse_args()
|
||||
files = sorted(args.root.glob("*/metrics.json"))
|
||||
if not files:
|
||||
raise SystemExit(f"No */metrics.json in {args.root}")
|
||||
print(f"Protocol: {args.protocol}; all AP/mAP numbers are percentages")
|
||||
print(f"{'run':<23} {'0.2s mAP':>11} {'BG AP':>11} {'wearer AP':>11} {'others AP':>11} {'mean mAP':>11}")
|
||||
groups = {}
|
||||
for path in files:
|
||||
metrics = json.loads(path.read_text())
|
||||
score = metrics[args.protocol]
|
||||
first = score["horizons"][0]
|
||||
ap = first["class_ap"]
|
||||
values = [100 * value for value in (first["map"], ap["background"],
|
||||
ap["camera_wearer_speaking"], ap["other_people_speaking"], score["mean_map"])]
|
||||
name = path.parent.name
|
||||
print(f"{name:<23}" + "".join(f" {v:11.5f}" for v in values))
|
||||
group = re.sub(r"_seed\d+$", "", name)
|
||||
groups.setdefault(group, []).append(values)
|
||||
print("\nMean ± sample standard deviation (AP/mAP percentage points)")
|
||||
for group, rows in sorted(groups.items()):
|
||||
stats = []
|
||||
for column in zip(*rows):
|
||||
spread = f"{statistics.stdev(column):.5f}" if len(column) > 1 else "n/a"
|
||||
stats.append(f"{statistics.mean(column):.5f} ± {spread}")
|
||||
print(f"{group:<23}" + " | ".join(stats))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,326 @@
|
||||
"""CPU tests for B1 alignment, causal pooling, fusion, and train/infer round trips."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
from easycom_head import TurnPredictionHead, checkpoint_head_config
|
||||
from extract_turn_local_features import (
|
||||
FrozenQwenLocalEncoder, ego4d_clip_boxes, extract_local,
|
||||
validate_source_timeline,
|
||||
model_config_digest,
|
||||
)
|
||||
from local_turn_features import (
|
||||
LOCAL_FORMAT, causal_frame_ticks, crop_box, easycom_boxes, load_local_features,
|
||||
local_head_dimensions, pooled_recent_visual, reference_alignment,
|
||||
)
|
||||
from train_turn_head import _load_examples
|
||||
|
||||
|
||||
def fixture(n=20, name="Session_4", dataset="easycom"):
|
||||
times = (torch.arange(n, dtype=torch.float64) + 1) / 5
|
||||
reference = {"states": torch.randn(n, 4, 8), "label_indices": torch.arange(n),
|
||||
"times": times, "mode": "open-stream-replay", "start": 0}
|
||||
side = {"format": LOCAL_FORMAT, "dataset": dataset, "sample_id": name,
|
||||
"times": times.clone(), "label_indices": torch.arange(n), "partial": False,
|
||||
"visual": torch.randn(n, 6), "audio": torch.randn(n, 5),
|
||||
"visual_valid": torch.ones(n, dtype=torch.bool),
|
||||
"audio_valid": torch.ones(n, dtype=torch.bool),
|
||||
"config": {"visual_dim": 6, "audio_dim": 5, "encoder_id": "synthetic-test",
|
||||
"causal_policy": "half-open-trailing-window", "uses_speech_labels": False}}
|
||||
side["visual_valid"][0] = False
|
||||
side["visual"][0] = 0
|
||||
labels = np.tile(np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 1]], dtype=np.float32), (n // 4 + 1, 1))[:n]
|
||||
return reference, side, labels
|
||||
|
||||
|
||||
class LocalTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
torch.set_num_threads(1)
|
||||
|
||||
def test_half_open_grid_and_pooling(self):
|
||||
self.assertEqual(list(causal_frame_ticks(.2, 2)), [0])
|
||||
self.assertEqual(list(causal_frame_ticks(1., 2)), [0, 1])
|
||||
ts = torch.tensor([0., .5, 1., 1.5], dtype=torch.float64)
|
||||
vectors = torch.tensor([[2.], [4.], [90.], [100.]])
|
||||
valid = torch.ones(4, dtype=torch.bool)
|
||||
value, present = pooled_recent_visual(ts, vectors, valid, 1., 1.)
|
||||
self.assertTrue(present)
|
||||
self.assertEqual(value.item(), 3.) # no frame at t=1
|
||||
value, _ = pooled_recent_visual(ts, vectors, valid, 1.2, 1.)
|
||||
self.assertEqual(value.item(), 47.) # .5 and 1.0, not 0.0
|
||||
value, present = pooled_recent_visual(ts, vectors, valid * False, 1., 1.)
|
||||
self.assertFalse(present)
|
||||
self.assertEqual(value.item(), 0.)
|
||||
|
||||
def test_box_formats_and_crop(self):
|
||||
person = {"Participant_ID": -1, "x1": 2, "y1": 3, "x2": 7, "y2": 9}
|
||||
faces = easycom_boxes([{"Frame_Number": 1, "Participants": [person]}], "face")
|
||||
heads = easycom_boxes([{"Frame_Number": 0, "Participants": person}], "head")
|
||||
self.assertEqual(faces, heads)
|
||||
self.assertEqual(crop_box([202., 3., 207., 9.], 1920, 1080, offset_x=200), (2, 3, 7, 9))
|
||||
self.assertIsNone(crop_box([4., 4., 3., 5.], 10, 10))
|
||||
self.assertIsNone(crop_box([float("nan"), 4., 9., 5.], 10, 10))
|
||||
|
||||
def test_ego4d_clip_frames(self):
|
||||
boxes = [{"frame": 3, "clip_frame": None, "x": 1, "y": 2, "width": 3, "height": 4}]
|
||||
person = {"person_id": "x", "camera_wearer": False, "tracking_paths": [{"track": boxes}, {"track": boxes}]}
|
||||
wearer = {**person, "camera_wearer": True}
|
||||
actual = ego4d_clip_boxes({"persons": [wearer, person]})
|
||||
self.assertEqual(actual, {3: [{"id": "x", "xyxy": [1., 2., 4., 6.]}]})
|
||||
|
||||
def test_sidecar_alignment_contract_and_modalities(self):
|
||||
reference, side, _ = fixture()
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / "Session_4.pt"
|
||||
for mode, width in (("visual", 7), ("audio", 6), ("av", 13)):
|
||||
torch.save(side, path)
|
||||
features, contract = load_local_features(path, reference, sample_id="Session_4", dataset="easycom", modalities=mode)
|
||||
self.assertEqual(features.shape, (20, width))
|
||||
dims = local_head_dimensions(contract)
|
||||
self.assertEqual(dims["local_visual_dim"], 0 if mode == "audio" else 6)
|
||||
for field, value in (("times", side["times"] + .2), ("label_indices", torch.arange(20) + 1),
|
||||
("sample_id", "Session_5"), ("partial", True),
|
||||
("visual", torch.full((20, 6), float("nan"))),
|
||||
("visual_valid", torch.ones(20))):
|
||||
broken = {**side, field: value}
|
||||
torch.save(broken, path)
|
||||
with self.subTest(field=field), self.assertRaises(ValueError):
|
||||
load_local_features(path, reference, sample_id="Session_4", dataset="easycom", modalities="av")
|
||||
torch.save(side, path)
|
||||
wrong_contract = copy.deepcopy(contract)
|
||||
wrong_contract["config"]["encoder_id"] = "other-weights"
|
||||
with self.assertRaisesRegex(ValueError, "contract"):
|
||||
load_local_features(path, reference, sample_id="Session_4", dataset="easycom", modalities="av", expected_contract=wrong_contract)
|
||||
with self.assertRaisesRegex(ValueError, "times"):
|
||||
reference_alignment({**reference, "times": reference["times"] + .2})
|
||||
|
||||
def test_head_shapes_gradients_missing_and_roundtrip(self):
|
||||
states = torch.randn(3, 4, 8)
|
||||
for v, a in ((0, 0), (6, 0), (0, 5), (6, 5)):
|
||||
for projection in (0, 3):
|
||||
head = TurnPredictionHead(hidden_size=8, num_input_layers=4, layer_fusion="concat-mlp",
|
||||
local_visual_dim=v, local_audio_dim=a, local_projection_dim=projection,
|
||||
transition_auxiliary=True, dropout=0)
|
||||
width = sum(d + 1 for d in (v, a) if d)
|
||||
local = torch.randn(3, width)
|
||||
offset = 0
|
||||
for d in (v, a):
|
||||
if d:
|
||||
local[:, offset + d] = torch.tensor([0., 1., 1.])
|
||||
offset += d + 1
|
||||
logits, transition = head.forward_with_transition(states, local)
|
||||
self.assertEqual(logits.shape, (3, 10, 3))
|
||||
self.assertEqual(transition.shape, (3, 10))
|
||||
(logits.square().mean() + transition.square().mean()).backward()
|
||||
self.assertIsNotNone(head.net[1].weight.grad)
|
||||
if width:
|
||||
# Changing invalid raw features cannot change the readout.
|
||||
changed = local.clone()
|
||||
offset = 0
|
||||
for d in (v, a):
|
||||
if d:
|
||||
changed[0, offset:offset + d] = 1000
|
||||
offset += d + 1
|
||||
self.assertTrue(torch.equal(head(states, local)[0], head(states, changed)[0]))
|
||||
with self.assertRaisesRegex(ValueError, "requires aligned"):
|
||||
head(states)
|
||||
config = {"hidden_size": 8, "num_input_layers": 4, "layer_fusion": "concat-mlp",
|
||||
"local_visual_dim": v, "local_audio_dim": a, "local_projection_dim": projection,
|
||||
"transition_auxiliary": True, "dropout": 0}
|
||||
restored = TurnPredictionHead(**checkpoint_head_config({"head_config": config}))
|
||||
restored.load_state_dict(head.state_dict())
|
||||
self.assertTrue(torch.equal(restored(states, local), head(states, local)))
|
||||
|
||||
def test_legacy_head_parameter_keys_and_values(self):
|
||||
torch.manual_seed(42)
|
||||
old = torch.nn.Sequential(torch.nn.LayerNorm(8), torch.nn.Linear(8, 4), torch.nn.GELU(),
|
||||
torch.nn.Dropout(.1), torch.nn.Linear(4, 30))
|
||||
torch.manual_seed(42)
|
||||
head = TurnPredictionHead(hidden_size=8)
|
||||
self.assertEqual(set(head.state_dict()), {f"net.{key}" for key in old.state_dict()})
|
||||
for key, value in old.state_dict().items():
|
||||
self.assertTrue(torch.equal(value, head.state_dict()[f"net.{key}"]))
|
||||
|
||||
def test_sparse_selection_keeps_local_rows(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
for folder in ("features", "local", "targets"):
|
||||
(root / folder).mkdir()
|
||||
reference, side, targets = fixture()
|
||||
torch.save(reference, root / "features/Session_4.pt")
|
||||
torch.save(side, root / "local/Session_4.pt")
|
||||
np.save(root / "targets/Session_4.npy", targets)
|
||||
data = _load_examples(root / "features", root / "targets", ["Session_4"], "open-stream-replay",
|
||||
"easycom", {"Session_4": {1, 4}}, root / "local", "audio")
|
||||
self.assertTrue(torch.equal(data[-1][:, :5], side["audio"][[1, 4]]))
|
||||
self.assertTrue(torch.equal(data[0], reference["states"][[1, 4]]))
|
||||
self.assertTrue(torch.equal(data[1][:, 0], torch.from_numpy(targets[[2, 5]])))
|
||||
|
||||
def test_source_inventory_rejects_different_clip_order(self):
|
||||
source = {"path": "/old/00-00-000.mp4", "source_frames": 1200, "dense_start": 0., "duration": 60.}
|
||||
media = SimpleNamespace(clips=[object()], metadata=lambda: {"source_clips": [source]})
|
||||
validate_source_timeline({"direct_media": {"source_clips": [source]}}, media)
|
||||
bad = {**source, "path": "/old/01-00-000.mp4"}
|
||||
with self.assertRaisesRegex(ValueError, "timeline differs"):
|
||||
validate_source_timeline({"direct_media": {"source_clips": [bad]}}, media)
|
||||
|
||||
def test_config_digest_ignores_only_checkpoint_location(self):
|
||||
a = {"_name_or_path": "/old/model", "audio_config": {"_name_or_path": "/old/model", "output_dim": 8}}
|
||||
b = {"_name_or_path": "/new/model", "audio_config": {"_name_or_path": "/new/model", "output_dim": 8}}
|
||||
self.assertEqual(model_config_digest(a), model_config_digest(b))
|
||||
b["audio_config"]["output_dim"] = 16
|
||||
self.assertNotEqual(model_config_digest(a), model_config_digest(b))
|
||||
|
||||
def test_encoder_image_pool_splits_people_before_mean(self):
|
||||
encoder = FrozenQwenLocalEncoder.__new__(FrozenQwenLocalEncoder)
|
||||
encoder.visual_dim, encoder.device, encoder.image_batch_size, encoder.max_pixels = 2, torch.device("cpu"), 8, 50176
|
||||
encoder.processor = SimpleNamespace(image_processor=lambda **kwargs: {
|
||||
"pixel_values": torch.ones(12, 2), "image_grid_thw": torch.tensor([[1, 2, 2], [1, 2, 4]])})
|
||||
encoder.model = SimpleNamespace(visual=SimpleNamespace(dtype=torch.float32),
|
||||
config=SimpleNamespace(vision_config=SimpleNamespace(spatial_merge_size=2)),
|
||||
get_image_features=lambda *_: torch.tensor([[1., 1.], [3., 3.], [5., 5.]]))
|
||||
actual = encoder.images([object(), object()])
|
||||
self.assertTrue(torch.equal(actual, torch.tensor([[1., 1.], [4., 4.]])))
|
||||
|
||||
def test_extraction_never_encodes_future_audio_and_saves_ragged_people(self):
|
||||
class Media:
|
||||
clips = [SimpleNamespace(crop=None, dense_start=0.)]
|
||||
def _source_frame_for_time(self, t):
|
||||
return 0, round(t * 20), t
|
||||
def frame_at_or_before(self, t):
|
||||
return Image.new("RGB", (16, 16))
|
||||
def audio_slice(self, start, end):
|
||||
return np.arange(round(start * 16000), round(end * 16000), dtype=np.float32)
|
||||
class Encoder:
|
||||
visual_dim = audio_dim = 2
|
||||
def __init__(self):
|
||||
self.ends = []
|
||||
def images(self, crops):
|
||||
return torch.ones(len(crops), 2)
|
||||
def audio(self, waveform):
|
||||
self.ends.append(float(waveform[-1]))
|
||||
return torch.tensor([float(waveform[0]), float(waveform[-1])])
|
||||
encoder = Encoder()
|
||||
args = SimpleNamespace(video_fps=2., visual_window_seconds=1., audio_window_seconds=1.,
|
||||
box_coordinate_space="camera", crop_expansion=1., no_progress=True)
|
||||
annotations = [{0: [{"id": "1", "xyxy": [0., 0., 8., 8.]}]}]
|
||||
result = extract_local(Media(), annotations, encoder, torch.arange(1, 11, dtype=torch.float64) / 5, args)
|
||||
self.assertEqual(encoder.ends, [3200 * (i + 1) - 1 for i in range(10)])
|
||||
self.assertEqual(result["persons"]["frame_offsets"].tolist(), [0, 1, 1, 1, 1])
|
||||
self.assertTrue(result["visual_valid"][:5].all())
|
||||
self.assertFalse(result["visual_valid"][5:].any())
|
||||
self.assertTrue(result["audio_valid"].all())
|
||||
|
||||
|
||||
@unittest.skipUnless(importlib.util.find_spec("transformers"), "optional: install transformers==4.52.3")
|
||||
class TinyQwenEncoderTests(unittest.TestCase):
|
||||
def test_real_encoder_apis_without_downloading_weights(self):
|
||||
import transformers
|
||||
if transformers.__version__ != "4.52.3":
|
||||
self.skipTest("Pinned encoder API test requires transformers==4.52.3")
|
||||
from transformers import (
|
||||
Qwen2_5OmniThinkerConfig, Qwen2_5OmniThinkerForConditionalGeneration,
|
||||
Qwen2VLImageProcessor, WhisperFeatureExtractor,
|
||||
)
|
||||
torch.set_num_threads(1)
|
||||
config = Qwen2_5OmniThinkerConfig(
|
||||
audio_config=dict(d_model=16, encoder_layers=1, encoder_attention_heads=2,
|
||||
encoder_ffn_dim=32, output_dim=8),
|
||||
vision_config=dict(depth=1, hidden_size=16, intermediate_size=32, num_heads=2,
|
||||
out_hidden_size=8, fullatt_block_indexes=[0]),
|
||||
text_config=dict(vocab_size=32, hidden_size=8, intermediate_size=16,
|
||||
num_hidden_layers=1, num_attention_heads=2, num_key_value_heads=2),
|
||||
)
|
||||
encoder = FrozenQwenLocalEncoder.__new__(FrozenQwenLocalEncoder)
|
||||
encoder.model = Qwen2_5OmniThinkerForConditionalGeneration(config).eval().requires_grad_(False)
|
||||
encoder.model.model = None
|
||||
encoder.model.lm_head = None
|
||||
encoder.visual_dim = encoder.audio_dim = 8
|
||||
encoder.device, encoder.max_pixels, encoder.image_batch_size = torch.device("cpu"), 224 * 224, 8
|
||||
encoder.processor = SimpleNamespace(image_processor=Qwen2VLImageProcessor(),
|
||||
feature_extractor=WhisperFeatureExtractor(feature_size=128))
|
||||
crops = [Image.new("RGB", (56, 84), "red"), Image.new("RGB", (112, 56), "blue")]
|
||||
batch = encoder.images(crops)
|
||||
single = encoder.images(crops[:1])
|
||||
self.assertEqual(batch.shape, (2, 8))
|
||||
self.assertTrue(torch.allclose(batch[0], single[0], atol=1e-5))
|
||||
for n in (3200, 6400, 16000):
|
||||
audio = encoder.audio(np.sin(np.arange(n, dtype=np.float32) / 10))
|
||||
self.assertEqual(audio.shape, (8,))
|
||||
self.assertTrue(torch.isfinite(audio).all())
|
||||
self.assertFalse(audio.requires_grad)
|
||||
|
||||
|
||||
class RoundTripTests(unittest.TestCase):
|
||||
def test_train_infer_easycom_and_ego4d(self):
|
||||
repo = Path(__file__).resolve().parent
|
||||
env = dict(os.environ, OMP_NUM_THREADS="1", MKL_NUM_THREADS="1", OPENBLAS_NUM_THREADS="1")
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
for folder in ("features", "local", "targets"):
|
||||
(root / folder).mkdir()
|
||||
train_uid = "00000000-0000-0000-0000-000000000004"
|
||||
test_uid = "00000000-0000-0000-0000-000000000001"
|
||||
manifest = root / "ego4d.tsv"
|
||||
manifest.write_text(f"split\tuid\ntrain\t{train_uid}\ntest\t{test_uid}\n")
|
||||
for name in ("Session_1", "Session_4", "Session_5", train_uid, test_uid):
|
||||
dataset = "easycom" if name.startswith("Session") else "ego4d"
|
||||
reference, side, targets = fixture(name=name, dataset=dataset)
|
||||
torch.save(reference, root / "features" / f"{name}.pt")
|
||||
torch.save(side, root / "local" / f"{name}.pt")
|
||||
np.save(root / "targets" / f"{name}.npy", targets)
|
||||
common = ["--feature-root", str(root / "features"), "--target-root", str(root / "targets"),
|
||||
"--device", "cpu", "--no-progress", "--expected-mode", "open-stream-replay"]
|
||||
for mode in ("baseline", "audio", "visual", "av", "ego4d"):
|
||||
checkpoint = root / f"{mode}.pth"
|
||||
train = [sys.executable, str(repo / "train_turn_head.py"), *common,
|
||||
"--epochs", "1", "--warmup-epochs", "0", "--batch-size", "4",
|
||||
"--layer-fusion", "concat-mlp", "--output", str(checkpoint)]
|
||||
infer = [sys.executable, str(repo / "infer_turn_head.py"), *common,
|
||||
"--checkpoint", str(checkpoint), "--output-dir", str(root / f"result_{mode}")]
|
||||
if mode == "ego4d":
|
||||
train += ["--dataset", "ego4d", "--manifest", str(manifest)]
|
||||
infer += ["--dataset", "ego4d", "--manifest", str(manifest)]
|
||||
else:
|
||||
train += ["--sessions", "4", "5"]
|
||||
infer += ["--sessions", "1"]
|
||||
if mode == "visual":
|
||||
train += ["--dev-sessions", "5", "--local-projection-dim", "0"]
|
||||
if mode == "av":
|
||||
train += ["--transition-objective", "r1-supervised"]
|
||||
if mode != "baseline":
|
||||
train += ["--local-feature-root", str(root / "local"), "--local-modalities", "av" if mode == "ego4d" else mode]
|
||||
infer += ["--local-feature-root", str(root / "local")]
|
||||
for command in (train, infer):
|
||||
completed = subprocess.run(command, cwd=repo, env=env, capture_output=True, text=True, timeout=60)
|
||||
self.assertEqual(completed.returncode, 0, f"{mode}: {completed.stdout}\n{completed.stderr}")
|
||||
metrics = json.loads((root / f"result_{mode}/metrics.json").read_text())
|
||||
self.assertEqual(metrics["checkpoint_epoch"], 1)
|
||||
self.assertEqual(bool(metrics["local_contract"]), mode != "baseline")
|
||||
if mode == "av":
|
||||
self.assertTrue(metrics["transition_auxiliary"])
|
||||
# The reporting command uses the same metric keys as inference.
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(repo / "summarize_local_baselines.py"), "--root", str(root)],
|
||||
cwd=repo, capture_output=True, text=True, timeout=20,
|
||||
)
|
||||
self.assertEqual(completed.returncode, 0, completed.stderr)
|
||||
self.assertIn("Mean ± sample standard deviation", completed.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+69
-7
@@ -53,6 +53,15 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument("--feature-root", type=Path, required=True)
|
||||
parser.add_argument("--target-root", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--local-feature-root", type=Path,
|
||||
help="Optional frozen local-feature sidecars; omit for the original baseline.",
|
||||
)
|
||||
parser.add_argument("--local-modalities", choices=("visual", "audio", "av"), default="av")
|
||||
parser.add_argument(
|
||||
"--local-projection-dim", type=int, default=256,
|
||||
help="Trainable projection per local modality; 0 retains encoder dimensions.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dataset",
|
||||
choices=DATASET_NAMES,
|
||||
@@ -154,6 +163,8 @@ def parse_args() -> argparse.Namespace:
|
||||
|
||||
|
||||
def _validate_args(args: argparse.Namespace) -> None:
|
||||
if args.local_projection_dim < 0:
|
||||
raise ValueError("--local-projection-dim must be non-negative")
|
||||
if args.epochs <= 0:
|
||||
raise ValueError("--epochs must be positive")
|
||||
if args.batch_size <= 0:
|
||||
@@ -210,6 +221,9 @@ def _load_examples(
|
||||
expected_mode: str | None,
|
||||
dataset: str,
|
||||
selected_indices: dict[str, set[int]] | None = None,
|
||||
local_feature_root: Path | None = None,
|
||||
local_modalities: str = "av",
|
||||
local_contract: dict[str, Any] | None = None,
|
||||
) -> tuple[
|
||||
torch.Tensor,
|
||||
torch.Tensor,
|
||||
@@ -217,10 +231,12 @@ def _load_examples(
|
||||
int,
|
||||
int,
|
||||
list[dict[str, Any]],
|
||||
torch.Tensor,
|
||||
]:
|
||||
all_states: list[torch.Tensor] = []
|
||||
all_targets: list[torch.Tensor] = []
|
||||
all_transition_targets: list[torch.Tensor] = []
|
||||
all_local: list[torch.Tensor] = []
|
||||
metadata: list[dict[str, Any]] = []
|
||||
hidden_size: int | None = None
|
||||
num_input_layers: int | None = None
|
||||
@@ -231,9 +247,18 @@ def _load_examples(
|
||||
name,
|
||||
expected_mode=expected_mode,
|
||||
dataset=dataset,
|
||||
local_feature_root=local_feature_root,
|
||||
local_modalities=local_modalities,
|
||||
local_contract=local_contract,
|
||||
)
|
||||
local_contract = data.feature_metadata.get("local_contract")
|
||||
states, targets, future_indices = data.future_examples(NUM_HORIZONS)
|
||||
current_indices = future_indices[:, 0] - 1
|
||||
local = (
|
||||
data.local_features[current_indices]
|
||||
if data.local_features is not None
|
||||
else torch.empty(len(states), 0)
|
||||
)
|
||||
activity_sequence = torch.cat(
|
||||
(data.targets[current_indices, None, :], targets),
|
||||
dim=1,
|
||||
@@ -251,6 +276,7 @@ def _load_examples(
|
||||
states = states[keep]
|
||||
targets = targets[keep]
|
||||
transition_targets = transition_targets[keep]
|
||||
local = local[keep]
|
||||
if not len(states):
|
||||
raise ValueError(f"Sparse anchor selection kept no examples for {name}")
|
||||
if hidden_size is None:
|
||||
@@ -270,6 +296,7 @@ def _load_examples(
|
||||
all_states.append(states)
|
||||
all_targets.append(targets)
|
||||
all_transition_targets.append(transition_targets)
|
||||
all_local.append(local)
|
||||
metadata.append(
|
||||
{
|
||||
"session": name,
|
||||
@@ -303,6 +330,7 @@ def _load_examples(
|
||||
hidden_size,
|
||||
num_input_layers,
|
||||
metadata,
|
||||
torch.cat(all_local, dim=0),
|
||||
)
|
||||
|
||||
|
||||
@@ -369,6 +397,7 @@ def _evaluate_valid_horizons(
|
||||
device: torch.device,
|
||||
batch_size: int,
|
||||
loss_name: str,
|
||||
local: torch.Tensor | None = None,
|
||||
) -> dict[str, Any]:
|
||||
head.eval()
|
||||
chunks: list[torch.Tensor] = []
|
||||
@@ -378,9 +407,11 @@ def _evaluate_valid_horizons(
|
||||
dtype=torch.float32,
|
||||
non_blocking=True,
|
||||
)
|
||||
chunks.append(
|
||||
marginal_probabilities_from_logits(head(x), loss_name).cpu()
|
||||
side = (
|
||||
local[start : start + batch_size].to(device=device, dtype=torch.float32)
|
||||
if local is not None else None
|
||||
)
|
||||
chunks.append(marginal_probabilities_from_logits(head(x, side), loss_name).cpu())
|
||||
probabilities = torch.cat(chunks, dim=0).numpy()
|
||||
target_array = targets.numpy()
|
||||
|
||||
@@ -415,7 +446,7 @@ def _checkpoint(
|
||||
history: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"format_version": 3,
|
||||
"format_version": 4,
|
||||
"head": head.state_dict(),
|
||||
"head_config": {
|
||||
"hidden_size": hidden_size,
|
||||
@@ -426,7 +457,11 @@ def _checkpoint(
|
||||
"layer_fusion": args.layer_fusion,
|
||||
"learnable_layer_mixture": args.learnable_layer_mixture,
|
||||
"transition_auxiliary": head.transition_auxiliary,
|
||||
"local_visual_dim": head.local_visual_dim,
|
||||
"local_audio_dim": head.local_audio_dim,
|
||||
"local_projection_dim": head.local_projection_dim,
|
||||
},
|
||||
"local_contract": feature_metadata[0].get("local_contract"),
|
||||
"alignment": {
|
||||
"state_label_offset": 0,
|
||||
"future_offsets": list(range(1, NUM_HORIZONS + 1)),
|
||||
@@ -467,6 +502,9 @@ def _checkpoint(
|
||||
"seed": args.seed,
|
||||
"expected_mode": args.expected_mode,
|
||||
"layer_fusion": args.layer_fusion,
|
||||
"local_feature_root": str(args.local_feature_root) if args.local_feature_root else None,
|
||||
"local_modalities": args.local_modalities if args.local_feature_root else None,
|
||||
"local_projection_dim": args.local_projection_dim,
|
||||
"learnable_layer_mixture": args.layer_fusion == "scalar-mixture",
|
||||
"supervision_policy": (
|
||||
"transition-sparse-manifest"
|
||||
@@ -557,6 +595,7 @@ def main() -> None:
|
||||
hidden_size,
|
||||
num_input_layers,
|
||||
feature_metadata,
|
||||
local_features,
|
||||
) = _load_examples(
|
||||
args.feature_root,
|
||||
args.target_root,
|
||||
@@ -564,7 +603,12 @@ def main() -> None:
|
||||
expected_mode,
|
||||
args.dataset,
|
||||
sparse_anchor_indices,
|
||||
args.local_feature_root,
|
||||
args.local_modalities,
|
||||
)
|
||||
local_contract = feature_metadata[0].get("local_contract")
|
||||
if not len(states):
|
||||
raise ValueError("No valid training examples")
|
||||
print(
|
||||
f"training examples: {len(states)}, feature_dim={hidden_size}, "
|
||||
f"input_layers={num_input_layers}"
|
||||
@@ -586,6 +630,7 @@ def main() -> None:
|
||||
)
|
||||
dev_states: torch.Tensor | None = None
|
||||
dev_targets: torch.Tensor | None = None
|
||||
dev_local: torch.Tensor | None = None
|
||||
if dev_sessions:
|
||||
(
|
||||
dev_states,
|
||||
@@ -594,6 +639,7 @@ def main() -> None:
|
||||
dev_hidden_size,
|
||||
dev_num_input_layers,
|
||||
_,
|
||||
dev_local,
|
||||
) = _load_examples(
|
||||
args.feature_root,
|
||||
args.target_root,
|
||||
@@ -601,6 +647,9 @@ def main() -> None:
|
||||
expected_mode,
|
||||
args.dataset,
|
||||
None,
|
||||
args.local_feature_root,
|
||||
args.local_modalities,
|
||||
local_contract,
|
||||
)
|
||||
if dev_hidden_size != hidden_size:
|
||||
raise ValueError(
|
||||
@@ -614,7 +663,7 @@ def main() -> None:
|
||||
)
|
||||
print(f"development examples: {len(dev_states)}")
|
||||
|
||||
dataset = TensorDataset(states, targets, transition_targets)
|
||||
dataset = TensorDataset(states, targets, transition_targets, local_features)
|
||||
generator = torch.Generator().manual_seed(args.seed)
|
||||
loader = DataLoader(
|
||||
dataset,
|
||||
@@ -626,6 +675,8 @@ def main() -> None:
|
||||
generator=generator,
|
||||
)
|
||||
num_output_classes = output_classes_for_loss(args.loss)
|
||||
from local_turn_features import local_head_dimensions
|
||||
|
||||
head = TurnPredictionHead(
|
||||
hidden_size=hidden_size,
|
||||
num_horizons=NUM_HORIZONS,
|
||||
@@ -635,7 +686,14 @@ def main() -> None:
|
||||
learnable_layer_mixture=args.learnable_layer_mixture,
|
||||
layer_fusion=args.layer_fusion,
|
||||
transition_auxiliary=args.transition_objective != "none",
|
||||
local_projection_dim=args.local_projection_dim,
|
||||
**local_head_dimensions(local_contract),
|
||||
).to(device)
|
||||
if local_contract is not None:
|
||||
print(
|
||||
f"local fusion: {args.local_modalities}; raw V={head.local_visual_dim}, "
|
||||
f"A={head.local_audio_dim}; projection={args.local_projection_dim or 'identity'}"
|
||||
)
|
||||
trainable_parameters = sum(
|
||||
parameter.numel() for parameter in head.parameters() if parameter.requires_grad
|
||||
)
|
||||
@@ -709,7 +767,10 @@ def main() -> None:
|
||||
unit="batch",
|
||||
disable=args.no_progress,
|
||||
)
|
||||
for state_batch, target_batch, transition_target_batch in progress:
|
||||
for state_batch, target_batch, transition_target_batch, local_batch in progress:
|
||||
local_batch = local_batch.to(
|
||||
device=device, dtype=torch.float32, non_blocking=True,
|
||||
)
|
||||
state_batch = state_batch.to(
|
||||
device=device,
|
||||
dtype=torch.float32,
|
||||
@@ -727,13 +788,13 @@ def main() -> None:
|
||||
)
|
||||
optimizer.zero_grad(set_to_none=True)
|
||||
if args.transition_objective == "none":
|
||||
logits = head(state_batch)
|
||||
logits = head(state_batch, local_batch)
|
||||
activity_loss = loss_function(logits, target_batch)
|
||||
loss = activity_loss
|
||||
transition_loss = activity_loss.detach().new_zeros(())
|
||||
expected_reward = activity_loss.detach().new_zeros(())
|
||||
else:
|
||||
logits, transition_logits = head.forward_with_transition(state_batch)
|
||||
logits, transition_logits = head.forward_with_transition(state_batch, local_batch)
|
||||
activity_loss = loss_function(logits, target_batch)
|
||||
components = transition_aware_objective(
|
||||
activity_loss,
|
||||
@@ -813,6 +874,7 @@ def main() -> None:
|
||||
device,
|
||||
args.batch_size,
|
||||
args.loss,
|
||||
dev_local,
|
||||
)
|
||||
dev_map = float(dev_metrics["mean_map"])
|
||||
record["dev_valid_mean_map"] = dev_map
|
||||
|
||||
Reference in New Issue
Block a user