diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7cf1785 --- /dev/null +++ b/.gitignore @@ -0,0 +1,19 @@ +.DS_Store +__pycache__/ +*.py[cod] +.pytest_cache/ +.coverage + +# Generated experiment assets; datasets/models are expected outside the repo. +/checkpoints/ +/features/ +/logs/ +/manifests/ +/outputs/ +/target_frame/ + +# Local editors and environments. +.idea/ +.vscode/ +.venv/ +venv/ diff --git a/AGENTS.md b/AGENTS.md index 2221f8b..32282bf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,2 +1,52 @@ -作为一名资深音视频，深度学习国际知名导师。 -对于我的想法，引导我以思辨的方式继续，指导我完成icassp2027论文实验和编写。 \ No newline at end of file +# AI collaboration rules + +你是一名资深音视频与深度学习研究导师。目标不是无条件接受想法，而是帮助用户形成 +可证伪、可复现、能支撑 ICASSP 2027 投稿的研究问题、实验和论文表述。 + +## 开始工作前的阅读顺序 + +1. 先读 [`PROJECT_CONTEXT.md`](PROJECT_CONTEXT.md)：代码入口、数据契约、当前状态。 +2. 涉及论文方向时读 + [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md)。 +3. 涉及 novelty/related work 时读 + [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) + 和 [`literature_notes/README.md`](literature_notes/README.md)。 +4. 只在需要具体命令或历史实现细节时读取长篇 [`README.md`](README.md) 的相关章节， + 不要默认把整份 README 当作当前实验优先级。 + +## 当前唯一优先级 + +先冻结 wearer floor-event 评测，再训练新模型。现有 +`evaluate_floor_state_changes.py` 的事件窗口仍需 competing-risk censoring：response 不能 +跨过下一次 other onset，yield 不能跨过下一次 other offset。在该问题修正前，事件结果 +只能称为 timing-derived proxy。 + +## 不可破坏的研究契约 + +- 严格因果：任何输入音频、视频帧、人物框或归一化统计不得来自预测截止时间之后。 +- 时间网格：目标和全局状态为 5 Hz；第 `t` 个状态对应截止到 `(t+1)/5` 秒的输入。 +- 输出：10 个未来 horizon，0.2--2.0 秒；类别顺序固定为 background、wearer、others。 +- EasyCom 官方划分：Session 4--12 训练，Session 1--3 测试。不能用测试集调阈值、窗口、 + loss 权重或 early stopping。 +- `others` 是聚合活动，不能据此声称知道具体说话人、addressee、语义 EOT 或 INT。 +- 冻结 Omni 特征加 head 是 frozen-backbone adaptation，不是任务级 end-to-end training。 +- 计划文档中的模块不得被写成“已实现”；以代码、测试和结果记录三者为准。 + +## 代码与目录约束 + +- 根目录 Python/shell 文件是远程 L20 命令正在使用的稳定 CLI。除非同时提供兼容 wrapper、 + 更新所有引用并运行测试，否则不要把它们移动到 `src/`、`scripts/` 或 `tests/`。 +- 数据集、模型、features、checkpoints 和 logs 都是外部/生成资产，不进入 Git。 +- 新实验结果放入 `results/__/`，至少包含 `README.md`、机器可读 + `summary.json` 或 `metrics.csv`，并记录 split、seed、checkpoint 和协议。 +- 新运行手册放 `docs/`；新论文摘要放 `literature_notes/`；自动审计输出放 `reports/`。 +- 使用相对仓库路径写文档，不固化某一台机器的 `/data` 或 `/data1` 路径为唯一配置。 + +## 验证要求 + +- 文档改动：运行 `git diff --check` 并确认相对链接存在。 +- evaluator/label 改动：运行对应 `test_*.py`，并人工打印至少一个事件时间轴。 +- feature contract 改动：同时测试 `easycom_head.py`、`local_turn_features.py` 的加载失败和 + 成功路径，不能静默 padding、clamp 或错位。 +- GPU 全量运行前：先做单 session/短片 smoke test，检查 shape、时间、valid mask 和无未来 + 泄漏，再给用户多卡命令。 diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md new file mode 100644 index 0000000..81d5d47 --- /dev/null +++ b/PROJECT_CONTEXT.md @@ -0,0 +1,248 @@ +# AI Project Context + +This is the short, authoritative repository map for humans and AI agents. +The long [`README.md`](README.md) is an operational history and command +reference; it is not the experiment scheduler. + +## One-sentence objective + +Determine whether strictly causal, speaker-grounded audiovisual evidence from +Qwen2.5-Omni can improve an invisible egocentric wearer's **floor-entry** and +**floor-yield** behavior in natural multiparty conversation. + +The paper is not simply “replace EgoSpeak's encoders with a newer backbone.” +The decisive test is whether correct person-aligned vision reduces false +wearer entries during other-to-other conversation at matched wearer-onset +recall, without hurting addressed responses or yield behavior. + +## Source-of-truth order + +| Need | Authoritative file | +|---|---| +| Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) | +| Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) | +| Code/data map and current blocker | This file | +| Runnable commands and implementation history | [`README.md`](README.md) and [`docs/README.md`](docs/README.md) | +| Archived measured results | [`results/README.md`](results/README.md) | +| Literature summaries | [`literature_notes/README.md`](literature_notes/README.md) | + +When files disagree, prefer the newest scoped document and verify behavior in +code/tests. A planned experiment is not an implemented feature or a measured +result. + +## Current state + +### Verified assets + +- Strictly causal 5 Hz Qwen2.5-Omni global state extraction on EasyCom. +- Prepared-media and direct-original-media extraction paths. +- Frozen head training/inference with EgoSpeak-compatible 10-horizon AP/mAP. +- EasyCom local Qwen visual crops and recent mixed-audio sidecars. +- Encoder-only A/V/AV and `G`, `G+A`, `G+V`, `G+A+V` baselines. +- Ego4D batch and 8-GPU local-feature launchers, with strict frame audits. +- Timing-derived wearer floor-state and event diagnostics. + +### Current blocker + +`evaluate_floor_state_changes.py` counts a wearer transition anywhere in a +fixed response window. Before this becomes the paper event protocol, add +competing-risk censoring: + +- an other-offset response window ends at the next other onset; +- an other-onset yield window ends at the next other offset. + +Until then, EOT/HOLD, response and yield results are proxy diagnostics, not +semantic event ground truth. + +### Next experiment gates + +1. **P0:** fix/freeze event attribution, splits and scenario definitions. +2. **P1:** probe aggregate activity and per-person active-speaker grounding; + compare correct, wrong-person and time-shifted visual inputs. +3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol. +4. **P3:** train an anonymous person-wise causal side branch with matched + temporal-capacity controls. +5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass. + +## Stable data contracts + +### Time and labels + +- Sampling rate: 5 Hz (`STEP_SECONDS = 0.2`). +- State row `j` uses media no later than `(j+1)/5` seconds. +- Future horizons: 10, corresponding to 0.2, 0.4, ..., 2.0 seconds. +- Class order: `background`, `camera_wearer_speaking`, + `other_people_speaking`. +- EasyCom targets are dense `[T, 3]`; predictions are + `[target_time, class, future_horizon]`. +- EasyCom split: train Session 4--12; test Session 1--3. + +### Global feature sidecar + +Produced by `streaming_turn_model.py` or +`streaming_turn_model_directly.py` as a `.pt` dictionary. Required training +fields include: + +- `states`: `[T, 2048]` or multi-layer `[T, L, 2048]`; +- `times`: `[0.2, 0.4, ...]`; +- `label_indices`: dense zero-origin indices into the target timeline; +- extraction `mode`, configuration and source metadata. + +Never silently accept non-zero origins, missing label indices or mismatched +time grids. + +### Local feature sidecar + +Produced by `extract_turn_local_features.py`, validated by +`local_turn_features.py`, and stored as `qwen-local-pooled-v1`. It contains +native 2048D visual/audio vectors, validity masks, times, label indices, +source metadata and the exact extraction config. A partial smoke sidecar must +not be reused as a full-session feature file. + +### Inference result + +`infer_turn_head.py` writes: + +```text +RESULT_DIR/ + metrics.json + pred/.npy # [T, 3, 10] + valid/.npy # [T, 10] + transition_pred/ # only when the auxiliary head exists +``` + +`evaluate_floor_state_changes.py` consumes `pred/` and `valid/`; it does not +run a GPU model. + +## Canonical pipelines + +### Direct global features (preferred semantic path) + +```text +original EasyCom/Ego4D media + -> streaming_turn_model_directly.py + -> global feature .pt + -> train_turn_head.py + -> checkpoint .pth + .json + -> infer_turn_head.py + -> pred/ + valid/ + metrics.json + -> evaluate_floor_state_changes.py +``` + +This path samples source video causally in Python and avoids requiring a +re-encoded fixed-FPS MP4. + +### Prepared-media compatibility path + +```text +prepare_easycom_session.py / prepare_ego4d_2fps.py + -> strict 2 FPS media + metadata + -> streaming_turn_model.py + -> the same training/inference pipeline +``` + +Keep it for reproducibility, cache-replay verification and comparisons with +already extracted features. + +### Local evidence path + +```text +raw media + person boxes + existing global reference .pt + -> extract_turn_local_features.py + (or extract_ego4d_local_multigpu.py supervisor) + -> local .pt sidecar + -> train_turn_head.py --local-feature-root ... + -> infer_turn_head.py --local-feature-root ... +``` + +The current local visual feature is pooled. It does not by itself prove +speaker identity, active-speaker alignment or addressee understanding. + +## Root-code map + +Root scripts remain flat because remote L20 commands call them directly. +Do not reorganize them physically without compatibility wrappers. + +### Shared libraries + +| File | Responsibility | +|---|---| +| `easycom_head.py` | Dataset constants, feature loading, heads, losses and AP metrics | +| `local_turn_features.py` | Local-sidecar contract, alignment, crops and pooling | +| `omni_cached_av.py` | Cached AV structures used by Omni extraction/LoRA paths | + +### Global feature extraction + +| File | Responsibility | +|---|---| +| `streaming_turn_model_directly.py` | Preferred direct raw-media causal extraction | +| `streaming_turn_model.py` | Prepared-media causal-prefix/open-stream extraction and replay checks | +| `qwen_omni_features.py` | Single-window feature inspection | +| `cache_omni_av_pieces.py` | Build reusable AV pieces for LoRA experiments | +| `extract_ego4d_batch.py` | Persistent-model Ego4D global batch extraction | +| `analyze_block_phase.py` | Streaming block-phase diagnostics | +| `profile_streaming_bottleneck_gpu7.py` | GPU extraction profiling | + +### Data preparation and audit + +| File | Responsibility | +|---|---| +| `easycom_manifest.py` | EasyCom clip inventory and dense-timeline validation | +| `prepare_easycom_session.py` | Optional strict-grid EasyCom media preparation | +| `prepare_ego4d_2fps.py` | Optional strict-grid Ego4D media preparation | +| `prepare_encoder_only_features.py` | Convert local sidecars into encoder-only references | +| `audit_egospeak_features.py` | Audit released EgoSpeak feature layout | +| `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies | +| `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication | +| `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates | + +### Local features + +| File | Responsibility | +|---|---| +| `extract_turn_local_features.py` | Qwen person-crop visual and recent-audio sidecars | +| `extract_ego4d_local_multigpu.py` | Foreground supervisor for disjoint multi-GPU workers | + +### Training, inference and evaluation + +| File | Responsibility | +|---|---| +| `train_turn_head.py` | Frozen-backbone head training, optional local fusion/auxiliary losses | +| `infer_turn_head.py` | Save EgoSpeak-compatible predictions and AP/mAP | +| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation | +| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files | +| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV | +| `eval_omni_turn_lora.py` | Thinker LoRA evaluation | + +### Launchers and tests + +- `run_*.sh` and `run_*.py` are reproducible launchers, not libraries. +- `smoke_test.py` is the model/environment smoke check. +- `test_*.py` are CPU-oriented regression tests colocated at root for current + import and command compatibility. + +## Terminology restrictions + +- **Global (`G`)**: a Thinker readout that may mix audio, video and language + context; do not describe it as audio-free. +- **Local audio (`A`)**: recent mixed audio encoded outside the Thinker. +- **Local visual (`V`)**: person-box visual features; current pooling discards + stable person structure. +- **Encoder-only**: Qwen modality encoder output without Thinker states. +- **Speaker grounding**: evidence that the correct visible person is linked to + the correct speech/time. AV improvement alone is insufficient. +- **Addressee grounding**: evidence about whom speech targets; aggregate + `others` labels cannot supervise it. +- **Floor entry/yield**: observed wearer OFF→ON / ON→OFF behavior under a + defined causal event window; not automatically semantic response or + interruption intent. + +## Environment boundaries + +- The open-stream replay implementation is pinned to + `transformers==4.52.3` because it uses version-specific internals. +- The local encoder path has been tested with 4.52.3 and 4.57.6; sidecars from + one experiment must share the same recorded contract/version. +- Remote paths differ (`/data/...` and `/data1/...`). Commands should expose + dataset/model roots as variables rather than hard-code one machine. +- Large models, datasets, checkpoints and extracted features live outside Git. diff --git a/README.md b/README.md index 3be10c7..b2d2807 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,25 @@ correct person-aligned vision should reduce false wearer entries while two other people are conversing, without reducing responses addressed to the wearer or degrading wearer yield decisions. +## Repository navigation + +Start with [`PROJECT_CONTEXT.md`](PROJECT_CONTEXT.md) for the concise code map, +data contracts, current blocker and canonical pipelines. This README is the +long operational guide and experiment history. + +| Location | Contents | +|---|---| +| [`AGENTS.md`](AGENTS.md) | Research and code constraints for AI collaborators | +| [`docs/`](docs/README.md) | Current paper plan, novelty audit and dataset runbooks | +| [`results/`](results/README.md) | Immutable measured-result records and interpretation rules | +| [`literature_notes/`](literature_notes/README.md) | Paper index and persistent summaries | +| [`reports/`](reports/README.md) | Generated dataset audits; not model results | + +Root Python and shell files remain intentionally stable because remote L20 +commands invoke them directly. They are categorized in `PROJECT_CONTEXT.md`; +moving them into `src/`, `scripts/` or `tests/` requires compatibility wrappers +and regression testing. + ## Current paper question and experiment plan (2026-08-30) The full staged plan is in diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..99f7fd8 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,42 @@ +# Documentation index + +Documents are grouped by authority rather than by date. “Planned” never means +implemented or validated. + +## Active paper definition + +| Document | Role | +|---|---| +| [`omni_speaker_floor_experiment_plan_2026-08-30.md`](omni_speaker_floor_experiment_plan_2026-08-30.md) | Current hypotheses, experiment gates, main table and stopping rules | +| [`multiparty_egocentric_floor_gap_audit_2026-08-30.md`](multiparty_egocentric_floor_gap_audit_2026-08-30.md) | Novelty boundary, closest work and restricted claims | + +These two files supersede earlier documents when discussing the paper's main +question. They do not supersede verified implementation details in code and +tests. + +## Current implementation runbooks + +| Document | Use when | +|---|---| +| [`local_feature_baseline.md`](local_feature_baseline.md) | Extracting/validating EasyCom pooled local Qwen visual and audio sidecars | +| [`easycom_encoder_only.md`](easycom_encoder_only.md) | Comparing EasyCom Qwen encoder-only A/V/AV representations | +| [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison | +| [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs | + +## Design history and implementation options + +| Document | Status | +|---|---| +| [`speaker_aware_thinker_plan_2026-08-28.md`](speaker_aware_thinker_plan_2026-08-28.md) | Historical design exploration; useful implementation detail, not current priority order | +| [`trainable_speaker_fusion_survey_2026-08-28.md`](trainable_speaker_fusion_survey_2026-08-28.md) | Detailed person-branch/adapter survey feeding the current plan | + +## Where new material belongs + +- Current research decisions: a dated file in `docs/`, then link it from this + index and `PROJECT_CONTEXT.md`. +- Dataset or command runbook: `docs/_.md`. +- Measured experiment: `results/__/` with its own + README and machine-readable summary. +- Paper summary: `literature_notes/NN_short_name.md` and update its index. +- Generated audit report: `reports/`; do not present it as a hand-written + source of truth. diff --git a/reports/README.md b/reports/README.md new file mode 100644 index 0000000..95b6f4f --- /dev/null +++ b/reports/README.md @@ -0,0 +1,14 @@ +# Generated audit reports + +This directory stores machine-generated dataset audits, not model results or +authoritative annotations. + +Current artifact: + +- [`egospeak_easycom_feature_audit.json`](egospeak_easycom_feature_audit.json): + released EgoSpeak EasyCom feature-layout audit. + +When adding a report, record the generating script, command, source dataset +version and timestamp in the payload or an adjacent README. Derived timing or +LLM-adjudicated labels must remain explicitly named proxies unless they have +been manually validated as ground truth. diff --git a/results/README.md b/results/README.md new file mode 100644 index 0000000..1e7c06d --- /dev/null +++ b/results/README.md @@ -0,0 +1,41 @@ +# Experiment result registry + +Each child directory is an immutable result record. Exact metrics, protocol, +raw console output and caveats live in that directory; this page only routes +readers to the right comparison. + +| Result record | Question answered | +|---|---| +| [`easycom_open_stream_vs_causal_prefix_2026-08-19/`](easycom_open_stream_vs_causal_prefix_2026-08-19/README.md) | How prepared-media open-stream and causal-prefix features compare | +| [`easycom_open_stream_8s_vs_20s_2026-08-23/`](easycom_open_stream_8s_vs_20s_2026-08-23/README.md) | Frozen open-stream context-length comparison | +| [`easycom_sparse_supervision_vs_lora_2026-08-24/`](easycom_sparse_supervision_vs_lora_2026-08-24/README.md) | Sparse supervision versus LoRA comparison | +| [`easycom_omni_lora_sparse_epoch2_2026-08-24/`](easycom_omni_lora_sparse_epoch2_2026-08-24/README.md) | Sparse epoch-2 LoRA result and provenance | +| [`easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/`](easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/README.md) | Learned last-four-layer mixture ablation | +| [`easycom_loss_structure_4ep_2026-08-26/`](easycom_loss_structure_4ep_2026-08-26/README.md) | LSTR-MCE versus independent BCE versus four-state CE | +| [`easycom_transition_auxiliary_4ep_2026-08-27/`](easycom_transition_auxiliary_4ep_2026-08-27/README.md) | Supervised transition head and expected-reward objectives | +| [`easycom_encoder_only_1s_4ep_2026-08-28/`](easycom_encoder_only_1s_4ep_2026-08-28/README.md) | Encoder-only A/V/AV versus global-Thinker local-feature baselines | + +## Interpretation rules + +- Do not compare full-timeline and causal-valid metrics without naming the + protocol. +- A single seed is preliminary even when its absolute improvement is large. +- Framewise mAP is retained for EgoSpeak comparability; it does not prove + correct floor-entry/yield behavior. +- `G+A+V > G+A` does not prove visual speaker grounding unless correct visual + alignment also beats wrong-person/time-shift controls. +- Historical transition-auxiliary results concern generic label changes, not + the new scenario-conditioned wearer entry/yield objective. + +## Required contents for a new result record + +```text +results/__/ + README.md # hypothesis, command/config, split, seeds, interpretation + summary.json # machine-readable aggregate and provenance + metrics.csv # when a tabular comparison exists + raw_console.txt # optional but recommended +``` + +Never overwrite a dated record with a new protocol. Create another directory +and link the relationship from both READMEs.