docs: add AI-friendly repository navigation

.gitignore

@@ -0,0 +1,19 @@
+.DS_Store
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.coverage
+
+# Generated experiment assets; datasets/models are expected outside the repo.
+/checkpoints/
+/features/
+/logs/
+/manifests/
+/outputs/
+/target_frame/
+
+# Local editors and environments.
+.idea/
+.vscode/
+.venv/
+venv/

AGENTS.md

@@ -1,2 +1,52 @@
-作为一名资深音视频，深度学习国际知名导师。
+# AI collaboration rules
-对于我的想法，引导我以思辨的方式继续，指导我完成icassp2027论文实验和编写。
+
+你是一名资深音视频与深度学习研究导师。目标不是无条件接受想法，而是帮助用户形成
+可证伪、可复现、能支撑 ICASSP 2027 投稿的研究问题、实验和论文表述。
+
+## 开始工作前的阅读顺序
+
+1. 先读 [`PROJECT_CONTEXT.md`](PROJECT_CONTEXT.md)：代码入口、数据契约、当前状态。
+2. 涉及论文方向时读
+   [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md)。
+3. 涉及 novelty/related work 时读
+   [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md)
+   和 [`literature_notes/README.md`](literature_notes/README.md)。
+4. 只在需要具体命令或历史实现细节时读取长篇 [`README.md`](README.md) 的相关章节，
+   不要默认把整份 README 当作当前实验优先级。
+
+## 当前唯一优先级
+
+先冻结 wearer floor-event 评测，再训练新模型。现有
+`evaluate_floor_state_changes.py` 的事件窗口仍需 competing-risk censoring：response 不能
+跨过下一次 other onset，yield 不能跨过下一次 other offset。在该问题修正前，事件结果
+只能称为 timing-derived proxy。
+
+## 不可破坏的研究契约
+
+- 严格因果：任何输入音频、视频帧、人物框或归一化统计不得来自预测截止时间之后。
+- 时间网格：目标和全局状态为 5 Hz；第 `t` 个状态对应截止到 `(t+1)/5` 秒的输入。
+- 输出：10 个未来 horizon，0.2--2.0 秒；类别顺序固定为 background、wearer、others。
+- EasyCom 官方划分：Session 4--12 训练，Session 1--3 测试。不能用测试集调阈值、窗口、
+  loss 权重或 early stopping。
+- `others` 是聚合活动，不能据此声称知道具体说话人、addressee、语义 EOT 或 INT。
+- 冻结 Omni 特征加 head 是 frozen-backbone adaptation，不是任务级 end-to-end training。
+- 计划文档中的模块不得被写成“已实现”；以代码、测试和结果记录三者为准。
+
+## 代码与目录约束
+
+- 根目录 Python/shell 文件是远程 L20 命令正在使用的稳定 CLI。除非同时提供兼容 wrapper、
+  更新所有引用并运行测试，否则不要把它们移动到 `src/`、`scripts/` 或 `tests/`。
+- 数据集、模型、features、checkpoints 和 logs 都是外部/生成资产，不进入 Git。
+- 新实验结果放入 `results/<dataset>_<question>_<date>/`，至少包含 `README.md`、机器可读
+  `summary.json` 或 `metrics.csv`，并记录 split、seed、checkpoint 和协议。
+- 新运行手册放 `docs/`；新论文摘要放 `literature_notes/`；自动审计输出放 `reports/`。
+- 使用相对仓库路径写文档，不固化某一台机器的 `/data` 或 `/data1` 路径为唯一配置。
+
+## 验证要求
+
+- 文档改动：运行 `git diff --check` 并确认相对链接存在。
+- evaluator/label 改动：运行对应 `test_*.py`，并人工打印至少一个事件时间轴。
+- feature contract 改动：同时测试 `easycom_head.py`、`local_turn_features.py` 的加载失败和
+  成功路径，不能静默 padding、clamp 或错位。
+- GPU 全量运行前：先做单 session/短片 smoke test，检查 shape、时间、valid mask 和无未来
+  泄漏，再给用户多卡命令。

PROJECT_CONTEXT.md

@@ -0,0 +1,248 @@
+# AI Project Context
+
+This is the short, authoritative repository map for humans and AI agents.
+The long [`README.md`](README.md) is an operational history and command
+reference; it is not the experiment scheduler.
+
+## One-sentence objective
+
+Determine whether strictly causal, speaker-grounded audiovisual evidence from
+Qwen2.5-Omni can improve an invisible egocentric wearer's **floor-entry** and
+**floor-yield** behavior in natural multiparty conversation.
+
+The paper is not simply “replace EgoSpeak's encoders with a newer backbone.”
+The decisive test is whether correct person-aligned vision reduces false
+wearer entries during other-to-other conversation at matched wearer-onset
+recall, without hurting addressed responses or yield behavior.
+
+## Source-of-truth order
+
+| Need | Authoritative file |
+|---|---|
+| Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) |
+| Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) |
+| Code/data map and current blocker | This file |
+| Runnable commands and implementation history | [`README.md`](README.md) and [`docs/README.md`](docs/README.md) |
+| Archived measured results | [`results/README.md`](results/README.md) |
+| Literature summaries | [`literature_notes/README.md`](literature_notes/README.md) |
+
+When files disagree, prefer the newest scoped document and verify behavior in
+code/tests. A planned experiment is not an implemented feature or a measured
+result.
+
+## Current state
+
+### Verified assets
+
+- Strictly causal 5 Hz Qwen2.5-Omni global state extraction on EasyCom.
+- Prepared-media and direct-original-media extraction paths.
+- Frozen head training/inference with EgoSpeak-compatible 10-horizon AP/mAP.
+- EasyCom local Qwen visual crops and recent mixed-audio sidecars.
+- Encoder-only A/V/AV and `G`, `G+A`, `G+V`, `G+A+V` baselines.
+- Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
+- Timing-derived wearer floor-state and event diagnostics.
+
+### Current blocker
+
+`evaluate_floor_state_changes.py` counts a wearer transition anywhere in a
+fixed response window. Before this becomes the paper event protocol, add
+competing-risk censoring:
+
+- an other-offset response window ends at the next other onset;
+- an other-onset yield window ends at the next other offset.
+
+Until then, EOT/HOLD, response and yield results are proxy diagnostics, not
+semantic event ground truth.
+
+### Next experiment gates
+
+1. **P0:** fix/freeze event attribution, splits and scenario definitions.
+2. **P1:** probe aggregate activity and per-person active-speaker grounding;
+   compare correct, wrong-person and time-shifted visual inputs.
+3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
+4. **P3:** train an anonymous person-wise causal side branch with matched
+   temporal-capacity controls.
+5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.
+
+## Stable data contracts
+
+### Time and labels
+
+- Sampling rate: 5 Hz (`STEP_SECONDS = 0.2`).
+- State row `j` uses media no later than `(j+1)/5` seconds.
+- Future horizons: 10, corresponding to 0.2, 0.4, ..., 2.0 seconds.
+- Class order: `background`, `camera_wearer_speaking`,
+  `other_people_speaking`.
+- EasyCom targets are dense `[T, 3]`; predictions are
+  `[target_time, class, future_horizon]`.
+- EasyCom split: train Session 4--12; test Session 1--3.
+
+### Global feature sidecar
+
+Produced by `streaming_turn_model.py` or
+`streaming_turn_model_directly.py` as a `.pt` dictionary. Required training
+fields include:
+
+- `states`: `[T, 2048]` or multi-layer `[T, L, 2048]`;
+- `times`: `[0.2, 0.4, ...]`;
+- `label_indices`: dense zero-origin indices into the target timeline;
+- extraction `mode`, configuration and source metadata.
+
+Never silently accept non-zero origins, missing label indices or mismatched
+time grids.
+
+### Local feature sidecar
+
+Produced by `extract_turn_local_features.py`, validated by
+`local_turn_features.py`, and stored as `qwen-local-pooled-v1`. It contains
+native 2048D visual/audio vectors, validity masks, times, label indices,
+source metadata and the exact extraction config. A partial smoke sidecar must
+not be reused as a full-session feature file.
+
+### Inference result
+
+`infer_turn_head.py` writes:
+
+```text
+RESULT_DIR/
+  metrics.json
+  pred/<sample>.npy       # [T, 3, 10]
+  valid/<sample>.npy      # [T, 10]
+  transition_pred/        # only when the auxiliary head exists
+```
+
+`evaluate_floor_state_changes.py` consumes `pred/` and `valid/`; it does not
+run a GPU model.
+
+## Canonical pipelines
+
+### Direct global features (preferred semantic path)
+
+```text
+original EasyCom/Ego4D media
+  -> streaming_turn_model_directly.py
+  -> global feature .pt
+  -> train_turn_head.py
+  -> checkpoint .pth + .json
+  -> infer_turn_head.py
+  -> pred/ + valid/ + metrics.json
+  -> evaluate_floor_state_changes.py
+```
+
+This path samples source video causally in Python and avoids requiring a
+re-encoded fixed-FPS MP4.
+
+### Prepared-media compatibility path
+
+```text
+prepare_easycom_session.py / prepare_ego4d_2fps.py
+  -> strict 2 FPS media + metadata
+  -> streaming_turn_model.py
+  -> the same training/inference pipeline
+```
+
+Keep it for reproducibility, cache-replay verification and comparisons with
+already extracted features.
+
+### Local evidence path
+
+```text
+raw media + person boxes + existing global reference .pt
+  -> extract_turn_local_features.py
+     (or extract_ego4d_local_multigpu.py supervisor)
+  -> local .pt sidecar
+  -> train_turn_head.py --local-feature-root ...
+  -> infer_turn_head.py --local-feature-root ...
+```
+
+The current local visual feature is pooled. It does not by itself prove
+speaker identity, active-speaker alignment or addressee understanding.
+
+## Root-code map
+
+Root scripts remain flat because remote L20 commands call them directly.
+Do not reorganize them physically without compatibility wrappers.
+
+### Shared libraries
+
+| File | Responsibility |
+|---|---|
+| `easycom_head.py` | Dataset constants, feature loading, heads, losses and AP metrics |
+| `local_turn_features.py` | Local-sidecar contract, alignment, crops and pooling |
+| `omni_cached_av.py` | Cached AV structures used by Omni extraction/LoRA paths |
+
+### Global feature extraction
+
+| File | Responsibility |
+|---|---|
+| `streaming_turn_model_directly.py` | Preferred direct raw-media causal extraction |
+| `streaming_turn_model.py` | Prepared-media causal-prefix/open-stream extraction and replay checks |
+| `qwen_omni_features.py` | Single-window feature inspection |
+| `cache_omni_av_pieces.py` | Build reusable AV pieces for LoRA experiments |
+| `extract_ego4d_batch.py` | Persistent-model Ego4D global batch extraction |
+| `analyze_block_phase.py` | Streaming block-phase diagnostics |
+| `profile_streaming_bottleneck_gpu7.py` | GPU extraction profiling |
+
+### Data preparation and audit
+
+| File | Responsibility |
+|---|---|
+| `easycom_manifest.py` | EasyCom clip inventory and dense-timeline validation |
+| `prepare_easycom_session.py` | Optional strict-grid EasyCom media preparation |
+| `prepare_ego4d_2fps.py` | Optional strict-grid Ego4D media preparation |
+| `prepare_encoder_only_features.py` | Convert local sidecars into encoder-only references |
+| `audit_egospeak_features.py` | Audit released EgoSpeak feature layout |
+| `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies |
+| `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication |
+| `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates |
+
+### Local features
+
+| File | Responsibility |
+|---|---|
+| `extract_turn_local_features.py` | Qwen person-crop visual and recent-audio sidecars |
+| `extract_ego4d_local_multigpu.py` | Foreground supervisor for disjoint multi-GPU workers |
+
+### Training, inference and evaluation
+
+| File | Responsibility |
+|---|---|
+| `train_turn_head.py` | Frozen-backbone head training, optional local fusion/auxiliary losses |
+| `infer_turn_head.py` | Save EgoSpeak-compatible predictions and AP/mAP |
+| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation |
+| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files |
+| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV |
+| `eval_omni_turn_lora.py` | Thinker LoRA evaluation |
+
+### Launchers and tests
+
+- `run_*.sh` and `run_*.py` are reproducible launchers, not libraries.
+- `smoke_test.py` is the model/environment smoke check.
+- `test_*.py` are CPU-oriented regression tests colocated at root for current
+  import and command compatibility.
+
+## Terminology restrictions
+
+- **Global (`G`)**: a Thinker readout that may mix audio, video and language
+  context; do not describe it as audio-free.
+- **Local audio (`A`)**: recent mixed audio encoded outside the Thinker.
+- **Local visual (`V`)**: person-box visual features; current pooling discards
+  stable person structure.
+- **Encoder-only**: Qwen modality encoder output without Thinker states.
+- **Speaker grounding**: evidence that the correct visible person is linked to
+  the correct speech/time. AV improvement alone is insufficient.
+- **Addressee grounding**: evidence about whom speech targets; aggregate
+  `others` labels cannot supervise it.
+- **Floor entry/yield**: observed wearer OFF→ON / ON→OFF behavior under a
+  defined causal event window; not automatically semantic response or
+  interruption intent.
+
+## Environment boundaries
+
+- The open-stream replay implementation is pinned to
+  `transformers==4.52.3` because it uses version-specific internals.
+- The local encoder path has been tested with 4.52.3 and 4.57.6; sidecars from
+  one experiment must share the same recorded contract/version.
+- Remote paths differ (`/data/...` and `/data1/...`). Commands should expose
+  dataset/model roots as variables rather than hard-code one machine.
+- Large models, datasets, checkpoints and extracted features live outside Git.

README.md

@@ -13,6 +13,25 @@ correct person-aligned vision should reduce false wearer entries while two
 other people are conversing, without reducing responses addressed to the
 wearer or degrading wearer yield decisions.
 
+## Repository navigation
+
+Start with [`PROJECT_CONTEXT.md`](PROJECT_CONTEXT.md) for the concise code map,
+data contracts, current blocker and canonical pipelines. This README is the
+long operational guide and experiment history.
+
+| Location | Contents |
+|---|---|
+| [`AGENTS.md`](AGENTS.md) | Research and code constraints for AI collaborators |
+| [`docs/`](docs/README.md) | Current paper plan, novelty audit and dataset runbooks |
+| [`results/`](results/README.md) | Immutable measured-result records and interpretation rules |
+| [`literature_notes/`](literature_notes/README.md) | Paper index and persistent summaries |
+| [`reports/`](reports/README.md) | Generated dataset audits; not model results |
+
+Root Python and shell files remain intentionally stable because remote L20
+commands invoke them directly. They are categorized in `PROJECT_CONTEXT.md`;
+moving them into `src/`, `scripts/` or `tests/` requires compatibility wrappers
+and regression testing.
+
 ## Current paper question and experiment plan (2026-08-30)
 
 The full staged plan is in

docs/README.md

@@ -0,0 +1,42 @@
+# Documentation index
+
+Documents are grouped by authority rather than by date. “Planned” never means
+implemented or validated.
+
+## Active paper definition
+
+| Document | Role |
+|---|---|
+| [`omni_speaker_floor_experiment_plan_2026-08-30.md`](omni_speaker_floor_experiment_plan_2026-08-30.md) | Current hypotheses, experiment gates, main table and stopping rules |
+| [`multiparty_egocentric_floor_gap_audit_2026-08-30.md`](multiparty_egocentric_floor_gap_audit_2026-08-30.md) | Novelty boundary, closest work and restricted claims |
+
+These two files supersede earlier documents when discussing the paper's main
+question. They do not supersede verified implementation details in code and
+tests.
+
+## Current implementation runbooks
+
+| Document | Use when |
+|---|---|
+| [`local_feature_baseline.md`](local_feature_baseline.md) | Extracting/validating EasyCom pooled local Qwen visual and audio sidecars |
+| [`easycom_encoder_only.md`](easycom_encoder_only.md) | Comparing EasyCom Qwen encoder-only A/V/AV representations |
+| [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison |
+| [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs |
+
+## Design history and implementation options
+
+| Document | Status |
+|---|---|
+| [`speaker_aware_thinker_plan_2026-08-28.md`](speaker_aware_thinker_plan_2026-08-28.md) | Historical design exploration; useful implementation detail, not current priority order |
+| [`trainable_speaker_fusion_survey_2026-08-28.md`](trainable_speaker_fusion_survey_2026-08-28.md) | Detailed person-branch/adapter survey feeding the current plan |
+
+## Where new material belongs
+
+- Current research decisions: a dated file in `docs/`, then link it from this
+  index and `PROJECT_CONTEXT.md`.
+- Dataset or command runbook: `docs/<dataset>_<task>.md`.
+- Measured experiment: `results/<dataset>_<question>_<date>/` with its own
+  README and machine-readable summary.
+- Paper summary: `literature_notes/NN_short_name.md` and update its index.
+- Generated audit report: `reports/`; do not present it as a hand-written
+  source of truth.

reports/README.md

@@ -0,0 +1,14 @@
+# Generated audit reports
+
+This directory stores machine-generated dataset audits, not model results or
+authoritative annotations.
+
+Current artifact:
+
+- [`egospeak_easycom_feature_audit.json`](egospeak_easycom_feature_audit.json):
+  released EgoSpeak EasyCom feature-layout audit.
+
+When adding a report, record the generating script, command, source dataset
+version and timestamp in the payload or an adjacent README. Derived timing or
+LLM-adjudicated labels must remain explicitly named proxies unless they have
+been manually validated as ground truth.

results/README.md

@@ -0,0 +1,41 @@
+# Experiment result registry
+
+Each child directory is an immutable result record. Exact metrics, protocol,
+raw console output and caveats live in that directory; this page only routes
+readers to the right comparison.
+
+| Result record | Question answered |
+|---|---|
+| [`easycom_open_stream_vs_causal_prefix_2026-08-19/`](easycom_open_stream_vs_causal_prefix_2026-08-19/README.md) | How prepared-media open-stream and causal-prefix features compare |
+| [`easycom_open_stream_8s_vs_20s_2026-08-23/`](easycom_open_stream_8s_vs_20s_2026-08-23/README.md) | Frozen open-stream context-length comparison |
+| [`easycom_sparse_supervision_vs_lora_2026-08-24/`](easycom_sparse_supervision_vs_lora_2026-08-24/README.md) | Sparse supervision versus LoRA comparison |
+| [`easycom_omni_lora_sparse_epoch2_2026-08-24/`](easycom_omni_lora_sparse_epoch2_2026-08-24/README.md) | Sparse epoch-2 LoRA result and provenance |
+| [`easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/`](easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/README.md) | Learned last-four-layer mixture ablation |
+| [`easycom_loss_structure_4ep_2026-08-26/`](easycom_loss_structure_4ep_2026-08-26/README.md) | LSTR-MCE versus independent BCE versus four-state CE |
+| [`easycom_transition_auxiliary_4ep_2026-08-27/`](easycom_transition_auxiliary_4ep_2026-08-27/README.md) | Supervised transition head and expected-reward objectives |
+| [`easycom_encoder_only_1s_4ep_2026-08-28/`](easycom_encoder_only_1s_4ep_2026-08-28/README.md) | Encoder-only A/V/AV versus global-Thinker local-feature baselines |
+
+## Interpretation rules
+
+- Do not compare full-timeline and causal-valid metrics without naming the
+  protocol.
+- A single seed is preliminary even when its absolute improvement is large.
+- Framewise mAP is retained for EgoSpeak comparability; it does not prove
+  correct floor-entry/yield behavior.
+- `G+A+V > G+A` does not prove visual speaker grounding unless correct visual
+  alignment also beats wrong-person/time-shift controls.
+- Historical transition-auxiliary results concern generic label changes, not
+  the new scenario-conditioned wearer entry/yield objective.
+
+## Required contents for a new result record
+
+```text
+results/<dataset>_<question>_<date>/
+  README.md       # hypothesis, command/config, split, seeds, interpretation
+  summary.json    # machine-readable aggregate and provenance
+  metrics.csv     # when a tabular comparison exists
+  raw_console.txt # optional but recommended
+```
+
+Never overwrite a dated record with a new protocol. Create another directory
+and link the relationship from both READMEs.