# Frozen EgoSpeak + Omni tail-aware residual fusion ## Scope This is the first-stage dual-stream screen. It preserves the released EgoSpeak MiniROADA RGB, wav2vec2-audio, GRU, and ten-horizon anticipation path as a frozen base. Existing causal 5 Hz Qwen2.5-Omni states provide a second branch. Only a tail-phase-aware residual fusion module is trained: ```text released EgoSpeak RGB + audio -> frozen MiniROADA -> h_ego -> z_ego ---------+ existing causal Omni g_full -> projection -----------------------> fusion -> Δz | z_fused = z_ego + Δz ``` The final residual projection is initialized to exactly zero. Before any optimization, the launcher verifies bit-exact equality between fused and base logits on a real training example. Unlike the earlier scalar-zero-gate track screen, the zero is placed only on the final residual output layer; after its first update, gradients can reach the fusion branch. This is **baseline-recoverable initialization**, not a guarantee that the trained fusion will be non-inferior on unseen Sessions 1--3. The launcher therefore exports matched frozen-base and fused predictions from the same forward pass. ## Inputs No Qwen re-extraction is needed for this stage. The Omni root must contain complete `Session_X.pt` files with: - `mode=open-stream-replay` by default; - dense zero-origin `label_indices`; - one state per 5 Hz target row; - `step_seconds=0.2` and `native_block_seconds=2.0`; - a full-session `times` grid and no `partial=True` marker. The saved state is already the readout over rolling completed blocks plus the current temporary partial block. The fusion branch derives the ten-value tail phase (`0.2, ..., 2.0` seconds) from the validated grid. It does not claim to separate a completed-history state from a partial-tail state. Such an ablation would require a new Omni extraction field. The EgoSpeak feature root must contain the released directories: ```text rgb_kinetics_resnet50/Session_X.npy [T,2048] audio_wav2vec2/Session_X.npy [T,5120] target_perframe/Session_X.npy [T,3] ``` The base checkpoint must be the released raw MiniROADA state dict. It is loaded strictly, and every base parameter remains frozen and in evaluation mode so its dropout cannot reactivate during residual training. ## Audit the inputs Set machine-specific paths rather than committing `/data1` paths: ```bash EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom OMNI_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack BASE_CKPT=third_party/EgoSpeak/MiniROAD/checkpoints/miniroad_easycom_AV.pth ``` The default `--omni-layer-fusion mean` uses a fixed mean for the existing last-four stack and is also valid for a single `[T,D]` state. Use `last` only for an explicit layer ablation. Audit the released EgoSpeak arrays: ```bash python3 audit_egospeak_features.py \ --feature-root "$EGO_ROOT" ``` The training launcher independently validates every selected Omni and EgoSpeak session and aborts on a length, time-grid, mode, dimension, or zero-origin mismatch. ## Session 4 smoke test Use a new output directory for every invocation; the launcher refuses to overwrite an existing nonempty result. ```bash CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_residual.py \ --egospeak-feature-root "$EGO_ROOT" \ --omni-feature-root "$OMNI_ROOT" \ --base-checkpoint "$BASE_CKPT" \ --train-sessions Session_4 \ --eval-sessions Session_4 \ --epochs 1 \ --max-train-examples 32 \ --batch-size 8 \ --num-workers 0 \ --output-dir results/easycom_egospeak_omni_residual_smoke ``` The required early output is: ```text Initialization audit: fused logits == frozen EgoSpeak logits exactly ``` If a raw released Session 4 prediction is available, add: ```bash --reference-prediction-root \ third_party/EgoSpeak/MiniROAD/output_cfg/miniroad_aa_easycom_session4_AV/eval ``` This is a stronger implementation audit: the launcher checks that its frozen base reproduces the upstream raw `[source,horizon,class]` probabilities. ## Fixed four-epoch Sessions 4--12 run After the smoke test passes, omit session lists to use the official EasyCom train/test split. Evaluation happens once after epoch 4; the test split is not used for checkpoint selection. ```bash CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_residual.py \ --egospeak-feature-root "$EGO_ROOT" \ --omni-feature-root "$OMNI_ROOT" \ --base-checkpoint "$BASE_CKPT" \ --epochs 4 \ --batch-size 64 \ --num-workers 4 \ --seed 0 \ --output-dir results/easycom_egospeak_omni_residual_seed0 ``` Do not add `--amp` to the first reproducibility run. Once the float32 base has been compared against released predictions, AMP can be screened separately. The default `--eval-chunk-size 0` processes each evaluation session in one GRU pass, matching the released evaluator. Set a positive value only if inference memory requires chunked recurrent execution, and repeat the reference audit. The output contains: ```text checkpoint_last.pt self-contained model/checkpoint provenance metrics.json fused dense metrics plus matched frozen base raw_eval/ fused MiniROADA source-indexed probabilities pred/ and valid/ fused evaluator-ready aligned arrays frozen_egospeak_raw_eval/ matched frozen-base raw probabilities frozen_egospeak_pred/ matched frozen-base aligned arrays ``` `metrics.json` keeps the fused `causal_valid_only` protocol at the top level, so it remains readable by `summarize_local_baselines.py`. ## Interpretation and next gate The first seed-0 dense/raw screen is now [`recorded`](../results/easycom_egospeak_omni_residual_2026-09-16/README.md). At the user's request, provisional semantic evaluation is paused for an annotation audit. An exploratory selective GRU/head continuation is implemented in the [`stage-two runbook`](easycom_egospeak_omni_finetune.md); this does not imply that the semantic P0 gate below has passed. This first stage uses only the released dense future-activity objective. It tests whether Omni adds useful context without discarding the task-specific EgoSpeak path. It does not train semantic ENTRY/YIELD heads, and Sessions 1--3 LLM-assisted reviews must never be used for optimization or model selection. Advance to selective MiniROADA unfreezing or semantic opportunity heads only if the frozen residual passes these audits: 1. exact zero-residual initialization and upstream base reproduction; 2. no causal/time-grid mismatch; 3. stable dense and boundary changes over seeds; 4. opportunity-conditioned ENTRY/YIELD reported separately, without treating dense wearer probability as semantic intent.