Frozen EgoSpeak + Omni tail-aware residual fusion

Scope

This is the first-stage dual-stream screen. It preserves the released EgoSpeak MiniROADA RGB, wav2vec2-audio, GRU, and ten-horizon anticipation path as a frozen base. Existing causal 5 Hz Qwen2.5-Omni states provide a second branch. Only a tail-phase-aware residual fusion module is trained:

released EgoSpeak RGB + audio -> frozen MiniROADA -> h_ego -> z_ego ---------+
existing causal Omni g_full -> projection -----------------------> fusion -> Δz
                                                                            |
                                                        z_fused = z_ego + Δz

The final residual projection is initialized to exactly zero. Before any optimization, the launcher verifies bit-exact equality between fused and base logits on a real training example. Unlike the earlier scalar-zero-gate track screen, the zero is placed only on the final residual output layer; after its first update, gradients can reach the fusion branch.

This is baseline-recoverable initialization, not a guarantee that the trained fusion will be non-inferior on unseen Sessions 1--3. The launcher therefore exports matched frozen-base and fused predictions from the same forward pass.

Inputs

No Qwen re-extraction is needed for this stage. The Omni root must contain complete Session_X.pt files with:

mode=open-stream-replay by default;
dense zero-origin label_indices;
one state per 5 Hz target row;
step_seconds=0.2 and native_block_seconds=2.0;
a full-session times grid and no partial=True marker.

The saved state is already the readout over rolling completed blocks plus the current temporary partial block. The fusion branch derives the ten-value tail phase (0.2, ..., 2.0 seconds) from the validated grid. It does not claim to separate a completed-history state from a partial-tail state. Such an ablation would require a new Omni extraction field.

The EgoSpeak feature root must contain the released directories:

rgb_kinetics_resnet50/Session_X.npy  [T,2048]
audio_wav2vec2/Session_X.npy         [T,5120]
target_perframe/Session_X.npy        [T,3]

The base checkpoint must be the released raw MiniROADA state dict. It is loaded strictly, and every base parameter remains frozen and in evaluation mode so its dropout cannot reactivate during residual training.

Audit the inputs

Set machine-specific paths rather than committing /data1 paths:

EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom
OMNI_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
BASE_CKPT=third_party/EgoSpeak/MiniROAD/checkpoints/miniroad_easycom_AV.pth

The default --omni-layer-fusion mean uses a fixed mean for the existing last-four stack and is also valid for a single [T,D] state. Use last only for an explicit layer ablation.

Audit the released EgoSpeak arrays:

python3 audit_egospeak_features.py \
  --feature-root "$EGO_ROOT"

The training launcher independently validates every selected Omni and EgoSpeak session and aborts on a length, time-grid, mode, dimension, or zero-origin mismatch.

Session 4 smoke test

Use a new output directory for every invocation; the launcher refuses to overwrite an existing nonempty result.

CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_residual.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --base-checkpoint "$BASE_CKPT" \
  --train-sessions Session_4 \
  --eval-sessions Session_4 \
  --epochs 1 \
  --max-train-examples 32 \
  --batch-size 8 \
  --num-workers 0 \
  --output-dir results/easycom_egospeak_omni_residual_smoke

The required early output is:

Initialization audit: fused logits == frozen EgoSpeak logits exactly

If a raw released Session 4 prediction is available, add:

  --reference-prediction-root \
    third_party/EgoSpeak/MiniROAD/output_cfg/miniroad_aa_easycom_session4_AV/eval

This is a stronger implementation audit: the launcher checks that its frozen base reproduces the upstream raw [source,horizon,class] probabilities.

Fixed four-epoch Sessions 4--12 run

After the smoke test passes, omit session lists to use the official EasyCom train/test split. Evaluation happens once after epoch 4; the test split is not used for checkpoint selection.

CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_residual.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --base-checkpoint "$BASE_CKPT" \
  --epochs 4 \
  --batch-size 64 \
  --num-workers 4 \
  --seed 0 \
  --output-dir results/easycom_egospeak_omni_residual_seed0

Do not add --amp to the first reproducibility run. Once the float32 base has been compared against released predictions, AMP can be screened separately. The default --eval-chunk-size 0 processes each evaluation session in one GRU pass, matching the released evaluator. Set a positive value only if inference memory requires chunked recurrent execution, and repeat the reference audit.

The output contains:

checkpoint_last.pt             self-contained model/checkpoint provenance
metrics.json                   fused dense metrics plus matched frozen base
raw_eval/                      fused MiniROADA source-indexed probabilities
pred/ and valid/               fused evaluator-ready aligned arrays
frozen_egospeak_raw_eval/      matched frozen-base raw probabilities
frozen_egospeak_pred/          matched frozen-base aligned arrays

metrics.json keeps the fused causal_valid_only protocol at the top level, so it remains readable by summarize_local_baselines.py.

Interpretation and next gate

The first seed-0 dense/raw screen is now recorded. At the user's request, provisional semantic evaluation is paused for an annotation audit. An exploratory selective GRU/head continuation is implemented in the stage-two runbook; this does not imply that the semantic P0 gate below has passed.

This first stage uses only the released dense future-activity objective. It tests whether Omni adds useful context without discarding the task-specific EgoSpeak path. It does not train semantic ENTRY/YIELD heads, and Sessions 1--3 LLM-assisted reviews must never be used for optimization or model selection.

Advance to selective MiniROADA unfreezing or semantic opportunity heads only if the frozen residual passes these audits:

exact zero-residual initialization and upstream base reproduction;
no causal/time-grid mismatch;
stable dense and boundary changes over seeds;
opportunity-conditioned ENTRY/YIELD reported separately, without treating dense wearer probability as semantic intent.

6.7 KiB Raw Permalink Blame History