icassp2027-experiment/docs/easycom_egospeak_omni_residual.md

# Frozen EgoSpeak + Omni tail-aware residual fusion

## Scope

This is the first-stage dual-stream screen.  It preserves the released
EgoSpeak MiniROADA RGB, wav2vec2-audio, GRU, and ten-horizon anticipation path
as a frozen base.  Existing causal 5 Hz Qwen2.5-Omni states provide a second
branch.  Only a tail-phase-aware residual fusion module is trained:

```text
released EgoSpeak RGB + audio -> frozen MiniROADA -> h_ego -> z_ego ---------+
existing causal Omni g_full -> projection -----------------------> fusion -> Δz
                                                                            |
                                                        z_fused = z_ego + Δz
```

The final residual projection is initialized to exactly zero.  Before any
optimization, the launcher verifies bit-exact equality between fused and base
logits on a real training example.  Unlike the earlier scalar-zero-gate track
screen, the zero is placed only on the final residual output layer; after its
first update, gradients can reach the fusion branch.

This is **baseline-recoverable initialization**, not a guarantee that the
trained fusion will be non-inferior on unseen Sessions 1--3.  The launcher
therefore exports matched frozen-base and fused predictions from the same
forward pass.

## Inputs

No Qwen re-extraction is needed for this stage.  The Omni root must contain
complete `Session_X.pt` files with:

- `mode=open-stream-replay` by default;
- dense zero-origin `label_indices`;
- one state per 5 Hz target row;
- `step_seconds=0.2` and `native_block_seconds=2.0`;
- a full-session `times` grid and no `partial=True` marker.

The saved state is already the readout over rolling completed blocks plus the
current temporary partial block.  The fusion branch derives the ten-value tail
phase (`0.2, ..., 2.0` seconds) from the validated grid.  It does not claim to
separate a completed-history state from a partial-tail state.  Such an
ablation would require a new Omni extraction field.

The EgoSpeak feature root must contain the released directories:

```text
rgb_kinetics_resnet50/Session_X.npy  [T,2048]
audio_wav2vec2/Session_X.npy         [T,5120]
target_perframe/Session_X.npy        [T,3]
```

The base checkpoint must be the released raw MiniROADA state dict.  It is
loaded strictly, and every base parameter remains frozen and in evaluation
mode so its dropout cannot reactivate during residual training.

## Audit the inputs

Set machine-specific paths rather than committing `/data1` paths:

```bash
EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom
OMNI_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
BASE_CKPT=third_party/EgoSpeak/MiniROAD/checkpoints/miniroad_easycom_AV.pth
```

The default `--omni-layer-fusion mean` uses a fixed mean for the existing
last-four stack and is also valid for a single `[T,D]` state.  Use `last` only
for an explicit layer ablation.

Audit the released EgoSpeak arrays:

```bash
python3 audit_egospeak_features.py \
  --feature-root "$EGO_ROOT"
```

The training launcher independently validates every selected Omni and
EgoSpeak session and aborts on a length, time-grid, mode, dimension, or
zero-origin mismatch.

## Session 4 smoke test

Use a new output directory for every invocation; the launcher refuses to
overwrite an existing nonempty result.

```bash
CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_residual.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --base-checkpoint "$BASE_CKPT" \
  --train-sessions Session_4 \
  --eval-sessions Session_4 \
  --epochs 1 \
  --max-train-examples 32 \
  --batch-size 8 \
  --num-workers 0 \
  --output-dir results/easycom_egospeak_omni_residual_smoke
```

The required early output is:

```text
Initialization audit: fused logits == frozen EgoSpeak logits exactly
```

If a raw released Session 4 prediction is available, add:

```bash
  --reference-prediction-root \
    third_party/EgoSpeak/MiniROAD/output_cfg/miniroad_aa_easycom_session4_AV/eval
```

This is a stronger implementation audit: the launcher checks that its frozen
base reproduces the upstream raw `[source,horizon,class]` probabilities.

## Fixed four-epoch Sessions 4--12 run

After the smoke test passes, omit session lists to use the official EasyCom
train/test split.  Evaluation happens once after epoch 4; the test split is not
used for checkpoint selection.

```bash
CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_residual.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --base-checkpoint "$BASE_CKPT" \
  --epochs 4 \
  --batch-size 64 \
  --num-workers 4 \
  --seed 0 \
  --output-dir results/easycom_egospeak_omni_residual_seed0
```

Do not add `--amp` to the first reproducibility run.  Once the float32 base has
been compared against released predictions, AMP can be screened separately.
The default `--eval-chunk-size 0` processes each evaluation session in one GRU
pass, matching the released evaluator.  Set a positive value only if inference
memory requires chunked recurrent execution, and repeat the reference audit.

The output contains:

```text
checkpoint_last.pt             self-contained model/checkpoint provenance
metrics.json                   fused dense metrics plus matched frozen base
raw_eval/                      fused MiniROADA source-indexed probabilities
pred/ and valid/               fused evaluator-ready aligned arrays
frozen_egospeak_raw_eval/      matched frozen-base raw probabilities
frozen_egospeak_pred/          matched frozen-base aligned arrays
```

`metrics.json` keeps the fused `causal_valid_only` protocol at the top level,
so it remains readable by `summarize_local_baselines.py`.

## Interpretation and next gate

This first stage uses only the released dense future-activity objective.  It
tests whether Omni adds useful context without discarding the task-specific
EgoSpeak path.  It does not train semantic ENTRY/YIELD heads, and Sessions
1--3 LLM-assisted reviews must never be used for optimization or model
selection.

Advance to selective MiniROADA unfreezing or semantic opportunity heads only
if the frozen residual passes these audits:

1. exact zero-residual initialization and upstream base reproduction;
2. no causal/time-grid mismatch;
3. stable dense and boundary changes over seeds;
4. opportunity-conditioned ENTRY/YIELD reported separately, without treating
   dense wearer probability as semantic intent.