Stage two: selective MiniROADA + Omni residual fine-tuning

Decision and claim boundary

The user requested this continuation after the stage-one screen: causal-valid dense mean mAP improved by +1.521 points, raw onset endpoint AP by +0.600 and offset endpoint AP by +2.580. These are single-seed reported results, not statistically significant semantic floor-control evidence. Provisional ENTRY/YIELD evaluation is paused pending an annotation audit; historical provisional results remain archived. P0 is still unfinished.

This stage means selective MiniROADA temporal/forecast adaptation, not Omni LoRA, a person-memory adapter, new feature extraction or a semantic head. It is an exploratory implementation of the earlier selective-unfreezing route. Do not use Sessions 1--3 to choose learning rates, windows, loss weights, unfreezing policy or the stopping epoch. A comparison observed on the exposed test split remains exploratory; choose any later settings on training-side development data and report the test exposure honestly.

What changes

Load checkpoint_last.pt from stage one strictly. Preserve its learned nonzero residual and verify exact initial recovery of its fused/base logits on a training example. The original released checkpoint is not needed again: the first-stage checkpoint already contains its full frozen base parameters.

Component	Stage-two default
Released RGB and wav2vec2 feature extractors	Frozen; reuse existing arrays
MiniROADA RGB+audio input projection (`layer1`)	Frozen, dropout remains off
MiniROADA GRU	Trainable, learning rate 1e-5
Anticipation projection and shared classification head	Trainable, learning rate 1e-5
Ego/Omni projections, tail embedding and fusion residual	Continue training, learning rate 1e-4
Qwen2.5-Omni backbone	Frozen; reuse exactly the first-stage sidecars

The GRU state contributes to both the original anticipation logits and the feature-fusion residual. Unlike stage one, neither path is enclosed in no_grad or detached when selected for training. The default trainable MiniROADA modules contain 19,942,403 parameters, in addition to the existing residual branch. The RGB/audio encoders themselves are not loaded or trained; this is partial task-model fine-tuning with frozen feature extractors, not raw-media end-to-end training.

Loss, window/stride, grid, class order and ten horizons stay unchanged. Window size and stride inherit the stage-one checkpoint and mismatched overrides fail. Readout mode/layer fusion/dimension must also match; when the stage-one sibling metrics.json is available, its full saved Omni extraction signature is checked. A relocated checkpoint without that provenance prints a warning, so the same original Omni root must be reused explicitly.

Default optimizer: AdamW, weight decay 0.05, gradient clipping at norm 1.0. These defaults are specified before the new run, not tuned on test results. The first training batch logs separate fusion/base gradient norms. Final evaluation occurs only after the fixed additional epoch budget, with no test-side early stopping or best-checkpoint selection.

Session 4 smoke test

From the repository root, reuse the same two feature roots as stage one:

EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom
OMNI_ROOT=/path/to/the/same/first_stage/omni_features
STAGE1=results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt

CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --stage-one-checkpoint "$STAGE1" \
  --train-sessions Session_4 \
  --eval-sessions Session_4 \
  --epochs 1 \
  --max-train-examples 32 \
  --batch-size 8 \
  --num-workers 0 \
  --output-dir results/easycom_egospeak_omni_finetune_smoke_2026-09-16

Check exact initialization, nonzero fusion/base gradients, finite loss, [T,3,10] predictions, [T,10] common valid masks and raw endpoint counts. The printed cutoff is (source_row+1)/5 s; input stops at that row, and future labels are rows source+1 through source+10. Smoke predictions are not the final model's calibration predictions. Start the formal run from the original stage-one checkpoint, not the smoke checkpoint.

Fixed-budget continuation

After the GPU smoke audit passes:

CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --stage-one-checkpoint "$STAGE1" \
  --base-trainable gru-heads \
  --fusion-lr 1e-4 \
  --base-lr 1e-5 \
  --epochs 4 \
  --batch-size 64 \
  --num-workers 4 \
  --seed 0 \
  --output-dir results/easycom_egospeak_omni_finetune_seed0_2026-09-16

Defaults use Sessions 4--12 for training and Sessions 1--3 for final evaluation. Training or a first-stage checkpoint trained on Sessions 1--3 is rejected. Float32 is the first reproducibility run; do not enable AMP until its output/causality audit passes. Full-session GRU inference is the default; --eval-chunk-size N carries recurrent state across chunks if memory requires. Nonempty output directories are never overwritten.

An additional four epochs means 4+4=8 total adaptation epochs. It cannot isolate unfreezing against the original four-epoch first stage. The required matched-budget continuation control changes only these arguments:

  --base-trainable none \
  --output-dir results/easycom_egospeak_omni_fusion_continue_seed0_2026-09-16

Keep the same stage-one checkpoint, seed, training split, four additional epochs, sampling, optimizer and fusion learning rate. --base-trainable heads is a separate predefined ablation that freezes the GRU and adapts only the forecast heads plus residual; do not select among policies on test scores.

Outputs and reading AP

RESULT/
  README.md
  checkpoint_last.pt              # full adapted model, config, parent hash, budget
  metrics.json                    # adapted dense + raw endpoint AP
  raw_endpoint_metrics.csv        # three methods, ten horizons, AP and counts
  pred/, valid/, raw_eval/         # second-stage predictions
  stage_one_reference/
    metrics.json, pred/, valid/, raw_eval/
  frozen_egospeak_reference/
    metrics.json, pred/, valid/, raw_eval/

Both references are immutable copies of the first-stage checkpoint: one includes its learned fusion, the other uses its original frozen MiniROADA path. The adapted base branch is never mislabeled as frozen EgoSpeak. Every method uses the same valid source rows. Reference directories are directly compatible with the existing evaluator. A loader for the new full model is egospeak_omni_finetune.load_stage_two_checkpoint; stage-one and stage-two checkpoint formats are intentionally distinct.

Dense metrics retain both full-timeline and causal-valid protocols. Raw onset uses current wearer OFF, future ON versus future OFF with score p_wearer; offset uses current wearer ON, future OFF versus future ON with score 1-p_wearer. AP pools eligible pairs over evaluated sessions separately for each horizon; the headline averages ten APs. This is endpoint anticipation, not timestamp detection or semantic ENTRY/YIELD. No Session 4 threshold calibration or annotation file is needed for these APs. An undefined horizon is preserved as null, not silently removed from the mean.

The CSV/JSON store AP fractions, while console output uses percentages. To print the raw results:

python3 - <<'PY'
import pandas as pd
p = "results/easycom_egospeak_omni_finetune_seed0_2026-09-16/raw_endpoint_metrics.csv"
x = pd.read_csv(p)
cols = ["onset_endpoint_ap", "offset_endpoint_ap"]
x[cols] *= 100
print(x[["run", "nominal_seconds", *cols]].round(3).to_string(index=False))
print(x.groupby("run")[cols].mean().round(3).to_string())
PY

If any horizon is undefined, use the JSON headline (which stays null) instead of pandas' default mean that skips NaN. No semantic opportunity evaluation is run by this launcher.

Local validation

python3 -m unittest test_egospeak_omni_finetune test_egospeak_omni_residual -v

Tests cover trained residual preservation, selected-module gradients, reference immutability, policy masks, causal perturbation, carried-state chunk equivalence, checkpoint failure/success paths, raw AP and a tiny CPU training/export/reload smoke. They do not substitute for the real Session 4 GPU smoke or establish second-stage performance; that run remains unmeasured.

8.4 KiB Raw Blame History