icassp2027-experiment/docs/easycom_egospeak_omni_finetune.md

# Stage two: selective MiniROADA + Omni residual fine-tuning

## Decision and claim boundary

The user requested this continuation after the
[`stage-one screen`](../results/easycom_egospeak_omni_residual_2026-09-16/README.md):
causal-valid dense mean mAP improved by +1.521 points, raw onset endpoint AP
by +0.600 and offset endpoint AP by +2.580. These are single-seed reported
results, not statistically significant semantic floor-control evidence.
Provisional ENTRY/YIELD evaluation is paused pending an annotation audit;
historical provisional results remain archived. P0 is still unfinished.

This stage means selective MiniROADA temporal/forecast adaptation, **not**
Omni LoRA, a person-memory adapter, new feature extraction or a semantic head.
It is an exploratory implementation of the earlier selective-unfreezing route.
Do not use Sessions 1--3 to choose learning rates, windows, loss weights,
unfreezing policy or the stopping epoch. A comparison observed on the exposed
test split remains exploratory; choose any later settings on training-side
development data and report the test exposure honestly.

## What changes

Load `checkpoint_last.pt` from stage one strictly. Preserve its learned
nonzero residual and verify exact initial recovery of its fused/base logits
on a training example. The original released checkpoint is not needed again:
the first-stage checkpoint already contains its full frozen base parameters.

| Component | Stage-two default |
|---|---|
| Released RGB and wav2vec2 feature extractors | Frozen; reuse existing arrays |
| MiniROADA RGB+audio input projection (`layer1`) | Frozen, dropout remains off |
| MiniROADA GRU | Trainable, learning rate 1e-5 |
| Anticipation projection and shared classification head | Trainable, learning rate 1e-5 |
| Ego/Omni projections, tail embedding and fusion residual | Continue training, learning rate 1e-4 |
| Qwen2.5-Omni backbone | Frozen; reuse exactly the first-stage sidecars |

The GRU state contributes to both the original anticipation logits and the
feature-fusion residual. Unlike stage one, neither path is enclosed in
`no_grad` or detached when selected for training. The default trainable
MiniROADA modules contain 19,942,403 parameters, in addition to the existing
residual branch. The RGB/audio encoders themselves are not loaded or trained;
this is partial task-model fine-tuning with frozen feature extractors, not
raw-media end-to-end training.

Loss, window/stride, grid, class order and ten horizons stay unchanged.
Window size and stride inherit the stage-one checkpoint and mismatched
overrides fail. Readout mode/layer fusion/dimension must also match; when the
stage-one sibling `metrics.json` is available, its full saved Omni extraction
signature is checked. A relocated checkpoint without that provenance prints
a warning, so the same original Omni root must be reused explicitly.

Default optimizer: AdamW, weight decay 0.05, gradient clipping at norm 1.0.
These defaults are specified before the new run, not tuned on test results.
The first training batch logs separate fusion/base gradient norms. Final
evaluation occurs only after the fixed additional epoch budget, with no
test-side early stopping or best-checkpoint selection.

## Session 4 smoke test

From the repository root, reuse the same two feature roots as stage one:

```bash
EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom
OMNI_ROOT=/path/to/the/same/first_stage/omni_features
STAGE1=results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt

CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --stage-one-checkpoint "$STAGE1" \
  --train-sessions Session_4 \
  --eval-sessions Session_4 \
  --epochs 1 \
  --max-train-examples 32 \
  --batch-size 8 \
  --num-workers 0 \
  --output-dir results/easycom_egospeak_omni_finetune_smoke_2026-09-16
```

Check exact initialization, nonzero fusion/base gradients, finite loss,
`[T,3,10]` predictions, `[T,10]` common valid masks and raw endpoint counts.
The printed cutoff is `(source_row+1)/5` s; input stops at that row, and
future labels are rows source+1 through source+10. Smoke predictions are not
the final model's calibration predictions. Start the formal run from the
original stage-one checkpoint, not the smoke checkpoint.

## Fixed-budget continuation

After the GPU smoke audit passes:

```bash
CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
  --egospeak-feature-root "$EGO_ROOT" \
  --omni-feature-root "$OMNI_ROOT" \
  --stage-one-checkpoint "$STAGE1" \
  --base-trainable gru-heads \
  --fusion-lr 1e-4 \
  --base-lr 1e-5 \
  --epochs 4 \
  --batch-size 64 \
  --num-workers 4 \
  --seed 0 \
  --output-dir results/easycom_egospeak_omni_finetune_seed0_2026-09-16
```

Defaults use Sessions 4--12 for training and Sessions 1--3 for final
evaluation. Training or a first-stage checkpoint trained on Sessions 1--3 is
rejected. Float32 is the first reproducibility run; do not enable AMP until
its output/causality audit passes. Full-session GRU inference is the default;
`--eval-chunk-size N` carries recurrent state across chunks if memory requires.
Nonempty output directories are never overwritten.

An additional four epochs means **4+4=8 total adaptation epochs**. It cannot
isolate unfreezing against the original four-epoch first stage. The required
matched-budget continuation control changes only these arguments:

```bash
  --base-trainable none \
  --output-dir results/easycom_egospeak_omni_fusion_continue_seed0_2026-09-16
```

Keep the same stage-one checkpoint, seed, training split, four additional
epochs, sampling, optimizer and fusion learning rate. `--base-trainable heads`
is a separate predefined ablation that freezes the GRU and adapts only the
forecast heads plus residual; do not select among policies on test scores.

## Outputs and reading AP

```text
RESULT/
  README.md
  checkpoint_last.pt              # full adapted model, config, parent hash, budget
  metrics.json                    # adapted dense + raw endpoint AP
  raw_endpoint_metrics.csv        # three methods, ten horizons, AP and counts
  pred/, valid/, raw_eval/         # second-stage predictions
  stage_one_reference/
    metrics.json, pred/, valid/, raw_eval/
  frozen_egospeak_reference/
    metrics.json, pred/, valid/, raw_eval/
```

Both references are immutable copies of the first-stage checkpoint: one
includes its learned fusion, the other uses its original frozen MiniROADA
path. The adapted base branch is never mislabeled as frozen EgoSpeak.
Every method uses the same valid source rows. Reference directories are
directly compatible with the existing evaluator. A loader for the new full
model is `egospeak_omni_finetune.load_stage_two_checkpoint`; stage-one and
stage-two checkpoint formats are intentionally distinct.

Dense metrics retain both full-timeline and causal-valid protocols. Raw onset
uses current wearer OFF, future ON versus future OFF with score p_wearer;
offset uses current wearer ON, future OFF versus future ON with score
1-p_wearer. AP pools eligible pairs over evaluated sessions separately for
each horizon; the headline averages ten APs. This is endpoint anticipation,
not timestamp detection or semantic ENTRY/YIELD. No Session 4 threshold
calibration or annotation file is needed for these APs. An undefined horizon
is preserved as null, not silently removed from the mean.

The CSV/JSON store AP **fractions**, while console output uses percentages.
To print the raw results:

```bash
python3 - <<'PY'
import pandas as pd
p = "results/easycom_egospeak_omni_finetune_seed0_2026-09-16/raw_endpoint_metrics.csv"
x = pd.read_csv(p)
cols = ["onset_endpoint_ap", "offset_endpoint_ap"]
x[cols] *= 100
print(x[["run", "nominal_seconds", *cols]].round(3).to_string(index=False))
print(x.groupby("run")[cols].mean().round(3).to_string())
PY
```

If any horizon is undefined, use the JSON headline (which stays null) instead
of pandas' default mean that skips NaN. No semantic opportunity evaluation is
run by this launcher.

## Local validation

### Evaluate a saved checkpoint without training again

For a completed short continuation, use the separate inference-only CLI:

```bash
CUDA_VISIBLE_DEVICES=0 python3 -u infer_egospeak_omni_finetune.py \
  --checkpoint results/easycom_egospeak_omni_finetune_low_lr_1ep_diagnostic_2026-09-16/checkpoint_last.pt \
  --sessions Session_1 Session_2 Session_3 \
  --output-dir results/easycom_egospeak_omni_finetune_low_lr_1ep_test_2026-09-16
```

Feature roots/readout settings inherit the checkpoint; optional
`--egospeak-feature-root`/`--omni-feature-root` override relocated roots while
saved extraction contracts are checked. Float32 inference, `pred/valid/raw_eval`,
dense and threshold-free raw endpoint AP are exported. No optimizer is created
and the checkpoint is unchanged. This repeated official-test evaluation is
exploratory; do not choose subsequent parameters or epochs from its scores.

### Tests

```bash
python3 -m unittest test_egospeak_omni_finetune test_egospeak_omni_residual -v
```

Tests cover trained residual preservation, selected-module gradients,
reference immutability, policy masks, causal perturbation, carried-state
chunk equivalence, checkpoint failure/success paths, raw AP and a tiny CPU
training/export/reload smoke. They do not substitute for the real Session 4
GPU smoke or establish second-stage performance; that run remains unmeasured.