Merge branch 'main' of http://182.92.86.220:8300/wx44wx/icassp2027-experiment

2026-09-17 01:30:39 +08:00
parent 6d0c8b733b 7344d0c469
commit 5c29457cc6
12 changed files with 1605 additions and 1 deletions
@@ -29,6 +29,7 @@ tests.
 | [`easycom_causal_person_adapter_v2.md`](easycom_causal_person_adapter_v2.md) | Dynamic-gated, zero-output causal person-track residual and matched L20 screen |
 | [`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md) | Controlled V2 temporal-stem ablation using EgoSpeak MiniROADA's layer order |
 | [`easycom_egospeak_omni_residual.md`](easycom_egospeak_omni_residual.md) | Stage-one frozen released MiniROADA plus zero-initialized tail-aware Omni residual fusion |
+| [`easycom_egospeak_omni_finetune.md`](easycom_egospeak_omni_finetune.md) | Stage-two selective MiniROADA GRU/head continuation with immutable stage-one/base references and raw endpoint AP |
 | [`easycom_highrate_person_av.md`](easycom_highrate_person_av.md) | Frozen 25 Hz causal ASD person evidence fused with 5 Hz global Omni context |
 | [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets |
 | [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
@@ -0,0 +1,208 @@
+# Stage two: selective MiniROADA + Omni residual fine-tuning
+
+## Decision and claim boundary
+
+The user requested this continuation after the
+[`stage-one screen`](../results/easycom_egospeak_omni_residual_2026-09-16/README.md):
+causal-valid dense mean mAP improved by +1.521 points, raw onset endpoint AP
+by +0.600 and offset endpoint AP by +2.580. These are single-seed reported
+results, not statistically significant semantic floor-control evidence.
+Provisional ENTRY/YIELD evaluation is paused pending an annotation audit;
+historical provisional results remain archived. P0 is still unfinished.
+
+This stage means selective MiniROADA temporal/forecast adaptation, **not**
+Omni LoRA, a person-memory adapter, new feature extraction or a semantic head.
+It is an exploratory implementation of the earlier selective-unfreezing route.
+Do not use Sessions 1--3 to choose learning rates, windows, loss weights,
+unfreezing policy or the stopping epoch. A comparison observed on the exposed
+test split remains exploratory; choose any later settings on training-side
+development data and report the test exposure honestly.
+
+## What changes
+
+Load `checkpoint_last.pt` from stage one strictly. Preserve its learned
+nonzero residual and verify exact initial recovery of its fused/base logits
+on a training example. The original released checkpoint is not needed again:
+the first-stage checkpoint already contains its full frozen base parameters.
+
+| Component | Stage-two default |
+|---|---|
+| Released RGB and wav2vec2 feature extractors | Frozen; reuse existing arrays |
+| MiniROADA RGB+audio input projection (`layer1`) | Frozen, dropout remains off |
+| MiniROADA GRU | Trainable, learning rate 1e-5 |
+| Anticipation projection and shared classification head | Trainable, learning rate 1e-5 |
+| Ego/Omni projections, tail embedding and fusion residual | Continue training, learning rate 1e-4 |
+| Qwen2.5-Omni backbone | Frozen; reuse exactly the first-stage sidecars |
+
+The GRU state contributes to both the original anticipation logits and the
+feature-fusion residual. Unlike stage one, neither path is enclosed in
+`no_grad` or detached when selected for training. The default trainable
+MiniROADA modules contain 19,942,403 parameters, in addition to the existing
+residual branch. The RGB/audio encoders themselves are not loaded or trained;
+this is partial task-model fine-tuning with frozen feature extractors, not
+raw-media end-to-end training.
+
+Loss, window/stride, grid, class order and ten horizons stay unchanged.
+Window size and stride inherit the stage-one checkpoint and mismatched
+overrides fail. Readout mode/layer fusion/dimension must also match; when the
+stage-one sibling `metrics.json` is available, its full saved Omni extraction
+signature is checked. A relocated checkpoint without that provenance prints
+a warning, so the same original Omni root must be reused explicitly.
+
+Default optimizer: AdamW, weight decay 0.05, gradient clipping at norm 1.0.
+These defaults are specified before the new run, not tuned on test results.
+The first training batch logs separate fusion/base gradient norms. Final
+evaluation occurs only after the fixed additional epoch budget, with no
+test-side early stopping or best-checkpoint selection.
+
+## Session 4 smoke test
+
+From the repository root, reuse the same two feature roots as stage one:
+
+```bash
+EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom
+OMNI_ROOT=/path/to/the/same/first_stage/omni_features
+STAGE1=results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt
+
+CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
+  --egospeak-feature-root "$EGO_ROOT" \
+  --omni-feature-root "$OMNI_ROOT" \
+  --stage-one-checkpoint "$STAGE1" \
+  --train-sessions Session_4 \
+  --eval-sessions Session_4 \
+  --epochs 1 \
+  --max-train-examples 32 \
+  --batch-size 8 \
+  --num-workers 0 \
+  --output-dir results/easycom_egospeak_omni_finetune_smoke_2026-09-16
+```
+
+Check exact initialization, nonzero fusion/base gradients, finite loss,
+`[T,3,10]` predictions, `[T,10]` common valid masks and raw endpoint counts.
+The printed cutoff is `(source_row+1)/5` s; input stops at that row, and
+future labels are rows source+1 through source+10. Smoke predictions are not
+the final model's calibration predictions. Start the formal run from the
+original stage-one checkpoint, not the smoke checkpoint.
+
+## Fixed-budget continuation
+
+After the GPU smoke audit passes:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
+  --egospeak-feature-root "$EGO_ROOT" \
+  --omni-feature-root "$OMNI_ROOT" \
+  --stage-one-checkpoint "$STAGE1" \
+  --base-trainable gru-heads \
+  --fusion-lr 1e-4 \
+  --base-lr 1e-5 \
+  --epochs 4 \
+  --batch-size 64 \
+  --num-workers 4 \
+  --seed 0 \
+  --output-dir results/easycom_egospeak_omni_finetune_seed0_2026-09-16
+```
+
+Defaults use Sessions 4--12 for training and Sessions 1--3 for final
+evaluation. Training or a first-stage checkpoint trained on Sessions 1--3 is
+rejected. Float32 is the first reproducibility run; do not enable AMP until
+its output/causality audit passes. Full-session GRU inference is the default;
+`--eval-chunk-size N` carries recurrent state across chunks if memory requires.
+Nonempty output directories are never overwritten.
+
+An additional four epochs means **4+4=8 total adaptation epochs**. It cannot
+isolate unfreezing against the original four-epoch first stage. The required
+matched-budget continuation control changes only these arguments:
+
+```bash
+  --base-trainable none \
+  --output-dir results/easycom_egospeak_omni_fusion_continue_seed0_2026-09-16
+```
+
+Keep the same stage-one checkpoint, seed, training split, four additional
+epochs, sampling, optimizer and fusion learning rate. `--base-trainable heads`
+is a separate predefined ablation that freezes the GRU and adapts only the
+forecast heads plus residual; do not select among policies on test scores.
+
+## Outputs and reading AP
+
+```text
+RESULT/
+  README.md
+  checkpoint_last.pt              # full adapted model, config, parent hash, budget
+  metrics.json                    # adapted dense + raw endpoint AP
+  raw_endpoint_metrics.csv        # three methods, ten horizons, AP and counts
+  pred/, valid/, raw_eval/         # second-stage predictions
+  stage_one_reference/
+    metrics.json, pred/, valid/, raw_eval/
+  frozen_egospeak_reference/
+    metrics.json, pred/, valid/, raw_eval/
+```
+
+Both references are immutable copies of the first-stage checkpoint: one
+includes its learned fusion, the other uses its original frozen MiniROADA
+path. The adapted base branch is never mislabeled as frozen EgoSpeak.
+Every method uses the same valid source rows. Reference directories are
+directly compatible with the existing evaluator. A loader for the new full
+model is `egospeak_omni_finetune.load_stage_two_checkpoint`; stage-one and
+stage-two checkpoint formats are intentionally distinct.
+
+Dense metrics retain both full-timeline and causal-valid protocols. Raw onset
+uses current wearer OFF, future ON versus future OFF with score p_wearer;
+offset uses current wearer ON, future OFF versus future ON with score
+1-p_wearer. AP pools eligible pairs over evaluated sessions separately for
+each horizon; the headline averages ten APs. This is endpoint anticipation,
+not timestamp detection or semantic ENTRY/YIELD. No Session 4 threshold
+calibration or annotation file is needed for these APs. An undefined horizon
+is preserved as null, not silently removed from the mean.
+
+The CSV/JSON store AP **fractions**, while console output uses percentages.
+To print the raw results:
+
+```bash
+python3 - <<'PY'
+import pandas as pd
+p = "results/easycom_egospeak_omni_finetune_seed0_2026-09-16/raw_endpoint_metrics.csv"
+x = pd.read_csv(p)
+cols = ["onset_endpoint_ap", "offset_endpoint_ap"]
+x[cols] *= 100
+print(x[["run", "nominal_seconds", *cols]].round(3).to_string(index=False))
+print(x.groupby("run")[cols].mean().round(3).to_string())
+PY
+```
+
+If any horizon is undefined, use the JSON headline (which stays null) instead
+of pandas' default mean that skips NaN. No semantic opportunity evaluation is
+run by this launcher.
+
+## Local validation
+
+### Evaluate a saved checkpoint without training again
+
+For a completed short continuation, use the separate inference-only CLI:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 -u infer_egospeak_omni_finetune.py \
+  --checkpoint results/easycom_egospeak_omni_finetune_low_lr_1ep_diagnostic_2026-09-16/checkpoint_last.pt \
+  --sessions Session_1 Session_2 Session_3 \
+  --output-dir results/easycom_egospeak_omni_finetune_low_lr_1ep_test_2026-09-16
+```
+
+Feature roots/readout settings inherit the checkpoint; optional
+`--egospeak-feature-root`/`--omni-feature-root` override relocated roots while
+saved extraction contracts are checked. Float32 inference, `pred/valid/raw_eval`,
+dense and threshold-free raw endpoint AP are exported. No optimizer is created
+and the checkpoint is unchanged. This repeated official-test evaluation is
+exploratory; do not choose subsequent parameters or epochs from its scores.
+
+### Tests
+
+```bash
+python3 -m unittest test_egospeak_omni_finetune test_egospeak_omni_residual -v
+```
+
+Tests cover trained residual preservation, selected-module gradients,
+reference immutability, policy masks, causal perturbation, carried-state
+chunk equivalence, checkpoint failure/success paths, raw AP and a tiny CPU
+training/export/reload smoke. They do not substitute for the real Session 4
+GPU smoke or establish second-stage performance; that run remains unmeasured.
@@ -154,6 +154,13 @@ so it remains readable by `summarize_local_baselines.py`.

 ## Interpretation and next gate

+The first seed-0 dense/raw screen is now
+[`recorded`](../results/easycom_egospeak_omni_residual_2026-09-16/README.md).
+At the user's request, provisional semantic evaluation is paused for an
+annotation audit. An exploratory selective GRU/head continuation is implemented
+in the [`stage-two runbook`](easycom_egospeak_omni_finetune.md); this does not
+imply that the semantic P0 gate below has passed.
+
 This first stage uses only the released dense future-activity objective.  It
 tests whether Omni adds useful context without discarding the task-specific
 EgoSpeak path.  It does not train semantic ENTRY/YIELD heads, and Sessions