Add offline sparse supervision control

This commit is contained in:
Xin Wang
2026-08-24 11:45:11 +08:00
parent 43cf1f386e
commit edd37dc36a
2 changed files with 149 additions and 3 deletions
+56
View File
@@ -904,6 +904,62 @@ Recommended experimental procedure:
Class-balanced/focal loss, a learned turn-query token, and LoRA are later
ablations. They should not replace the frozen-Omni linear/MLP-head baseline.
### Offline frozen states with transition-sparse supervision
This is the clean sampling-only control. It does not reconstruct cached AV
tokens and does not run the Thinker during training. It loads the same dense
5 Hz frozen states used by the standard head baseline, then retains only the
`target_index` values listed by the existing transition-sparse cache
manifests. The representation, head architecture, optimizer, 50 epochs, loss,
targets, and final dense test evaluator are otherwise unchanged.
Set `FROZEN_FEATURE_ROOT` to the exact feature directory used by the matching
dense-supervision baseline. Do not compare sparse 8-second states with a
dense 20-second baseline (or vice versa):
```bash
FROZEN_FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_20s_decord
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
SPARSE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s
CUDA_VISIBLE_DEVICES=3 \
python train_turn_head.py \
--feature-root "$FROZEN_FEATURE_ROOT" \
--target-root "$TARGET_ROOT" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_7" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_11" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_10" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_6" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_9" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_4" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_8" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_5" \
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_12" \
--output checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
--epochs 50 \
--batch-size 512 \
--expected-mode any \
--device cuda:0
```
Evaluation remains dense 5 Hz on the official test sessions:
```bash
CUDA_VISIBLE_DEVICES=3 \
python infer_turn_head.py \
--feature-root "$FROZEN_FEATURE_ROOT" \
--target-root "$TARGET_ROOT" \
--checkpoint checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
--output-dir results/easycom_frozen_states_sparse_supervision_50ep \
--expected-mode any \
--device cuda:0
```
Compare this result against a head trained for the same 50 epochs on every
dense 5 Hz anchor from the same `FROZEN_FEATURE_ROOT`. A large drop isolates
the problem to sparse supervision; a close result means sparse supervision is
not sufficient to explain the LoRA regression.
## Thinker-LoRA sampling and cache policy
This is an implemented second-stage experiment, not yet a reported benchmark