Add offline sparse supervision control
This commit is contained in:
@@ -904,6 +904,62 @@ Recommended experimental procedure:
|
||||
Class-balanced/focal loss, a learned turn-query token, and LoRA are later
|
||||
ablations. They should not replace the frozen-Omni linear/MLP-head baseline.
|
||||
|
||||
### Offline frozen states with transition-sparse supervision
|
||||
|
||||
This is the clean sampling-only control. It does not reconstruct cached AV
|
||||
tokens and does not run the Thinker during training. It loads the same dense
|
||||
5 Hz frozen states used by the standard head baseline, then retains only the
|
||||
`target_index` values listed by the existing transition-sparse cache
|
||||
manifests. The representation, head architecture, optimizer, 50 epochs, loss,
|
||||
targets, and final dense test evaluator are otherwise unchanged.
|
||||
|
||||
Set `FROZEN_FEATURE_ROOT` to the exact feature directory used by the matching
|
||||
dense-supervision baseline. Do not compare sparse 8-second states with a
|
||||
dense 20-second baseline (or vice versa):
|
||||
|
||||
```bash
|
||||
FROZEN_FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_20s_decord
|
||||
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
||||
SPARSE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s
|
||||
|
||||
CUDA_VISIBLE_DEVICES=3 \
|
||||
python train_turn_head.py \
|
||||
--feature-root "$FROZEN_FEATURE_ROOT" \
|
||||
--target-root "$TARGET_ROOT" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_7" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_11" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_10" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_6" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_9" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_4" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_8" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_5" \
|
||||
--sparse-anchor-cache-dir "$SPARSE_ROOT/Session_12" \
|
||||
--output checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
|
||||
--epochs 50 \
|
||||
--batch-size 512 \
|
||||
--expected-mode any \
|
||||
--device cuda:0
|
||||
```
|
||||
|
||||
Evaluation remains dense 5 Hz on the official test sessions:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=3 \
|
||||
python infer_turn_head.py \
|
||||
--feature-root "$FROZEN_FEATURE_ROOT" \
|
||||
--target-root "$TARGET_ROOT" \
|
||||
--checkpoint checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
|
||||
--output-dir results/easycom_frozen_states_sparse_supervision_50ep \
|
||||
--expected-mode any \
|
||||
--device cuda:0
|
||||
```
|
||||
|
||||
Compare this result against a head trained for the same 50 epochs on every
|
||||
dense 5 Hz anchor from the same `FROZEN_FEATURE_ROOT`. A large drop isolates
|
||||
the problem to sparse supervision; a close result means sparse supervision is
|
||||
not sufficient to explain the LoRA regression.
|
||||
|
||||
## Thinker-LoRA sampling and cache policy
|
||||
|
||||
This is an implemented second-stage experiment, not yet a reported benchmark
|
||||
|
||||
Reference in New Issue
Block a user