wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add overlap working-memory ablation and LoRA epoch2 record

Browse Source
This commit is contained in:
Xin Wang
2026-08-24 09:52:56 +08:00
parent 3cc0c3da85
commit 20446c0bd8
6 changed files with 851 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+171
View File
@@ -1104,6 +1104,24 @@ This control should approach the previously reported frozen 8-second score of
difference between dense-cache reconstruction and the earlier open-stream
feature files; a close match isolates the regression to LoRA/head training.
### Recorded sparse-LoRA results
The current single-run EasyCom results confirm negative transfer from this
sparse LoRA recipe:
| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
|---|---:|---:|
| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
| Sparse LoRA weights-only epoch2 | 54.19850 | 54.20711 |
Epoch2 is a weights-only continuation: adapter/head weights resume, but AdamW
and the scheduler restart. It is 2.32630 mAP points below epoch1 and 3.96241
below the frozen dense-cache control in the full-timeline protocol. The largest
regression is camera-wearer speaking. Do not continue the same recipe to a
third run. The complete snapshot and per-class metrics are archived in
[`results/easycom_omni_lora_sparse_epoch2_2026-08-24/`](results/easycom_omni_lora_sparse_epoch2_2026-08-24/).
For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
producer-consumer benchmark. The producer GPU runs the frozen AV towers and
places current-tail embeddings in a bounded CPU queue while the consumer GPU
@@ -1233,6 +1251,159 @@ per-class AP over identical seeds. A useful result requires the fused model to
beat both controls consistently; otherwise a gain could merely come from the
extra fusion parameters or the stronger recent-window shortcut.
### Frozen serialized-recap ablation (implemented)
Before adding a learned two-branch fusion module, test a stricter frozen-Omni
variant with exactly the same 2048-dimensional readout and prediction head as
the current 8-second baseline. Mode `overlap-working-memory` serializes the two
timescales inside one still-open Thinker media turn:
```text
total AV presentation budget = 8 s
fixed history budget = 6 s = three cached 2 s blocks
working recap = 2 s = [t-2,t]
t = 20.4 s:
fixed: [14,16] [16,18] [18,20]
working: [18.4,20.4]
-> one frozen Thinker state [2048]
```
The overlap is intentional. It presents recent evidence twice but does not add
AV-token duration beyond the matched 8-second budget. Thus this ablation trades
older unique context for stronger recent evidence instead of silently giving
the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS
grid. The working recap uses an endpoint-aligned 2 FPS grid and includes the
current causal frame:
```text
fixed [18,20): 18.0 18.5 19.0 19.5
working [18.4,20.4]: 18.9 19.4 19.9 20.4
```
Keeping four frames means the oldest visual tick is replaced by the current
frame rather than adding a fifth frame; this isolates endpoint recency without
increasing visual tokens. Audio remains the full, exactly half-open
`[start,end)` slice. Consequently, call this an endpoint-aligned recent-video
intervention, not an exact timestamp-preserving resampling of both modalities.
The recap is serialized as the next TMRoPE chunk; it is deliberately a
recent-evidence recap, not a claim that duplicated samples occur later in
physical time. This mode is an ablation, not native Qwen streaming semantics,
and the Omni backbone stays frozen.
One deliberate side effect is a 2-second phase cycle: overlap falls toward
0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the
boundary. Every output stores this value in
`overlap_working_memory_policy.layout`. If aggregate mAP changes, also inspect
performance by `step_index % 10`; a boundary-only gain or failure is a chunk
phase artifact rather than general evidence for better recent semantics.
Run a 24-second L20 smoke test first:
```bash
mkdir -p features/easycom_overlap_working_8s_smoke
CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
python streaming_turn_model_directly.py \
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
--end 24 \
--mode overlap-working-memory \
--memory-seconds 8 \
--working-memory-seconds 2 \
--video-fps 2 \
--frame-cache-size 128 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--dtype bfloat16 \
--flash-attention-2 \
--device cuda:0 \
--verify-every 10 \
--output features/easycom_overlap_working_8s_smoke/Session_7_24s.pt
```
Expected shape is `(120, 2048)`. `overlap_cache_replay` should be numerically
exact or effectively exact. Inspect the saved timing policy with:
```bash
python - <<'PY'
import torch
p = torch.load(
"features/easycom_overlap_working_8s_smoke/Session_7_24s.pt",
map_location="cpu",
weights_only=False,
)
print(p["mode"], tuple(p["states"].shape))
print(p["overlap_working_memory_policy"]["working_media"])
for row in p["overlap_working_memory_policy"]["layout"][98:102]:
print(row)
PY
```
For one complete session, replace `--end 24` with its target and use a
session-level output filename:
```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s
CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
python streaming_turn_model_directly.py \
--model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
--mode overlap-working-memory \
--memory-seconds 8 \
--working-memory-seconds 2 \
--video-fps 2 \
--frame-cache-size 128 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--dtype bfloat16 \
--flash-attention-2 \
--device cuda:0 \
--output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s/Session_7.pt
```
After all train/test sessions have been extracted, train the unchanged frozen
head by setting only the expected feature mode:
```bash
python train_turn_head.py \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--output checkpoints/easycom_overlap_working_frozen_8s.pt \
--expected-mode overlap-working-memory \
--epochs 50 \
--batch-size 512 \
--device cuda:0
python infer_turn_head.py \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \
--output-dir results/easycom_overlap_working_frozen_8s \
--expected-mode overlap-working-memory \
--device cuda:0
```
The decisive comparison is against the frozen single-state 8-second baseline
using the same split, head, optimization, and seeds. Report the full
per-horizon curve: the hypothesis predicts the clearest gain at 0.2--1.0
seconds; degradation at 1.2--2.0 seconds would indicate that repeated local
cues displaced too much useful long context. Only if this parameter-matched
test is positive should the learned two-branch fusion experiment be added.
## Single-window feature inspection
`qwen_omni_features.py` is useful for inspecting a short window or layer, but