Add higher-capacity last-four-layer fusion

2026-08-26 13:06:18 +08:00
parent 43e0e14cf5
commit f45ea9cbc6
9 changed files with 397 additions and 17 deletions
@@ -0,0 +1,111 @@
+# EasyCom frozen Qwen2.5-Omni-3B last-4 learned mixture
+
+Recorded on **2026-08-26** from the user-reported remote training and
+`infer_turn_head.py` output. This is a single-run diagnostic result, not yet a
+multi-seed paper number.
+
+## Experiment contract
+
+- Dataset: EasyCom.
+- Test split: confirmed as EgoSpeak official Sessions 3, 2, and 1. Training is
+  recorded as the script's default official Sessions 4--12; verify this against
+  the remote checkpoint JSON because the supplied excerpt starts at epoch 24.
+- Backbone: frozen Qwen2.5-Omni-3B Thinker.
+- Streaming mode: open-stream replay with an 8-second causal-memory budget.
+- Input sampling: 2 FPS video and raw audio.
+- Prediction rate: 5 Hz, with ten future horizons from 0.2 to 2.0 seconds.
+- Readout: the query states from the final four Thinker layers, ordered oldest
+  to newest, followed by a learned global softmax layer mixture and the same
+  `10 x 3` anticipation MLP.
+- Training: head and four scalar layer logits only; Thinker remains frozen.
+- Evaluator: EgoSpeak full-timeline per-frame AP/mAP.
+
+The remote artifact is named
+`easycom_frozen_3b_8s_last4_learned_mix_50ep.pth`, but the supplied log and the
+user report both show that the actual run ended at **epoch 25/25**. The
+`50ep` suffix is therefore a naming error and must not be used as provenance.
+
+## Summary
+
+All values below are percentages. The comparison baseline is the previously
+recorded frozen 3B, 8-second, open-stream, single-final-layer run evaluated on
+the same official test sessions.
+
+| Readout | 0.2s full mAP | Full mean mAP | Delta at 0.2s | Delta mean |
+|---|---:|---:|---:|---:|
+| Single final-layer query | 71.99768 | 58.51937 | -- | -- |
+| **Last-4 learned mixture** | **73.45085** | **60.53423** | **+1.45317** | **+2.01486** |
+
+The last-4 readout improves every evaluated horizon. The largest gain is
+`+2.41084` at 1.0 seconds; the smallest is still a positive `+1.45317` at
+0.2 seconds.
+
+## Per-horizon full-timeline results
+
+| Horizon | Last-4 mAP | Single-layer mAP | Delta | Background AP | Wearer AP | Others AP |
+|---:|---:|---:|---:|---:|---:|---:|
+| 0.2s | 73.45085 | 71.99768 | +1.45317 | 56.69385 | 78.35111 | 85.30760 |
+| 0.4s | 68.58645 | 66.74289 | +1.84356 | 49.77891 | 73.48058 | 82.49988 |
+| 0.6s | 64.44213 | 62.38972 | +2.05241 | 44.51563 | 68.39317 | 80.41758 |
+| 0.8s | 61.34852 | 59.12229 | +2.22623 | 40.98328 | 64.20310 | 78.85917 |
+| 1.0s | 59.12464 | 56.71380 | +2.41084 | 38.99833 | 60.75541 | 77.62018 |
+| 1.2s | 57.53667 | 55.36300 | +2.17367 | 37.76815 | 58.15110 | 76.69076 |
+| 1.4s | 56.43710 | 54.21297 | +2.22413 | 37.22701 | 56.12642 | 75.95786 |
+| 1.6s | 55.42035 | 53.43221 | +1.98814 | 36.72606 | 54.14045 | 75.39454 |
+| 1.8s | 54.76334 | 52.84703 | +1.91631 | 36.45931 | 52.80226 | 75.02845 |
+| 2.0s | 54.23227 | 52.37207 | +1.86020 | 36.45292 | 51.56488 | 74.67902 |
+| **Mean** | **60.53423** | **58.51937** | **+2.01486** | **41.56035** | **61.79685** | **78.24550** |
+
+Averaged over horizons, the AP gains relative to the single-layer baseline are
+`+1.92941` for background, `+2.88478` for camera-wearer speaking, and
+`+1.23041` for other-people speaking. The wearer class contributes the largest
+improvement, which is encouraging for the short-horizon turn-taking objective.
+
+## Learned layer mixture
+
+| Layer order | Weight |
+|---|---:|
+| Oldest of selected four | 0.246384 |
+| Second | 0.248327 |
+| Third | 0.241774 |
+| Final layer | 0.263515 |
+
+The weights remain close to uniform: normalized entropy is `0.999619`, the L1
+distance from `[0.25, 0.25, 0.25, 0.25]` is only `0.02703`, and the largest
+weight exceeds the smallest by only `0.021741`. Consequently, this experiment
+supports **multi-layer query readout**, but does not yet show that learning the
+mixture is better than a fixed last-4 mean.
+
+## Interpretation and next controls
+
+1. This is the strongest recorded frozen-Omni EasyCom result in the project so
+   far: `73.45085` at 0.2 seconds and `60.53423` mean mAP.
+2. The gain occurs at all horizons and all three classes, rather than coming
+   from a single AP outlier.
+3. Because the learned weights are nearly uniform, the first required ablation
+   is a **fixed uniform last-4 mean** with the same feature files, head recipe,
+   epoch count, and seed.
+4. Then compare each of the four individual query layers. This separates layer
+   ensembling from access to a specifically better intermediate layer.
+5. Repeat the single-layer, fixed-mean, and learned-mixture variants with at
+   least three identical seeds before using the gain as a paper result.
+6. The supplied output contains no strict causal-valid table. Archive the
+   remote `metrics.json` or rerun the evaluator before the publication table.
+7. The single-layer comparison came from an earlier run; its random seed and
+   head-training duration must be matched in the fixed-mean ablation.
+
+## Provenance
+
+- Remote feature root:
+  `/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack`
+- Remote checkpoint:
+  `checkpoints/easycom_frozen_3b_8s_last4_learned_mix_50ep.pth`
+- Remote result directory:
+  `results/easycom_frozen_3b_8s_last4_learned_mix_50ep`
+- Final training loss: `0.822849`.
+- Raw console excerpt: [`raw_console.txt`](raw_console.txt).
+- Machine-readable values: [`summary.json`](summary.json) and
+  [`metrics.csv`](metrics.csv).
+- Raw remote `metrics.json` archived locally: no.
+- Exact training-session list verified from the remote checkpoint JSON: no.
+- Multi-seed: no.
@@ -0,0 +1,12 @@
+horizon_seconds,last4_map,single_final_layer_map,delta,background_ap,camera_wearer_ap,other_people_ap
+0.2,73.45085,71.99768,1.45317,56.69385,78.35111,85.30760
+0.4,68.58645,66.74289,1.84356,49.77891,73.48058,82.49988
+0.6,64.44213,62.38972,2.05241,44.51563,68.39317,80.41758
+0.8,61.34852,59.12229,2.22623,40.98328,64.20310,78.85917
+1.0,59.12464,56.71380,2.41084,38.99833,60.75541,77.62018
+1.2,57.53667,55.36300,2.17367,37.76815,58.15110,76.69076
+1.4,56.43710,54.21297,2.22413,37.22701,56.12642,75.95786
+1.6,55.42035,53.43221,1.98814,36.72606,54.14045,75.39454
+1.8,54.76334,52.84703,1.91631,36.45931,52.80226,75.02845
+2.0,54.23227,52.37207,1.86020,36.45292,51.56488,74.67902
+mean,60.53423,58.51937,2.01486,41.56035,61.79685,78.24550
@@ -0,0 +1,33 @@
+head epoch 24/25: 100% | 142/142 | 137.53 batch/s
+epoch 24: train_loss=0.822925, layer_weights=[0.2464,0.2483,0.2418,0.2635]
+head epoch 25/25: 100% | 142/142 | 139.49 batch/s, lr=0
+epoch 25: train_loss=0.822849, layer_weights=[0.2464,0.2483,0.2418,0.2635]
+checkpoint: checkpoints/easycom_frozen_3b_8s_last4_learned_mix_50ep.pth
+training summary: checkpoints/easycom_frozen_3b_8s_last4_learned_mix_50ep.pth.json
+saved layer mixture weights (oldest->newest): 0.246384, 0.248327, 0.241774, 0.263515
+
+feature root: /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
+test sessions: Session_3, Session_2, Session_1
+
+egospeak_full_timeline:
+Action anticipation (0.20s) perframe mAP: 73.45085
+  AP: background=56.69385, camera_wearer_speaking=78.35111, other_people_speaking=85.30760; n=23048
+Action anticipation (0.40s) perframe mAP: 68.58645
+  AP: background=49.77891, camera_wearer_speaking=73.48058, other_people_speaking=82.49988; n=23048
+Action anticipation (0.60s) perframe mAP: 64.44213
+  AP: background=44.51563, camera_wearer_speaking=68.39317, other_people_speaking=80.41758; n=23048
+Action anticipation (0.80s) perframe mAP: 61.34852
+  AP: background=40.98328, camera_wearer_speaking=64.20310, other_people_speaking=78.85917; n=23048
+Action anticipation (1.00s) perframe mAP: 59.12464
+  AP: background=38.99833, camera_wearer_speaking=60.75541, other_people_speaking=77.62018; n=23048
+Action anticipation (1.20s) perframe mAP: 57.53667
+  AP: background=37.76815, camera_wearer_speaking=58.15110, other_people_speaking=76.69076; n=23048
+Action anticipation (1.40s) perframe mAP: 56.43710
+  AP: background=37.22701, camera_wearer_speaking=56.12642, other_people_speaking=75.95786; n=23048
+Action anticipation (1.60s) perframe mAP: 55.42035
+  AP: background=36.72606, camera_wearer_speaking=54.14045, other_people_speaking=75.39454; n=23048
+Action anticipation (1.80s) perframe mAP: 54.76334
+  AP: background=36.45931, camera_wearer_speaking=52.80226, other_people_speaking=75.02845; n=23048
+Action anticipation (2.00s) perframe mAP: 54.23227
+  AP: background=36.45292, camera_wearer_speaking=51.56488, other_people_speaking=74.67902; n=23048
+Action anticipation (mean) perframe mAP: 60.53423
@@ -0,0 +1,66 @@
+{
+  "schema_version": 1,
+  "recorded_date": "2026-08-26",
+  "status": "single_run_diagnostic_snapshot",
+  "dataset": "EasyCom",
+  "split": {
+    "train_sessions": [4, 5, 6, 7, 8, 9, 10, 11, 12],
+    "train_sessions_source": "train_turn_head.py default; not visible in supplied epoch-24 excerpt",
+    "test_sessions": [3, 2, 1],
+    "test_sessions_confirmed_in_console": true
+  },
+  "model": {
+    "backbone": "Qwen2.5-Omni-3B",
+    "thinker_frozen": true,
+    "streaming_mode": "open-stream-replay",
+    "memory_seconds": 8,
+    "video_fps": 2,
+    "prediction_hz": 5
+  },
+  "readout": {
+    "selected_layers": "last_4_query_states",
+    "layer_order": "oldest_to_newest",
+    "fusion": "learned_global_softmax",
+    "weights": [0.246384, 0.248327, 0.241774, 0.263515],
+    "normalized_entropy": 0.999619394230947,
+    "l1_distance_from_uniform": 0.02703,
+    "max_min_weight_gap": 0.021741
+  },
+  "training": {
+    "actual_epochs": 25,
+    "final_train_loss": 0.822849,
+    "artifact_name": "easycom_frozen_3b_8s_last4_learned_mix_50ep.pth",
+    "artifact_epoch_suffix_is_incorrect": true
+  },
+  "metrics_percent": {
+    "protocol": "egospeak_full_timeline",
+    "horizon_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0],
+    "map": [73.45085, 68.58645, 64.44213, 61.34852, 59.12464, 57.53667, 56.43710, 55.42035, 54.76334, 54.23227],
+    "background_ap": [56.69385, 49.77891, 44.51563, 40.98328, 38.99833, 37.76815, 37.22701, 36.72606, 36.45931, 36.45292],
+    "camera_wearer_ap": [78.35111, 73.48058, 68.39317, 64.20310, 60.75541, 58.15110, 56.12642, 54.14045, 52.80226, 51.56488],
+    "other_people_ap": [85.30760, 82.49988, 80.41758, 78.85917, 77.62018, 76.69076, 75.95786, 75.39454, 75.02845, 74.67902],
+    "mean_map": 60.53423,
+    "strict_causal_valid_reported": false
+  },
+  "comparison_to_single_final_layer_8s_percent": {
+    "baseline_mean_map": 58.51937,
+    "baseline_horizon_0.2_map": 71.99768,
+    "mean_map_delta": 2.01486,
+    "horizon_0.2_map_delta": 1.45317,
+    "horizon_map_delta": [1.45317, 1.84356, 2.05241, 2.22623, 2.41084, 2.17367, 2.22413, 1.98814, 1.91631, 1.86020],
+    "mean_class_ap_delta": {
+      "background": 1.92941,
+      "camera_wearer": 2.88478,
+      "other_people": 1.23041
+    }
+  },
+  "provenance": {
+    "source": "user-reported remote console output",
+    "remote_feature_root": "/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack",
+    "remote_checkpoint": "checkpoints/easycom_frozen_3b_8s_last4_learned_mix_50ep.pth",
+    "remote_result_dir": "results/easycom_frozen_3b_8s_last4_learned_mix_50ep",
+    "raw_metrics_json_archived": false,
+    "remote_checkpoint_json_archived": false,
+    "multi_seed": false
+  }
+}