Add overlap working-memory ablation and LoRA epoch2 record

2026-08-24 09:52:56 +08:00
parent 3cc0c3da85
commit 20446c0bd8
6 changed files with 851 additions and 8 deletions
@@ -0,0 +1,77 @@
+# EasyCom: sparse Thinker-LoRA weights-only continuation (epoch2 snapshot)
+
+Recorded on **2026-08-24** from the remote EasyCom evaluation console. This is
+a single-run experiment snapshot, not a multi-seed paper result.
+
+## Experiment contract
+
+- Dataset: EasyCom with EgoSpeak's official split.
+- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
+- Test sessions: 3, 2, and 1 (`n=23048` full-timeline rows).
+- Backbone: Qwen2.5-Omni-3B Thinker with LoRA plus the `10 x 3` turn head.
+- Context: 8 seconds; 2 FPS video; predictions and labels at 5 Hz.
+- Training sampling: transition-sparse anchors (ordinary 1 Hz plus dense 5 Hz
+  anchors near target transitions).
+- Evaluation: dense 5 Hz AV cache on every test timestamp.
+- Continuation: adapter and head weights loaded from the epoch1 checkpoint,
+  but optimizer moments and the learning-rate scheduler were reinitialized.
+  Therefore, "epoch2" here means a weights-only second training run, not an
+  exact optimizer-state resume.
+- Reported checkpoint directory:
+  `checkpoints/omni_turn_lora_easycom_sparse_8s_epoch2_ddp4/`.
+
+## Summary
+
+All values are percentages.
+
+| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
+|---|---:|---:|
+| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
+| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
+| Sparse LoRA weights-only epoch2 | **54.19850** | **54.20711** |
+| Epoch2 minus epoch1 | -2.32630 | -2.32739 |
+| Epoch2 minus frozen control | -3.96241 | -3.96435 |
+
+The second run is materially worse than both epoch1 and the frozen control.
+This is not consistent with normal evaluator noise.
+
+## Per-horizon results
+
+| Horizon | Full mAP | Strict mAP | Background AP | Wearer AP | Others AP |
+|---:|---:|---:|---:|---:|---:|
+| 0.2 s | 65.55214 | 65.55385 | 49.03788 | 65.76312 | 81.85542 |
+| 0.4 s | 60.96517 | 60.96812 | 43.06922 | 61.14251 | 78.68378 |
+| 0.6 s | 56.79766 | 56.80301 | 39.37427 | 55.10528 | 75.91344 |
+| 0.8 s | 54.11741 | 54.12457 | 36.94546 | 51.56720 | 73.83955 |
+| 1.0 s | 52.09538 | 52.10400 | 35.40706 | 48.38835 | 72.49075 |
+| 1.2 s | 51.35516 | 51.36504 | 35.30874 | 46.86261 | 71.89412 |
+| 1.4 s | 50.99774 | 51.00914 | 35.29891 | 45.48818 | 72.20613 |
+| 1.6 s | 50.47830 | 50.49067 | 34.98109 | 44.74277 | 71.71105 |
+| 1.8 s | 50.00999 | 50.02296 | 34.64341 | 43.86161 | 71.52496 |
+| 2.0 s | 49.61605 | 49.62970 | 34.26290 | 43.14791 | 71.43735 |
+
+The AP columns above are from the full-timeline protocol. Complete AP values
+for both protocols are stored in [`metrics.csv`](metrics.csv), and
+[`summary.json`](summary.json) stores the machine-readable aggregate result.
+
+## Interpretation and decision
+
+The largest class-specific regression is camera-wearer speaking. Relative to
+epoch1, wearer AP falls from 68.77118 to 65.76312 at 0.2 seconds and from
+47.53875 to 43.14791 at 2.0 seconds. Background and other-speaker AP decline
+less at the long horizon. The pattern indicates negative transfer in the
+wearer-speaking representation rather than a uniform metric offset.
+
+Likely contributors are sparse-train/dense-evaluation distribution mismatch,
+too much LoRA capacity for EasyCom, and restarting the learning-rate schedule
+during weights-only continuation. Do not run a third continuation epoch with
+the same recipe. Keep the frozen 8-second result as the main representation
+baseline and next test the parameter-matched frozen overlap-working-memory
+ablation.
+
+## Provenance
+
+- Source: user-reported `eval_omni_turn_lora.py` console output.
+- Raw remote `metrics.json` archived here: no; values were transcribed.
+- Multi-seed: no.
+- Publication status: diagnostic negative result only.
@@ -0,0 +1,21 @@
+protocol,horizon,seconds,background_ap_percent,camera_wearer_ap_percent,other_people_ap_percent,map_percent,num_steps
+egospeak_full_timeline,1,0.2,49.03788,65.76312,81.85542,65.55214,23048
+egospeak_full_timeline,2,0.4,43.06922,61.14251,78.68378,60.96517,23048
+egospeak_full_timeline,3,0.6,39.37427,55.10528,75.91344,56.79766,23048
+egospeak_full_timeline,4,0.8,36.94546,51.56720,73.83955,54.11741,23048
+egospeak_full_timeline,5,1.0,35.40706,48.38835,72.49075,52.09538,23048
+egospeak_full_timeline,6,1.2,35.30874,46.86261,71.89412,51.35516,23048
+egospeak_full_timeline,7,1.4,35.29891,45.48818,72.20613,50.99774,23048
+egospeak_full_timeline,8,1.6,34.98109,44.74277,71.71105,50.47830,23048
+egospeak_full_timeline,9,1.8,34.64341,43.86161,71.52496,50.00999,23048
+egospeak_full_timeline,10,2.0,34.26290,43.14791,71.43735,49.61605,23048
+causal_valid_only,1,0.2,49.03788,65.76312,81.86055,65.55385,23045
+causal_valid_only,2,0.4,43.06922,61.14251,78.69264,60.96812,23042
+causal_valid_only,3,0.6,39.37427,55.10988,75.92489,56.80301,23039
+causal_valid_only,4,0.8,36.94546,51.57526,73.85298,54.12457,23036
+causal_valid_only,5,1.0,35.40706,48.39889,72.50604,52.10400,23033
+causal_valid_only,6,1.2,35.30874,46.87568,71.91070,51.36504,23030
+causal_valid_only,7,1.4,35.29891,45.50340,72.22511,51.00914,23027
+causal_valid_only,8,1.6,34.98337,44.75739,71.73126,50.49067,23024
+causal_valid_only,9,1.8,34.64563,43.87551,71.54773,50.02296,23021
+causal_valid_only,10,2.0,34.26505,43.16123,71.46280,49.62970,23018
@@ -0,0 +1,70 @@
+{
+  "schema_version": 1,
+  "recorded_date": "2026-08-24",
+  "status": "single_run_snapshot",
+  "dataset": "EasyCom",
+  "split": {
+    "train_sessions": [7, 11, 10, 6, 9, 4, 8, 5, 12],
+    "test_sessions": [3, 2, 1]
+  },
+  "model": "Qwen2.5-Omni-3B Thinker LoRA",
+  "checkpoint": "checkpoints/omni_turn_lora_easycom_sparse_8s_epoch2_ddp4",
+  "continuation": "adapter_and_head_weights_only_optimizer_and_scheduler_reinitialized",
+  "training_anchor_policy": "transition_sparse_ordinary_1hz_plus_transition_5hz",
+  "evaluation_anchor_policy": "dense_5hz",
+  "prediction_step_seconds": 0.2,
+  "video_fps": 2,
+  "memory_seconds": 8,
+  "metrics_percent": {
+    "epoch2": {
+      "egospeak_full_timeline_mean_map": 54.19850,
+      "causal_valid_only_mean_map": 54.20711,
+      "egospeak_full_timeline_horizon_map": [
+        65.55214,
+        60.96517,
+        56.79766,
+        54.11741,
+        52.09538,
+        51.35516,
+        50.99774,
+        50.47830,
+        50.00999,
+        49.61605
+      ],
+      "causal_valid_only_horizon_map": [
+        65.55385,
+        60.96812,
+        56.80301,
+        54.12457,
+        52.10400,
+        51.36504,
+        51.00914,
+        50.49067,
+        50.02296,
+        49.62970
+      ]
+    },
+    "epoch1_reference": {
+      "egospeak_full_timeline_mean_map": 56.52480,
+      "causal_valid_only_mean_map": 56.53450
+    },
+    "frozen_dense_cache_control": {
+      "egospeak_full_timeline_mean_map": 58.16091,
+      "causal_valid_only_mean_map": 58.17146
+    },
+    "epoch2_minus_epoch1": {
+      "egospeak_full_timeline_mean_map": -2.32630,
+      "causal_valid_only_mean_map": -2.32739
+    },
+    "epoch2_minus_frozen_dense_cache_control": {
+      "egospeak_full_timeline_mean_map": -3.96241,
+      "causal_valid_only_mean_map": -3.96435
+    }
+  },
+  "decision": "stop_same_recipe_continuation_and_test_frozen_overlap_working_memory",
+  "provenance": {
+    "source": "User-reported remote eval_omni_turn_lora.py console tables",
+    "raw_metrics_json_archived_here": false,
+    "multi_seed": false
+  }
+}