Add competing-risk evaluation and align ICASSP story

2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
@@ -0,0 +1,123 @@
+# EasyCom frozen-ROMA last-four-layer readout study
+
+Recorded on **2026-08-27** from three user-reported remote evaluations. These
+are single-seed diagnostic results on official EasyCom test Sessions 1--3, not
+test-independent paper numbers.
+
+## Shared experiment contract
+
+- Dataset: EasyCom; EgoSpeak full-timeline evaluator on Sessions 1--3.
+- Backbone checkpoint: ROMA, based on Qwen2.5-Omni-7B.
+- Backbone use: frozen representation extraction; no end-to-end ROMA update.
+- Streaming path: this project's 8-second Qwen-style open-stream replay, 2 FPS
+  video, one prediction every 200 ms.
+- Saved state: final four query states, shape `[T, 4, 3584]`.
+- Prediction target: ten future horizons from 0.2 to 2.0 seconds, three
+  EgoSpeak AP classes.
+
+ROMA's released model is trained with one-second aligned streaming units, but
+these features were extracted with this project's matched Qwen open-stream
+protocol. The result therefore tests the **frozen ROMA representation under
+our protocol**; it does not reproduce ROMA's own streaming training recipe.
+
+## Main result
+
+| Readout | Epochs | Approx. fusion parameters | 0.2s mAP | Mean mAP |
+|---|---:|---:|---:|---:|
+| Last-4 concat-MLP | 1 | 51.4M | 68.27145 | 58.89219 |
+| Last-4 concat-MLP | 4 | 51.4M | 68.65486 | 56.52425 |
+| Last-4 fixed mean | 4 | 0 | **69.21863** | **58.90402** |
+
+The fixed uniform mean is the strongest ROMA readout. It is effectively tied
+with the one-epoch concat model in mean mAP (`+0.01183`) while gaining
+`+0.94718` at 0.2 seconds. The 51.4M-parameter fusion is unnecessary here.
+
+## Per-horizon mAP
+
+| Horizon | Concat 1ep | Concat 4ep | Fixed mean 4ep |
+|---:|---:|---:|---:|
+| 0.2s | 68.27145 | 68.65486 | **69.21863** |
+| 0.4s | 65.29126 | 64.32941 | **65.67454** |
+| 0.6s | 62.15965 | 60.41845 | **62.52327** |
+| 0.8s | **59.95476** | 57.35198 | 59.92499 |
+| 1.0s | **58.25407** | 55.17851 | 58.14859 |
+| 1.2s | **56.83849** | 53.70716 | 56.61984 |
+| 1.4s | 55.65764 | 52.59370 | **55.68495** |
+| 1.6s | **54.91836** | 51.83772 | 54.52641 |
+| 1.8s | **54.20725** | 50.70519 | 53.67463 |
+| 2.0s | **53.36895** | 50.46558 | 53.04431 |
+| **Mean** | 58.89219 | 56.52425 | **58.90402** |
+
+## Mean class AP over ten horizons
+
+| Readout | Background | Camera wearer | Other people |
+|---|---:|---:|---:|
+| Concat 1ep | 42.17735 | **57.38537** | 77.11384 |
+| Concat 4ep | 40.19574 | 52.75190 | 76.62513 |
+| Fixed mean 4ep | **42.19246** | 57.26979 | **77.24980** |
+
+Extending concat training from one to four epochs raises 0.2-second mAP by
+only `+0.38341` but reduces mean mAP by `-2.36794`. Camera-wearer mean AP drops
+by `-4.63347`, much more than background (`-1.98161`) or other people
+(`-0.48871`). This is the same overfitting pattern observed with the 3B
+concat-MLP, amplified by ROMA's wider 3584-dimensional state.
+
+Replacing concat-MLP with a fixed mean at four epochs restores `+4.51789`
+camera-wearer AP and `+2.37977` mean mAP. This isolates readout capacity as the
+proximate cause of the four-epoch collapse.
+
+## Comparison with the strongest frozen Qwen2.5-Omni-3B controls
+
+| Frozen representation/readout | 0.2s mAP | Mean mAP |
+|---|---:|---:|
+| Qwen-3B final-layer query, 8s | 71.99768 | 58.51937 |
+| Qwen-3B last-4 scalar mixture | **73.45085** | 60.53423 |
+| Qwen-3B last-4 concat-MLP, 4ep | 73.42020 | **60.87144** |
+| ROMA last-4 fixed mean, 4ep | 69.21863 | 58.90402 |
+| ROMA last-4 concat-MLP, 4ep | 68.65486 | 56.52425 |
+
+The same concat design and epoch count gives a strong diagnostic: ROMA is
+`-4.76534` at 0.2 seconds and `-4.34719` in mean mAP relative to Qwen-3B.
+It is not a parameter-matched attribution test, because the 3584-dimensional
+ROMA fusion has 51.4M parameters versus about 16.8M for Qwen-3B. Together with
+the fixed-mean result, it shows that ROMA's proactive streaming adaptation is
+not automatically useful for 5 Hz egocentric future-speaker anticipation, but
+does not yet isolate checkpoint adaptation from width and protocol.
+
+The fixed-mean ROMA result is not directly architecture-matched to the 3B
+scalar-mixture/concat models. An exact Qwen-3B fixed-mean four-epoch control is
+still required. A second clean attribution test is ROMA versus its original
+Qwen2.5-Omni-7B base using the same last-four stack and fixed-mean head.
+
+## Interpretation and decision
+
+1. **Do not continue tuning the ROMA concat head.** Its capacity is much too
+   high for nine EasyCom training conversations.
+2. **Do not conclude that ROMA itself is universally worse.** The experiment
+   changes checkpoint family and uses a protocol different from ROMA's native
+   one-second training units.
+3. **The practical EasyCom backbone remains frozen Qwen2.5-Omni-3B.** Its
+   last-four readouts are stronger, cheaper, and already validated throughout
+   the current pipeline.
+4. **The next method should change readout structure and supervision.** The
+   strongest hypothesis is a low-capacity semantic query state fused with
+   recent 200 ms payload tokens, followed by a structured future-activity
+   projection rather than ten independent horizon classifiers.
+
+## Validity caveats
+
+- Only user-reported full-timeline metrics were available; strict
+  causal-valid tables and remote `metrics.json` files were not archived.
+- Results are single-seed and Sessions 1--3 were inspected repeatedly.
+- Exact optimizer settings and the remote checkpoint filenames for all three
+  runs were not supplied here.
+- The train split is assumed to follow the project's Sessions 4--12 contract;
+  verify each remote command before using the values in a paper table.
+- ROMA's official one-second streaming-unit path was not reproduced.
+
+Machine-readable values are in [`summary.json`](summary.json), and a formatted
+workbook with per-horizon metrics and comparison deltas is in
+[`metrics.xlsx`](metrics.xlsx).
+
+The literature-backed diagnosis and proposed ICASSP research plan are in
+[`reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md`](../../reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md).
@@ -0,0 +1,121 @@
+{
+  "schema_version": 1,
+  "recorded_date": "2026-08-27",
+  "status": "single_seed_user_reported_test_inspected_diagnostic",
+  "dataset": "EasyCom",
+  "evaluation": {
+    "protocol": "EgoSpeak full timeline",
+    "test_sessions": [1, 2, 3],
+    "prediction_hz": 5,
+    "future_horizons_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0],
+    "classes": ["background", "camera_wearer_speaking", "other_people_speaking"],
+    "n": 23048
+  },
+  "representation": {
+    "checkpoint_family": "ROMA based on Qwen2.5-Omni-7B",
+    "thinker_frozen": true,
+    "streaming_mode": "open-stream-replay",
+    "memory_seconds": 8,
+    "video_fps": 2,
+    "state": "last_four_query_states",
+    "state_shape": [4, 3584],
+    "native_roma_one_second_streaming_recipe_reproduced": false
+  },
+  "runs": [
+    {
+      "id": "roma_last4_concat_mlp_1ep",
+      "layer_fusion": "concat-mlp",
+      "epochs": 1,
+      "fusion_parameters_approx": 51412480,
+      "reported_mean_map": 58.89219,
+      "mean_class_ap": {
+        "background": 42.17735,
+        "camera_wearer_speaking": 57.38537,
+        "other_people_speaking": 77.11384
+      },
+      "per_horizon": [
+        {"seconds": 0.2, "background": 52.51021, "camera_wearer_speaking": 70.83688, "other_people_speaking": 81.46725, "map": 68.27145},
+        {"seconds": 0.4, "background": 48.71387, "camera_wearer_speaking": 67.22686, "other_people_speaking": 79.93305, "map": 65.29126},
+        {"seconds": 0.6, "background": 45.13005, "camera_wearer_speaking": 62.85818, "other_people_speaking": 78.49071, "map": 62.15965},
+        {"seconds": 0.8, "background": 42.40835, "camera_wearer_speaking": 59.85171, "other_people_speaking": 77.60421, "map": 59.95476},
+        {"seconds": 1.0, "background": 41.04191, "camera_wearer_speaking": 56.82736, "other_people_speaking": 76.89293, "map": 58.25407},
+        {"seconds": 1.2, "background": 39.74672, "camera_wearer_speaking": 54.55271, "other_people_speaking": 76.21605, "map": 56.83849},
+        {"seconds": 1.4, "background": 38.80110, "camera_wearer_speaking": 52.54133, "other_people_speaking": 75.63049, "map": 55.65764},
+        {"seconds": 1.6, "background": 38.10108, "camera_wearer_speaking": 51.26856, "other_people_speaking": 75.38544, "map": 54.91836},
+        {"seconds": 1.8, "background": 37.64821, "camera_wearer_speaking": 49.94810, "other_people_speaking": 75.02545, "map": 54.20725},
+        {"seconds": 2.0, "background": 37.67202, "camera_wearer_speaking": 47.94197, "other_people_speaking": 74.49285, "map": 53.36895}
+      ]
+    },
+    {
+      "id": "roma_last4_concat_mlp_4ep",
+      "layer_fusion": "concat-mlp",
+      "epochs": 4,
+      "fusion_parameters_approx": 51412480,
+      "reported_mean_map": 56.52425,
+      "mean_class_ap": {
+        "background": 40.19574,
+        "camera_wearer_speaking": 52.75190,
+        "other_people_speaking": 76.62513
+      },
+      "per_horizon": [
+        {"seconds": 0.2, "background": 52.12125, "camera_wearer_speaking": 71.51813, "other_people_speaking": 82.32518, "map": 68.65486},
+        {"seconds": 0.4, "background": 47.23227, "camera_wearer_speaking": 65.34609, "other_people_speaking": 80.40987, "map": 64.32941},
+        {"seconds": 0.6, "background": 42.97315, "camera_wearer_speaking": 59.62350, "other_people_speaking": 78.65869, "map": 60.41845},
+        {"seconds": 0.8, "background": 39.98155, "camera_wearer_speaking": 54.81910, "other_people_speaking": 77.25528, "map": 57.35198},
+        {"seconds": 1.0, "background": 38.30512, "camera_wearer_speaking": 50.99168, "other_people_speaking": 76.23874, "map": 55.17851},
+        {"seconds": 1.2, "background": 37.25808, "camera_wearer_speaking": 48.42603, "other_people_speaking": 75.43737, "map": 53.70716},
+        {"seconds": 1.4, "background": 36.84681, "camera_wearer_speaking": 46.09807, "other_people_speaking": 74.83621, "map": 52.59370},
+        {"seconds": 1.6, "background": 36.30368, "camera_wearer_speaking": 44.70198, "other_people_speaking": 74.50749, "map": 51.83772},
+        {"seconds": 1.8, "background": 35.37777, "camera_wearer_speaking": 43.20047, "other_people_speaking": 73.53734, "map": 50.70519},
+        {"seconds": 2.0, "background": 35.55772, "camera_wearer_speaking": 42.79393, "other_people_speaking": 73.04510, "map": 50.46558}
+      ]
+    },
+    {
+      "id": "roma_last4_fixed_mean_4ep",
+      "layer_fusion": "fixed-mean",
+      "epochs": 4,
+      "fusion_parameters_approx": 0,
+      "reported_mean_map": 58.90402,
+      "mean_class_ap": {
+        "background": 42.19246,
+        "camera_wearer_speaking": 57.26979,
+        "other_people_speaking": 77.24980
+      },
+      "per_horizon": [
+        {"seconds": 0.2, "background": 53.55709, "camera_wearer_speaking": 72.09031, "other_people_speaking": 82.00848, "map": 69.21863},
+        {"seconds": 0.4, "background": 49.15182, "camera_wearer_speaking": 67.68575, "other_people_speaking": 80.18605, "map": 65.67454},
+        {"seconds": 0.6, "background": 45.55752, "camera_wearer_speaking": 63.19862, "other_people_speaking": 78.81368, "map": 62.52327},
+        {"seconds": 0.8, "background": 42.39609, "camera_wearer_speaking": 59.81524, "other_people_speaking": 77.56363, "map": 59.92499},
+        {"seconds": 1.0, "background": 41.02210, "camera_wearer_speaking": 56.50674, "other_people_speaking": 76.91692, "map": 58.14859},
+        {"seconds": 1.2, "background": 39.45131, "camera_wearer_speaking": 53.93851, "other_people_speaking": 76.46970, "map": 56.61984},
+        {"seconds": 1.4, "background": 38.78924, "camera_wearer_speaking": 52.30126, "other_people_speaking": 75.96435, "map": 55.68495},
+        {"seconds": 1.6, "background": 38.08597, "camera_wearer_speaking": 50.22392, "other_people_speaking": 75.26934, "map": 54.52641},
+        {"seconds": 1.8, "background": 37.22956, "camera_wearer_speaking": 49.02115, "other_people_speaking": 74.77319, "map": 53.67463},
+        {"seconds": 2.0, "background": 36.68386, "camera_wearer_speaking": 47.91641, "other_people_speaking": 74.53267, "map": 53.04431}
+      ]
+    }
+  ],
+  "comparisons_percent": {
+    "concat_4ep_minus_concat_1ep_horizon_0.2_map": 0.38341,
+    "concat_4ep_minus_concat_1ep_mean_map": -2.36794,
+    "fixed_mean_4ep_minus_concat_4ep_horizon_0.2_map": 0.56377,
+    "fixed_mean_4ep_minus_concat_4ep_mean_map": 2.37977,
+    "fixed_mean_4ep_minus_concat_1ep_horizon_0.2_map": 0.94718,
+    "fixed_mean_4ep_minus_concat_1ep_mean_map": 0.01183,
+    "roma_concat_4ep_minus_qwen3b_concat_4ep_horizon_0.2_map": -4.76534,
+    "roma_concat_4ep_minus_qwen3b_concat_4ep_mean_map": -4.34719
+  },
+  "validity": {
+    "single_seed": true,
+    "test_sessions_repeatedly_inspected": true,
+    "strict_causal_valid_reported": false,
+    "remote_metrics_json_archived": false,
+    "exact_training_commands_archived": false,
+    "train_sessions_assumed_from_project_protocol": [4, 5, 6, 7, 8, 9, 10, 11, 12]
+  },
+  "provenance": {
+    "source": "user-reported remote console tables",
+    "recorded_by": "Codex",
+    "values_are_percent": true
+  }
+}