Archive EasyCom sparse supervision results

2026-08-24 12:02:49 +08:00
parent edd37dc36a
commit b7856ece4d
8 changed files with 430 additions and 0 deletions
@@ -0,0 +1,80 @@
+# EasyCom open-stream context: 8 seconds versus 20 seconds
+
+Recorded on **2026-08-23**. This is a single-run experiment snapshot, not a
+final multi-seed paper result.
+
+## Experiment contract
+
+- Dataset: EasyCom with EgoSpeak's official split.
+- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
+- Test sessions: 3, 2, and 1.
+- Representation: frozen Qwen2.5-Omni-3B Thinker, open-stream replay.
+- Head: MLP `2048 -> 1024 -> 30`, reshaped to `10 x 3`.
+- Video sampling: 2 FPS.
+- Prediction rate: 5 Hz.
+- Context budgets: 8 seconds and 20 seconds.
+- Targets: ten future horizons from 0.2 through 2.0 seconds.
+- Loss/evaluation: EgoSpeak LSTR MCE and Table 2 frame AP/mAP protocol.
+
+The heads were trained separately for their corresponding representations. A
+head trained on the 20-second states was not reused on the 8-second states.
+
+## Summary
+
+All values are percentages.
+
+| Protocol | 8 seconds | 20 seconds | 8s minus 20s |
+|---|---:|---:|---:|
+| EgoSpeak full timeline | **58.51937** | 58.316 | **+0.20337** |
+| Strict causal valid only | **58.53015** | 58.326 | **+0.20415** |
+
+The full-timeline per-horizon comparison is:
+
+| Horizon | Time | 8s mAP | 20s mAP | 8s minus 20s |
+|---:|---:|---:|---:|---:|
+| 1 | 0.2s | **71.99768** | 69.756 | **+2.24168** |
+| 2 | 0.4s | **66.74289** | 65.163 | **+1.57989** |
+| 3 | 0.6s | **62.38972** | 61.578 | **+0.81172** |
+| 4 | 0.8s | **59.12229** | 58.783 | **+0.33929** |
+| 5 | 1.0s | 56.71380 | **56.913** | -0.19920 |
+| 6 | 1.2s | 55.36300 | **55.727** | -0.36400 |
+| 7 | 1.4s | 54.21297 | **54.819** | -0.60603 |
+| 8 | 1.6s | 53.43221 | **53.957** | -0.52479 |
+| 9 | 1.8s | 52.84703 | **53.459** | -0.61197 |
+| 10 | 2.0s | 52.37207 | **53.002** | -0.62993 |
+| Mean | — | **58.51937** | 58.316 | **+0.20337** |
+
+For the full-timeline protocol, averaging AP over the ten horizons gives:
+
+| Class | 8s mean AP | 20s mean AP | 8s minus 20s |
+|---|---:|---:|---:|
+| Background | 39.63094 | **40.49350** | -0.86256 |
+| Camera wearer speaking | **58.91206** | 58.59400 | +0.31806 |
+| Other people speaking | **77.01510** | 75.85970 | +1.15540 |
+
+## Interpretation
+
+The aggregate 8-second advantage is only 0.20 mAP, but it is not distributed
+uniformly. The 8-second representation is substantially stronger at 0.2--0.6
+seconds, while the 20-second representation becomes stronger from 1.0 seconds
+onward. This looks like a short/long-horizon tradeoff rather than evidence that
+one context budget is uniformly better.
+
+The class pattern is also mixed: 8 seconds improves the two speaking classes,
+especially other-person speech, but reduces background AP. Because the heads
+were trained once and the aggregate difference is small, training variance
+could plausibly reverse the mean comparison. Identical multi-seed runs are
+required before selecting the context budget for the paper.
+
+## Files and provenance
+
+- `raw_8s_console.txt` preserves the reported 8-second console output.
+- `metrics.csv` contains both contexts, protocols, horizons, and classes.
+- `summary.json` contains the machine-readable aggregate comparison.
+- The 20-second values come from the previously archived
+  `easycom_open_stream_vs_causal_prefix_2026-08-19` snapshot and therefore
+  have only three-decimal precision.
+- Aggregate differences use the separately reported aggregate means. Averaging
+  differences reconstructed from the rounded 20-second horizon rows can vary
+  slightly from those aggregate differences.
+- The original remote `metrics.json` files have not yet been copied here.
@@ -0,0 +1,41 @@
+context_seconds,protocol,horizon,seconds,background_ap_percent,camera_wearer_ap_percent,other_people_ap_percent,map_percent,num_steps
+8,egospeak_full_timeline,1,0.2,54.84736,76.45639,84.68928,71.99768,23048
+8,egospeak_full_timeline,2,0.4,47.46373,71.02340,81.74154,66.74289,23048
+8,egospeak_full_timeline,3,0.6,42.19659,65.57303,79.39953,62.38972,23048
+8,egospeak_full_timeline,4,0.8,38.81557,60.96834,77.58298,59.12229,23048
+8,egospeak_full_timeline,5,1.0,36.79821,57.21128,76.13193,56.71380,23048
+8,egospeak_full_timeline,6,1.2,36.09944,54.87811,75.11144,55.36300,23048
+8,egospeak_full_timeline,7,1.4,35.41214,52.71169,74.51507,54.21297,23048
+8,egospeak_full_timeline,8,1.6,35.15599,51.16932,73.97133,53.43221,23048
+8,egospeak_full_timeline,9,1.8,34.85984,50.01901,73.66226,52.84703,23048
+8,egospeak_full_timeline,10,2.0,34.66053,49.11007,73.34560,52.37207,23048
+8,causal_valid_only,1,0.2,54.84736,76.45639,84.69504,71.99960,23045
+8,causal_valid_only,2,0.4,47.46373,71.02340,81.75175,66.74630,23042
+8,causal_valid_only,3,0.6,42.19659,65.57933,79.41330,62.39641,23039
+8,causal_valid_only,4,0.8,38.81557,60.97945,77.59974,59.13158,23036
+8,causal_valid_only,5,1.0,36.79821,57.22612,76.15127,56.72520,23033
+8,causal_valid_only,6,1.2,36.09944,54.89639,75.13207,55.37597,23030
+8,causal_valid_only,7,1.4,35.41214,52.73278,74.53730,54.22741,23027
+8,causal_valid_only,8,1.6,35.15831,51.18915,73.99506,53.44751,23024
+8,causal_valid_only,9,1.8,34.86210,50.03791,73.68883,52.86295,23021
+8,causal_valid_only,10,2.0,34.66276,49.12823,73.37487,52.38862,23018
+20,egospeak_full_timeline,1,0.2,53.605,74.331,81.331,69.756,23048
+20,egospeak_full_timeline,2,0.4,47.066,69.402,79.020,65.163,23048
+20,egospeak_full_timeline,3,0.6,42.615,64.600,77.520,61.578,23048
+20,egospeak_full_timeline,4,0.8,39.463,60.641,76.244,58.783,23048
+20,egospeak_full_timeline,5,1.0,38.002,57.371,75.367,56.913,23048
+20,egospeak_full_timeline,6,1.2,37.398,55.258,74.523,55.727,23048
+20,egospeak_full_timeline,7,1.4,36.933,53.400,74.126,54.819,23048
+20,egospeak_full_timeline,8,1.6,36.653,51.502,73.716,53.957,23048
+20,egospeak_full_timeline,9,1.8,36.527,50.346,73.504,53.459,23048
+20,egospeak_full_timeline,10,2.0,36.673,49.089,73.246,53.002,23048
+20,causal_valid_only,1,0.2,53.605,74.331,81.336,69.757,23045
+20,causal_valid_only,2,0.4,47.066,69.402,79.029,65.166,23042
+20,causal_valid_only,3,0.6,42.615,64.607,77.532,61.585,23039
+20,causal_valid_only,4,0.8,39.463,60.652,76.259,58.792,23036
+20,causal_valid_only,5,1.0,38.002,57.386,75.385,56.924,23033
+20,causal_valid_only,6,1.2,37.398,55.277,74.543,55.739,23030
+20,causal_valid_only,7,1.4,36.933,53.421,74.148,54.834,23027
+20,causal_valid_only,8,1.6,36.656,51.522,73.740,53.973,23024
+20,causal_valid_only,9,1.8,36.529,50.365,73.530,53.475,23021
+20,causal_valid_only,10,2.0,36.676,49.107,73.275,53.019,23018
@@ -0,0 +1,45 @@
+egospeak_full_timeline:
+Action anticipation (0.20s) perframe mAP: 71.99768
+  AP: background=54.84736, camera_wearer_speaking=76.45639, other_people_speaking=84.68928; n=23048
+Action anticipation (0.40s) perframe mAP: 66.74289
+  AP: background=47.46373, camera_wearer_speaking=71.02340, other_people_speaking=81.74154; n=23048
+Action anticipation (0.60s) perframe mAP: 62.38972
+  AP: background=42.19659, camera_wearer_speaking=65.57303, other_people_speaking=79.39953; n=23048
+Action anticipation (0.80s) perframe mAP: 59.12229
+  AP: background=38.81557, camera_wearer_speaking=60.96834, other_people_speaking=77.58298; n=23048
+Action anticipation (1.00s) perframe mAP: 56.71380
+  AP: background=36.79821, camera_wearer_speaking=57.21128, other_people_speaking=76.13193; n=23048
+Action anticipation (1.20s) perframe mAP: 55.36300
+  AP: background=36.09944, camera_wearer_speaking=54.87811, other_people_speaking=75.11144; n=23048
+Action anticipation (1.40s) perframe mAP: 54.21297
+  AP: background=35.41214, camera_wearer_speaking=52.71169, other_people_speaking=74.51507; n=23048
+Action anticipation (1.60s) perframe mAP: 53.43221
+  AP: background=35.15599, camera_wearer_speaking=51.16932, other_people_speaking=73.97133; n=23048
+Action anticipation (1.80s) perframe mAP: 52.84703
+  AP: background=34.85984, camera_wearer_speaking=50.01901, other_people_speaking=73.66226; n=23048
+Action anticipation (2.00s) perframe mAP: 52.37207
+  AP: background=34.66053, camera_wearer_speaking=49.11007, other_people_speaking=73.34560; n=23048
+Action anticipation (mean) perframe mAP: 58.51937
+
+causal_valid_only:
+Action anticipation (0.20s) perframe mAP: 71.99960
+  AP: background=54.84736, camera_wearer_speaking=76.45639, other_people_speaking=84.69504; n=23045
+Action anticipation (0.40s) perframe mAP: 66.74630
+  AP: background=47.46373, camera_wearer_speaking=71.02340, other_people_speaking=81.75175; n=23042
+Action anticipation (0.60s) perframe mAP: 62.39641
+  AP: background=42.19659, camera_wearer_speaking=65.57933, other_people_speaking=79.41330; n=23039
+Action anticipation (0.80s) perframe mAP: 59.13158
+  AP: background=38.81557, camera_wearer_speaking=60.97945, other_people_speaking=77.59974; n=23036
+Action anticipation (1.00s) perframe mAP: 56.72520
+  AP: background=36.79821, camera_wearer_speaking=57.22612, other_people_speaking=76.15127; n=23033
+Action anticipation (1.20s) perframe mAP: 55.37597
+  AP: background=36.09944, camera_wearer_speaking=54.89639, other_people_speaking=75.13207; n=23030
+Action anticipation (1.40s) perframe mAP: 54.22741
+  AP: background=35.41214, camera_wearer_speaking=52.73278, other_people_speaking=74.53730; n=23027
+Action anticipation (1.60s) perframe mAP: 53.44751
+  AP: background=35.15831, camera_wearer_speaking=51.18915, other_people_speaking=73.99506; n=23024
+Action anticipation (1.80s) perframe mAP: 52.86295
+  AP: background=34.86210, camera_wearer_speaking=50.03791, other_people_speaking=73.68883; n=23021
+Action anticipation (2.00s) perframe mAP: 52.38862
+  AP: background=34.66276, camera_wearer_speaking=49.12823, other_people_speaking=73.37487; n=23018
+Action anticipation (mean) perframe mAP: 58.53015
@@ -0,0 +1,76 @@
+{
+  "schema_version": 1,
+  "recorded_date": "2026-08-23",
+  "status": "single_run_snapshot",
+  "dataset": "EasyCom",
+  "representation": "Qwen2.5-Omni-3B open-stream replay",
+  "video_fps": 2,
+  "prediction_step_seconds": 0.2,
+  "split": {
+    "train_sessions": [7, 11, 10, 6, 9, 4, 8, 5, 12],
+    "test_sessions": [3, 2, 1]
+  },
+  "metrics_percent": {
+    "8_seconds": {
+      "egospeak_full_timeline_mean_map": 58.51937,
+      "causal_valid_only_mean_map": 58.53015,
+      "egospeak_full_timeline_horizon_map": [
+        71.99768,
+        66.74289,
+        62.38972,
+        59.12229,
+        56.71380,
+        55.36300,
+        54.21297,
+        53.43221,
+        52.84703,
+        52.37207
+      ]
+    },
+    "20_seconds": {
+      "egospeak_full_timeline_mean_map": 58.316,
+      "causal_valid_only_mean_map": 58.326,
+      "egospeak_full_timeline_horizon_map": [
+        69.756,
+        65.163,
+        61.578,
+        58.783,
+        56.913,
+        55.727,
+        54.819,
+        53.957,
+        53.459,
+        53.002
+      ]
+    },
+    "8_seconds_minus_20_seconds": {
+      "egospeak_full_timeline_mean_map": 0.20337,
+      "causal_valid_only_mean_map": 0.20415,
+      "egospeak_full_timeline_horizon_map": [
+        2.24168,
+        1.57989,
+        0.81172,
+        0.33929,
+        -0.19920,
+        -0.36400,
+        -0.60603,
+        -0.52479,
+        -0.61197,
+        -0.62993
+      ],
+      "egospeak_full_timeline_mean_class_ap": {
+        "background": -0.86256,
+        "camera_wearer_speaking": 0.31806,
+        "other_people_speaking": 1.15540
+      }
+    }
+  },
+  "provenance": {
+    "8_seconds": "User-reported infer_turn_head.py console output; preserved in raw_8s_console.txt",
+    "20_seconds": "Previously archived 2026-08-19 open-stream snapshot; values available to three decimals",
+    "precision_note": "Aggregate differences use the separately reported aggregate means. Per-horizon and per-class differences use the available row values and can differ slightly when averaged because the archived 20-second rows were rounded to three decimals.",
+    "raw_8_seconds_metrics_json_archived_here": false,
+    "raw_20_seconds_metrics_json_archived_here": false,
+    "multi_seed": false
+  }
+}