81 lines
3.4 KiB
Markdown
81 lines
3.4 KiB
Markdown
|
# EasyCom open-stream context: 8 seconds versus 20 seconds
|
||
|
|
|
||
|
|
Recorded on **2026-08-23**. This is a single-run experiment snapshot, not a
|
||
|
|
final multi-seed paper result.
|
||
|
|
|
||
|
|
## Experiment contract
|
||
|
|
|
||
|
|
- Dataset: EasyCom with EgoSpeak's official split.
|
||
|
|
- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
|
||
|
|
- Test sessions: 3, 2, and 1.
|
||
|
|
- Representation: frozen Qwen2.5-Omni-3B Thinker, open-stream replay.
|
||
|
|
- Head: MLP `2048 -> 1024 -> 30`, reshaped to `10 x 3`.
|
||
|
|
- Video sampling: 2 FPS.
|
||
|
|
- Prediction rate: 5 Hz.
|
||
|
|
- Context budgets: 8 seconds and 20 seconds.
|
||
|
|
- Targets: ten future horizons from 0.2 through 2.0 seconds.
|
||
|
|
- Loss/evaluation: EgoSpeak LSTR MCE and Table 2 frame AP/mAP protocol.
|
||
|
|
|
||
|
|
The heads were trained separately for their corresponding representations. A
|
||
|
|
head trained on the 20-second states was not reused on the 8-second states.
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
All values are percentages.
|
||
|
|
|
||
|
|
| Protocol | 8 seconds | 20 seconds | 8s minus 20s |
|
||
|
|
|---|---:|---:|---:|
|
||
|
|
| EgoSpeak full timeline | **58.51937** | 58.316 | **+0.20337** |
|
||
|
|
| Strict causal valid only | **58.53015** | 58.326 | **+0.20415** |
|
||
|
|
|
||
|
|
The full-timeline per-horizon comparison is:
|
||
|
|
|
||
|
|
| Horizon | Time | 8s mAP | 20s mAP | 8s minus 20s |
|
||
|
|
|---:|---:|---:|---:|---:|
|
||
|
|
| 1 | 0.2s | **71.99768** | 69.756 | **+2.24168** |
|
||
|
|
| 2 | 0.4s | **66.74289** | 65.163 | **+1.57989** |
|
||
|
|
| 3 | 0.6s | **62.38972** | 61.578 | **+0.81172** |
|
||
|
|
| 4 | 0.8s | **59.12229** | 58.783 | **+0.33929** |
|
||
|
|
| 5 | 1.0s | 56.71380 | **56.913** | -0.19920 |
|
||
|
|
| 6 | 1.2s | 55.36300 | **55.727** | -0.36400 |
|
||
|
|
| 7 | 1.4s | 54.21297 | **54.819** | -0.60603 |
|
||
|
|
| 8 | 1.6s | 53.43221 | **53.957** | -0.52479 |
|
||
|
|
| 9 | 1.8s | 52.84703 | **53.459** | -0.61197 |
|
||
|
|
| 10 | 2.0s | 52.37207 | **53.002** | -0.62993 |
|
||
|
|
| Mean | — | **58.51937** | 58.316 | **+0.20337** |
|
||
|
|
|
||
|
|
For the full-timeline protocol, averaging AP over the ten horizons gives:
|
||
|
|
|
||
|
|
| Class | 8s mean AP | 20s mean AP | 8s minus 20s |
|
||
|
|
|---|---:|---:|---:|
|
||
|
|
| Background | 39.63094 | **40.49350** | -0.86256 |
|
||
|
|
| Camera wearer speaking | **58.91206** | 58.59400 | +0.31806 |
|
||
|
|
| Other people speaking | **77.01510** | 75.85970 | +1.15540 |
|
||
|
|
|
||
|
|
## Interpretation
|
||
|
|
|
||
|
|
The aggregate 8-second advantage is only 0.20 mAP, but it is not distributed
|
||
|
|
uniformly. The 8-second representation is substantially stronger at 0.2--0.6
|
||
|
|
seconds, while the 20-second representation becomes stronger from 1.0 seconds
|
||
|
|
onward. This looks like a short/long-horizon tradeoff rather than evidence that
|
||
|
|
one context budget is uniformly better.
|
||
|
|
|
||
|
|
The class pattern is also mixed: 8 seconds improves the two speaking classes,
|
||
|
|
especially other-person speech, but reduces background AP. Because the heads
|
||
|
|
were trained once and the aggregate difference is small, training variance
|
||
|
|
could plausibly reverse the mean comparison. Identical multi-seed runs are
|
||
|
|
required before selecting the context budget for the paper.
|
||
|
|
|
||
|
|
## Files and provenance
|
||
|
|
|
||
|
|
- `raw_8s_console.txt` preserves the reported 8-second console output.
|
||
|
|
- `metrics.csv` contains both contexts, protocols, horizons, and classes.
|
||
|
|
- `summary.json` contains the machine-readable aggregate comparison.
|
||
|
|
- The 20-second values come from the previously archived
|
||
|
|
`easycom_open_stream_vs_causal_prefix_2026-08-19` snapshot and therefore
|
||
|
|
have only three-decimal precision.
|
||
|
|
- Aggregate differences use the separately reported aggregate means. Averaging
|
||
|
|
differences reconstructed from the rounded 20-second horizon rows can vary
|
||
|
|
slightly from those aggregate differences.
|
||
|
|
- The original remote `metrics.json` files have not yet been copied here.
|