4.5 KiB
EasyCom causal oracle-information screen (4 epochs, 2026-09-07)
Question
Does perfectly timed, non-wearer speech activity contain information that is missing from the frozen Qwen2.5-Omni global state and the existing recent mixed-audio branch? If so, is the useful information aggregate activity or person-continuous activity?
This is an information upper bound, not a deployable feature condition. The oracle inputs use EasyCom transcript-derived speech labels at or before the current cutoff; no future wearer target, transcript text, target-of-speech, or close-microphone signal is exposed.
Protocol and provenance
- Dataset: EasyCom using the EgoSpeak target and split.
- Model development: Sessions 4--11 for training and Session 12 for checkpoint selection and this screen. Sessions 1--3 were not evaluated.
- Seed: 0 only; three-seed confirmation is pending.
- Training: 4 epochs, LSTR-MCE activity loss, frozen Qwen2.5-Omni-3B global
states,
concat-mlplayer fusion, and the same recent local-audio branch. - Evaluation:
causal_valid_only; AP/mAP values below are percentages. - Oracle history: 10 samples at 5 Hz (2 seconds). The delayed control shifts person activity by 5 samples (1 second).
- Target alignment: EgoSpeak right-edge sampling,
Start_Frame <= 4*(t+1) < End_Frame; all twelve sessions must have zero transcript/target mismatch before any sidecar is written. - Raw run directory on the L20 host:
results/easycom_oracle_information_4ep/.
Results
| Condition | 0.2-s mAP | BG AP | wearer AP | others AP | 10-horizon mean mAP |
|---|---|---|---|---|---|
| audio | 72.57922 | 59.84555 | 66.82464 | 91.06747 | 58.77336 |
| visibility-only | 71.95065 | 57.85640 | 66.73511 | 91.26045 | 58.86491 |
| delayed | 72.02265 | 57.86302 | 66.88550 | 91.31943 | 58.89590 |
| aggregate-visible | 79.54571 | 69.79448 | 72.65278 | 96.18987 | 61.68383 |
| per-person | 78.80474 | 69.74061 | 71.00535 | 95.66825 | 61.10890 |
| shuffled-id | 78.86300 | 68.70055 | 71.85510 | 96.03333 | 61.18615 |
Important paired descriptive deltas for this single seed:
aggregate-visible - audio: +6.96649 at 0.2 seconds and +2.91047 mean mAP.aggregate-visible - delayed: +7.52306 at 0.2 seconds and +2.78793 mean mAP.aggregate-visible - per-person: +0.74097 at 0.2 seconds and +0.57493 mean mAP.per-person - shuffled-id: -0.05826 at 0.2 seconds and -0.07725 mean mAP.visibility-only - audio: -0.62857 at 0.2 seconds and +0.09155 mean mAP.
Interpretation
The positive result is an information-value result: accurately aligned current activity from visible non-wearers supplies substantial information beyond the frozen global Omni state and recent mixed audio. The visibility-only and one-second-delayed controls are close to the audio baseline, so the gain is not explained by the number of slots, extra parameters, or stale activity.
The person-continuity hypothesis is not supported by this run. Aggregate
visible activity is stronger than the person-resolved representation, and
independently shuffling historical activity across current person slots does
not hurt. The present dense aggregate-others objective can exploit per-lag
activity counts without identifying who produced them, and the simpler
aggregate representation is also easier for the head to consume. Therefore
this result cannot support claims of speaker identity, speaker continuity,
addressee understanding, or other-to-other interaction understanding.
The defensible method hypothesis is now global Omni context plus causal local audiovisual floor evidence organized around visible people. A learned causal active-speaker/floor branch should first attempt to recover the aggregate oracle gain. Anonymous track continuity becomes part of the selected method only if correct tracks beat no-ID, broken-continuity, time-shifted, and capacity-matched controls and improve the frozen event protocol. Otherwise the paper must use the simpler global-plus-local-floor-evidence formulation.
Required follow-up
- Repeat the six conditions with seeds 1 and 2 on Session 12.
- Do not use Sessions 1--3 until the architecture and event protocol are frozen.
- Train a causal learned visible-person activity estimator from deployable audio/video inputs; report how much of the oracle gap it recovers.
- Freeze competing-risk-censored entry/yield evaluation. Dense mAP does not demonstrate fewer erroneous agent entries.
- Test person continuity only with matched no-ID/broken-track controls; stop the track claim if they do not separate.