|
|
|
@@ -0,0 +1,86 @@
|
|
|
|
|
# EasyCom causal oracle-information screen (4 epochs, 2026-09-07)
|
|
|
|
|
|
|
|
|
|
## Question
|
|
|
|
|
|
|
|
|
|
Does perfectly timed, non-wearer speech activity contain information that is
|
|
|
|
|
missing from the frozen Qwen2.5-Omni global state and the existing recent
|
|
|
|
|
mixed-audio branch? If so, is the useful information aggregate activity or
|
|
|
|
|
person-continuous activity?
|
|
|
|
|
|
|
|
|
|
This is an information upper bound, not a deployable feature condition. The
|
|
|
|
|
oracle inputs use EasyCom transcript-derived speech labels at or before the
|
|
|
|
|
current cutoff; no future wearer target, transcript text, target-of-speech, or
|
|
|
|
|
close-microphone signal is exposed.
|
|
|
|
|
|
|
|
|
|
## Protocol and provenance
|
|
|
|
|
|
|
|
|
|
- Dataset: EasyCom using the EgoSpeak target and split.
|
|
|
|
|
- Model development: Sessions 4--11 for training and Session 12 for checkpoint
|
|
|
|
|
selection and this screen. Sessions 1--3 were not evaluated.
|
|
|
|
|
- Seed: 0 only; three-seed confirmation is pending.
|
|
|
|
|
- Training: 4 epochs, LSTR-MCE activity loss, frozen Qwen2.5-Omni-3B global
|
|
|
|
|
states, `concat-mlp` layer fusion, and the same recent local-audio branch.
|
|
|
|
|
- Evaluation: `causal_valid_only`; AP/mAP values below are percentages.
|
|
|
|
|
- Oracle history: 10 samples at 5 Hz (2 seconds). The delayed control shifts
|
|
|
|
|
person activity by 5 samples (1 second).
|
|
|
|
|
- Target alignment: EgoSpeak right-edge sampling,
|
|
|
|
|
`Start_Frame <= 4*(t+1) < End_Frame`; all twelve sessions must have zero
|
|
|
|
|
transcript/target mismatch before any sidecar is written.
|
|
|
|
|
- Raw run directory on the L20 host:
|
|
|
|
|
`results/easycom_oracle_information_4ep/`.
|
|
|
|
|
|
|
|
|
|
## Results
|
|
|
|
|
|
|
|
|
|
| Condition | 0.2-s mAP | BG AP | wearer AP | others AP | 10-horizon mean mAP |
|
|
|
|
|
|---|---:|---:|---:|---:|---:|
|
|
|
|
|
| audio | 72.57922 | 59.84555 | 66.82464 | 91.06747 | 58.77336 |
|
|
|
|
|
| visibility-only | 71.95065 | 57.85640 | 66.73511 | 91.26045 | 58.86491 |
|
|
|
|
|
| delayed | 72.02265 | 57.86302 | 66.88550 | 91.31943 | 58.89590 |
|
|
|
|
|
| aggregate-visible | **79.54571** | **69.79448** | **72.65278** | **96.18987** | **61.68383** |
|
|
|
|
|
| per-person | 78.80474 | 69.74061 | 71.00535 | 95.66825 | 61.10890 |
|
|
|
|
|
| shuffled-id | 78.86300 | 68.70055 | 71.85510 | 96.03333 | 61.18615 |
|
|
|
|
|
|
|
|
|
|
Important paired descriptive deltas for this single seed:
|
|
|
|
|
|
|
|
|
|
- `aggregate-visible - audio`: +6.96649 at 0.2 seconds and +2.91047 mean mAP.
|
|
|
|
|
- `aggregate-visible - delayed`: +7.52306 at 0.2 seconds and +2.78793 mean mAP.
|
|
|
|
|
- `aggregate-visible - per-person`: +0.74097 at 0.2 seconds and +0.57493 mean mAP.
|
|
|
|
|
- `per-person - shuffled-id`: -0.05826 at 0.2 seconds and -0.07725 mean mAP.
|
|
|
|
|
- `visibility-only - audio`: -0.62857 at 0.2 seconds and +0.09155 mean mAP.
|
|
|
|
|
|
|
|
|
|
## Interpretation
|
|
|
|
|
|
|
|
|
|
The positive result is an information-value result: accurately aligned current
|
|
|
|
|
activity from visible non-wearers supplies substantial information beyond the
|
|
|
|
|
frozen global Omni state and recent mixed audio. The visibility-only and
|
|
|
|
|
one-second-delayed controls are close to the audio baseline, so the gain is
|
|
|
|
|
not explained by the number of slots, extra parameters, or stale activity.
|
|
|
|
|
|
|
|
|
|
The person-continuity hypothesis is not supported by this run. Aggregate
|
|
|
|
|
visible activity is stronger than the person-resolved representation, and
|
|
|
|
|
independently shuffling historical activity across current person slots does
|
|
|
|
|
not hurt. The present dense aggregate-`others` objective can exploit per-lag
|
|
|
|
|
activity counts without identifying who produced them, and the simpler
|
|
|
|
|
aggregate representation is also easier for the head to consume. Therefore
|
|
|
|
|
this result cannot support claims of speaker identity, speaker continuity,
|
|
|
|
|
addressee understanding, or other-to-other interaction understanding.
|
|
|
|
|
|
|
|
|
|
The defensible method hypothesis is now **global Omni context plus causal local
|
|
|
|
|
audiovisual floor evidence organized around visible people**. A learned causal
|
|
|
|
|
active-speaker/floor branch should first attempt to recover the aggregate
|
|
|
|
|
oracle gain. Anonymous track continuity becomes part of the selected method
|
|
|
|
|
only if correct tracks beat no-ID, broken-continuity, time-shifted, and
|
|
|
|
|
capacity-matched controls and improve the frozen event protocol. Otherwise the
|
|
|
|
|
paper must use the simpler global-plus-local-floor-evidence formulation.
|
|
|
|
|
|
|
|
|
|
## Required follow-up
|
|
|
|
|
|
|
|
|
|
1. Repeat the six conditions with seeds 1 and 2 on Session 12.
|
|
|
|
|
2. Do not use Sessions 1--3 until the architecture and event protocol are
|
|
|
|
|
frozen.
|
|
|
|
|
3. Train a causal learned visible-person activity estimator from deployable
|
|
|
|
|
audio/video inputs; report how much of the oracle gap it recovers.
|
|
|
|
|
4. Freeze competing-risk-censored entry/yield evaluation. Dense mAP does not
|
|
|
|
|
demonstrate fewer erroneous agent entries.
|
|
|
|
|
5. Test person continuity only with matched no-ID/broken-track controls; stop
|
|
|
|
|
the track claim if they do not separate.
|
Xin Wang