Document global-local Omni paper position and oracle evidence

This commit is contained in:
Xin Wang
2026-09-07 14:33:21 +08:00
parent 2632402a70
commit da77c4983a
6 changed files with 254 additions and 121 deletions
+22 -5
View File
@@ -6,9 +6,10 @@ reference; it is not the experiment scheduler.
## One-sentence objective
Determine whether strictly causal, speaker-grounded audiovisual evidence from
Qwen2.5-Omni can improve an invisible egocentric wearer's **floor-entry** and
**floor-yield** behavior in natural multiparty conversation.
Determine whether strictly causal global Omni context plus local audiovisual
floor evidence organized around visible people can improve an invisible
egocentric wearer's **floor-entry** and **floor-yield** behavior in natural
multiparty conversation.
The paper is not simply “replace EgoSpeak's encoders with a newer backbone.”
The decisive test is whether correct person-aligned vision reduces false
@@ -19,6 +20,7 @@ recall, without hurting addressed responses or yield behavior.
| Need | Authoritative file |
|---|---|
| Current paper thesis and claim boundary | [docs/global_local_omni_paper_position_2026-09-07.md](docs/global_local_omni_paper_position_2026-09-07.md) |
| Progressive feature-fusion implementation route | [docs/progressive_feature_fusion_route_2026-09-02.md](docs/progressive_feature_fusion_route_2026-09-02.md) |
| Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) |
| Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) |
@@ -72,6 +74,19 @@ result.
[`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md)
and [`implementation note`](docs/easycom_track_gru_residual.md).
### Measured oracle-information screen
- A seed-0, four-epoch Session-12 development screen supplied transcript-derived
current/past speech activity only through the explicit analysis-only oracle
contract. `aggregate-visible` improved over the matched local-audio baseline
by `+6.96649` points at 0.2 seconds and `+2.91047` points in ten-horizon mean
mAP. Visibility-only and 1-second-delayed controls stayed near audio.
- Person identity continuity is not supported: `aggregate-visible` exceeded
`per-person`, while `per-person` and `shuffled-id` were effectively tied.
The result motivates a learned causal local active-speaker/floor-evidence
branch, but does not justify a track-continuity claim. See the immutable
[`result record`](results/easycom_oracle_information_4ep_2026-09-07/README.md).
### Current blocker
`evaluate_floor_state_changes.py` counts a wearer transition anywhere in a
@@ -91,10 +106,12 @@ unfinished. This does not relax the paper claim: event numbers remain proxy
diagnostics until competing-risk censoring is fixed.
1. **P0:** fix/freeze event attribution, splits and scenario definitions.
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
2. **P1:** confirm the aggregate/per-person oracle screen over seeds 1--2,
then train a causal visible-person active-speaker/floor-evidence estimator;
compare correct, wrong-person and time-shifted visual inputs.
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
4. **P3:** repair the matched anonymous-track-GRU residual optimization by
4. **P3:** only if learned person-organized evidence beats aggregate/no-ID
controls, repair the matched anonymous-track-GRU residual optimization by
replacing the scalar zero gate with a zero-initialized residual output
layer, and verify nonzero branch gradients on Session 12. Add broken-ID,
wrong-time and remove-person controls only if that corrected screen improves.