Document global-local Omni paper position and oracle evidence
This commit is contained in:
+22
-5
@@ -6,9 +6,10 @@ reference; it is not the experiment scheduler.
|
||||
|
||||
## One-sentence objective
|
||||
|
||||
Determine whether strictly causal, speaker-grounded audiovisual evidence from
|
||||
Qwen2.5-Omni can improve an invisible egocentric wearer's **floor-entry** and
|
||||
**floor-yield** behavior in natural multiparty conversation.
|
||||
Determine whether strictly causal global Omni context plus local audiovisual
|
||||
floor evidence organized around visible people can improve an invisible
|
||||
egocentric wearer's **floor-entry** and **floor-yield** behavior in natural
|
||||
multiparty conversation.
|
||||
|
||||
The paper is not simply “replace EgoSpeak's encoders with a newer backbone.”
|
||||
The decisive test is whether correct person-aligned vision reduces false
|
||||
@@ -19,6 +20,7 @@ recall, without hurting addressed responses or yield behavior.
|
||||
|
||||
| Need | Authoritative file |
|
||||
|---|---|
|
||||
| Current paper thesis and claim boundary | [docs/global_local_omni_paper_position_2026-09-07.md](docs/global_local_omni_paper_position_2026-09-07.md) |
|
||||
| Progressive feature-fusion implementation route | [docs/progressive_feature_fusion_route_2026-09-02.md](docs/progressive_feature_fusion_route_2026-09-02.md) |
|
||||
| Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) |
|
||||
| Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) |
|
||||
@@ -72,6 +74,19 @@ result.
|
||||
[`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md)
|
||||
and [`implementation note`](docs/easycom_track_gru_residual.md).
|
||||
|
||||
### Measured oracle-information screen
|
||||
|
||||
- A seed-0, four-epoch Session-12 development screen supplied transcript-derived
|
||||
current/past speech activity only through the explicit analysis-only oracle
|
||||
contract. `aggregate-visible` improved over the matched local-audio baseline
|
||||
by `+6.96649` points at 0.2 seconds and `+2.91047` points in ten-horizon mean
|
||||
mAP. Visibility-only and 1-second-delayed controls stayed near audio.
|
||||
- Person identity continuity is not supported: `aggregate-visible` exceeded
|
||||
`per-person`, while `per-person` and `shuffled-id` were effectively tied.
|
||||
The result motivates a learned causal local active-speaker/floor-evidence
|
||||
branch, but does not justify a track-continuity claim. See the immutable
|
||||
[`result record`](results/easycom_oracle_information_4ep_2026-09-07/README.md).
|
||||
|
||||
### Current blocker
|
||||
|
||||
`evaluate_floor_state_changes.py` counts a wearer transition anywhere in a
|
||||
@@ -91,10 +106,12 @@ unfinished. This does not relax the paper claim: event numbers remain proxy
|
||||
diagnostics until competing-risk censoring is fixed.
|
||||
|
||||
1. **P0:** fix/freeze event attribution, splits and scenario definitions.
|
||||
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
|
||||
2. **P1:** confirm the aggregate/per-person oracle screen over seeds 1--2,
|
||||
then train a causal visible-person active-speaker/floor-evidence estimator;
|
||||
compare correct, wrong-person and time-shifted visual inputs.
|
||||
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
|
||||
4. **P3:** repair the matched anonymous-track-GRU residual optimization by
|
||||
4. **P3:** only if learned person-organized evidence beats aggregate/no-ID
|
||||
controls, repair the matched anonymous-track-GRU residual optimization by
|
||||
replacing the scalar zero gate with a zero-initialized residual output
|
||||
layer, and verify nonzero branch gradients on Session 12. Add broken-ID,
|
||||
wrong-time and remove-person controls only if that corrected screen improves.
|
||||
|
||||
Reference in New Issue
Block a user