docs: define speaker-grounded floor experiment plan

README.md

@@ -1,8 +1,45 @@
-# Omni-only Turn-Taking Experiment
+# Speaker-Grounded Omni Floor-State Forecasting
 
-This project tests whether **Qwen2.5-Omni-3B can replace both the handcrafted
-audio/visual feature extractors and the downstream RNN/Transformer used by
-EgoSpeak** for egocentric turn-taking anticipation.
+This project studies whether a strictly causal, end-to-end pretrained
+**Qwen2.5-Omni-3B** backbone contains speaker-grounded audiovisual evidence,
+and whether that evidence can improve the **floor-entry and floor-yield
+actions of an invisible egocentric wearer** in natural multiparty
+conversations.
+
+Replacing EgoSpeak's handcrafted encoders and downstream GRU/Transformer is
+retained as a Stage-0 representation baseline. It is no longer the final paper
+question. The central test is behavioral: at matched wearer-onset recall,
+correct person-aligned vision should reduce false wearer entries while two
+other people are conversing, without reducing responses addressed to the
+wearer or degrading wearer yield decisions.
+
+## Current paper question and experiment plan (2026-08-30)
+
+The full staged plan is in
+[the Omni speaker-floor experiment plan](docs/omni_speaker_floor_experiment_plan_2026-08-30.md).
+It separates four questions that must not be conflated:
+
+1. Does a frozen global Omni state encode aggregate wearer/others activity?
+2. Can Omni features identify **which visible person** is speaking, rather
+   than merely detect that somebody is speaking?
+3. Does anonymous person-wise causal fusion reduce third-party false entries
+   at matched wearer onset recall?
+4. Does any suppression preserve addressed-to-wearer response recall/latency
+   and wearer yield behavior, rather than making the model globally passive?
+
+The immediate order is: freeze the event protocol and competing-risk
+attribution; run speaker-grounding probes and wrong-person/time-shift controls;
+re-evaluate `G`, `G+A`, and `G+A+pooled-V`; then train an anonymous person-wise
+causal side branch. A person-memory adapter or Thinker LoRA is a second-stage
+experiment only after the simpler branch passes those diagnostic gates.
+
+The literature boundary and novelty threats are recorded in
+[the multiparty egocentric floor gap audit](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md).
+The precise target is the intersection of egocentric wearer action, strict
+causality, speaker/addressee grounding, both entry and yield, and event-level
+evaluation. This repository does **not** claim the first multiparty AV
+turn-taking system, the first online multiparty EOT detector, or that a frozen
+Omni backbone is task-level end-to-end training.
 
 The persistent paper index and Chinese summaries are maintained in
 [literature_notes](literature_notes/README.md).
@@ -22,7 +59,7 @@ horizon curves are archived in
 It keeps causal-valid summaries separate from full-timeline curves and records
 the head-capacity/context caveats for comparisons with G+A and G+A+V.
 
-The next research plan is
+The earlier engineering plan is
 [speaker-aware Thinker and wearer-transition evaluation](docs/speaker_aware_thinker_plan_2026-08-28.md)
 (2026-08-28): person-organized video chunks, frozen-Thinker local-evidence
 interventions, and onset/offset plus third-party-conversation diagnostics.
@@ -34,7 +71,7 @@ Thinker LoRA on 8×L20, with matched-onset-recall false-entry evaluation and
 addressee-specific safeguards. It also records related-work and input-permission
 limits; none of these new variants has been implemented or measured yet.
 
-The main experiment is deliberately simple:
+The Stage-0 representation experiment is deliberately simple:
 
 ```text
 causal audio + video up to time t
@@ -51,9 +88,11 @@ causal audio + video up to time t
   future logits [10 horizons, 3 classes]
 ```
 
-There is **no RNN after Qwen**. The Omni Thinker is the temporal model. The
-only supervised component in the first experiment is a small `10 x 3`
-classification head.
+There is **no RNN after Qwen in this Stage-0 baseline**. The Omni Thinker is
+its temporal model, and its only supervised component is a small `10 x 3`
+classification head. Later person-wise GRU/TCN branches and local-memory
+adapters are controlled methods for testing speaker grounding; they do not
+change what the Stage-0 result measures.
 
 ## Current status
 
@@ -66,6 +105,8 @@ classification head.
 | `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
 | `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
 | `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
+| `extract_turn_local_features.py` | Verified on EasyCom; Ego4D extraction in progress | Extract person-crop and recent-audio Qwen sidecars |
+| `evaluate_floor_state_changes.py` | Implemented; attribution revision required | Diagnose wearer entry/yield events from saved predictions |
 | `eval_omni_turn_lora.py` | Implemented, remote validation pending | Evaluate Thinker LoRA from dense AV caches with the same AP/mAP protocol |
 
 The full Thinker input path, embedded audio, direct video decoding,
@@ -1687,28 +1728,40 @@ Reference Table 2 targets from the EgoSpeak paper:
 Do not compare these numbers with EgoSpeak Table 3 target-speaker-only AP;
 Table 3 uses a different metric.
 
-## Experiments that demonstrate Omni's semantic advantage
+## Current paper experiment ladder
 
-The headline comparison alone cannot establish that improvements come from
-semantics. Run the following controlled experiments:
+The headline comparison cannot establish speaker grounding or social
+understanding. Experiments therefore proceed through explicit gates:
 
-1. **Omni audio vs Omni audio+video**: measure whether visual grounding helps.
-2. **Per-horizon curve**: semantic completion should be especially visible at
-   longer horizons such as 1.2--2.0 seconds.
-3. **Context length**: compare 4, 10, 15, and 20 seconds without adding an RNN.
-4. **Layer ablation**: compare audio-encoder output and several Thinker layers,
-   because an intermediate layer may be more discriminative than the final
-   generation-oriented layer.
-5. **Semantic-completion subset**: contrast completed utterances with
-   hesitation/incomplete pauses while matching pause duration and basic
-   prosody as closely as possible.
-6. **Question/statement/backchannel subsets**: report target-speaker AP and
-   premature-interruption rate on linguistically meaningful events.
-7. **Causal transcript baseline**: compare Omni audio with causal ASR text plus
-   a language model to isolate lexical/semantic contribution.
+1. **P0 — protocol correctness:** preserve physical 5 Hz timing and causal
+   masks, then prevent an other-offset response from being attributed after a
+   new other onset (and symmetrically censor yield after the next other
+   offset). Freeze scenario rules without tuning on EasyCom test Sessions 1--3.
+2. **P1 — representation diagnosis:** compare EgoSpeak features, Qwen
+   encoder-only A/V/AV, the global Thinker state, and anonymous person slots on
+   aggregate activity and per-person active-speaker probes. Include wrong-person,
+   time-shifted, visual-only, and audio-only controls.
+3. **P2 — behavioral baselines:** evaluate `G`, `G+A`, `G+A+pooled-V`, shuffled
+   visual features, and a parameter/history-matched temporal control using
+   entry/yield event AP and third-party false entries at matched onset recall.
+4. **P3 — primary method:** use a shared causal person encoder plus
+   wearer-state-conditioned attention, then ablate per-person active-speaker,
+   addressee, and wearer event supervision.
+5. **P4 — Omni adaptation ceiling:** compare parameter-matched Thinker LoRA,
+   local person-memory adapters, and adapter+LoRA only if person features pass
+   P1 and improve floor behavior in P3.
 
-The frozen-Omni head is the cleanest representation test. LoRA can be reported
-as a second-stage ceiling, not as the only result.
+Original 0.2--2.0-second EgoSpeak AP/mAP remains a secondary comparability
+metric. The primary paper evidence is selective behavior: correct person
+alignment must reduce false wearer entry during third-party conversation while
+preserving addressed response and yield. Full hypotheses, tables, stopping
+rules, and the 8×L20 execution order are in
+[the current experiment plan](docs/omni_speaker_floor_experiment_plan_2026-08-30.md).
+
+Secondary semantic analyses can still include context/layer ablations,
+completed-versus-hesitation subsets, question/statement/backchannel subsets,
+and a causal ASR-text baseline. They support interpretation but cannot replace
+the person-alignment interventions above.
 
 ## Transcript-derived interaction-event proxies
 
@@ -2125,7 +2178,7 @@ Do not launch full EasyCom/Ego4D extraction until all of these pass:
 - test session IDs never enter training, validation, normalization statistics,
   or hyperparameter selection.
 
-Once these checks pass, the first publishable experiment is:
+Once these checks pass, the historical Stage-0 representation baseline is:
 
 ```text
 frozen Qwen2.5-Omni-3B Thinker
@@ -2136,4 +2189,5 @@ frozen Qwen2.5-Omni-3B Thinker
 + exact EgoSpeak split and Table 2 metric
 ```
 
-Repository workflow verified on 2026-08-19.
+The repository workflow was first verified on 2026-08-19. The paper question
+and gated experiment plan were revised on 2026-08-30.