docs: generalize manuscript across omni backbones

manuscript/main/REVIEWER_CHECKLIST.md

@@ -4,138 +4,157 @@ This is an internal execution checklist. It is not part of the submitted PDF.
 
 ## Reviewer verdict on the current draft
 
-The research question is potentially publishable: **at matched wearer-response
-recall, can correctly aligned person-level vision reduce false wearer entries
-during third-party conversation without degrading legitimate wearer responses?**
+The potentially publishable question is now model-agnostic: **do
+general-purpose omni representations expose the fine-grained evidence needed
+for 200 ms egocentric floor action, and can causal local evidence reduce false
+wearer entries at matched onset recall when frozen readout or PEFT is
+insufficient?**
 
 The contribution is not “the first audio-visual multiparty turn model.” VAP,
 EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
-gap is continuous, causal floor-entry projection for an **invisible camera
-wearer**, conditioned on addressee evidence from a variable participant set.
+gap is continuous, causal floor-action projection for an invisible camera
+wearer, combined with a controlled diagnosis of global omni states and a
+behavior-aligned multi-other event protocol.
 
-The current aggregate result (G+A+pooled-V minus G+A: +0.30 points at 0.2 s,
-+0.29 mean mAP, one seed) motivates the question but does not establish social
-grounding. Acceptance depends on the event protocol and identity controls.
+Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
+prove a limitation shared by omni models nor establish track-consistent fusion
+as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
+Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
 
-## P0 — Freeze the task before training a new model
+## P0 — Freeze the task before the backbone sweep
 
 - [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
   of the ten 0.2 s horizons.
-- [ ] Freeze addressed-response and third-party event definitions.
-- [ ] Add competing-risk rules: an addressed response ends at the first wearer
-  onset, unrelated-other onset, timeout, or invalid annotation boundary.
+- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
+- [ ] Add competing-risk rules: a candidate event ends at the first matched
+  wearer action, incompatible floor change, timeout, or invalid source boundary.
 - [ ] Specify overlap, minimum silence, maximum response delay, duplicate
-  detections, clip boundaries, group/none/multiple/unknown targets, and
-  offscreen speakers.
+  detections, source discontinuities, track visibility, offscreen speakers,
+  and whether target-of-speech strata are diagnostic only.
 - [ ] Produce train/test event-count tables, duration, class prevalence, and
   visibility coverage before examining model scores.
-- [ ] Mark all already viewed Sessions 1–3 results as exploratory. From this
-  point forward, lock the event protocol and select thresholds, checkpoints,
-  and all hyperparameters using a training-side fold or cross-fitting over
-  Sessions 4–12. Run the fixed final model on Sessions 1–3 only after that
-  lock; do not describe the historically viewed split as untouched.
+- [ ] Mark all already viewed Sessions 1–3 results as exploratory. Lock the
+  protocol and select thresholds, checkpoints, and hyperparameters using a
+  training-side fold or cross-fitting over Sessions 4–12.
 
 **Gate:** two independent code paths or a manual sample audit must agree on at
-least 100 randomly selected events. Do not start the main model sweep before
-this gate passes.
+least 100 randomly selected events. Publish the number of unique other IDs and
+multi-other opportunity minutes before the main sweep.
 
-## P1 — Establish matched baselines
+## P1 — Audit each backbone interface
 
-Run three seeds with identical optimizer, history, parameter budget, and event
-evaluator:
+For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:
 
-1. G: frozen streaming omni state.
-2. G+A: global state plus recent audio.
-3. G+A+V-pool: the current pooled visual baseline.
-4. Temporal control: G+A plus a parameter-matched non-person temporal branch.
+- [ ] exact model/code revision, processor, dtype, and hardware;
+- [ ] native audio/video sampling and streaming/chunk semantics;
+- [ ] exact causal hidden-state readout and dimensionality;
+- [ ] context limit, cache/reset behavior, and source-boundary handling;
+- [ ] a cutoff/impulse test proving no future frame or sample enters the state;
+- [ ] P50/P95 latency, throughput, and peak memory.
 
-Report original ten-horizon AP/mAP and the event metrics. Confirm that changing
-batching, cached features, or track padding does not change baseline outputs.
+Do not force an offline or future-conditioned feature into the common table. If
+a model cannot expose a comparable causal state, report that protocol as
+unsupported. Raw scores are not a pure architecture ranking; emphasize
+within-backbone deltas.
 
-**Gate:** baseline standard deviations are known, and the event evaluator
-reproduces identical scores from saved predictions.
+## P2 — Establish the compact representation matrix
 
-## P2 — Implement the minimum person-set model
+Run three seeds per supported backbone with the same data, physical cutoffs,
+target, prediction-head capacity, optimizer exposure, and evaluator:
 
-- [ ] Shared temporal encoder for every face track.
-- [ ] Each person representation cross-attends to the same recent mixed-audio
-  token sequence; no separated/close-mic audio at inference.
-- [ ] Masked permutation-equivariant set encoder.
-- [ ] Learned OFFSCREEN/NULL element.
-- [ ] Fixed floor/address/third-party role queries.
-- [ ] Zero-initialized gated residual into the frozen global state.
-- [ ] Unit tests for variable `N`, empty visible set, missing frames, track-slot
-  permutation, padding invariance, and strict causal cutoff.
+1. Frozen global causal state.
+2. Pre-registered PEFT/LoRA budget.
+3. Global state plus recent local audio.
+4. Global state plus pooled local audio and vision.
 
-**Gate:** with residual gate zero, logits are numerically identical to G+A; a
-random permutation of track slots leaves logits unchanged.
+Use a common local encoder as the primary cross-backbone control. If native
+local encoders are tested, report them in a separate within-model panel. Do not
+mix common-encoder and native-encoder deltas in one claim.
 
-## P3 — Train only the ablations that answer the claim
+**Gate:** baseline standard deviations are known; the evaluator reproduces
+identical scores from saved predictions; parameter counts and training budgets
+are reported. A single failed LoRA recipe is evidence about that recipe, not
+proof that the model cannot adapt.
 
-Use the same three seeds:
+## P3 — Select local fusion only after P2
 
-1. WearerSet with activity loss only.
-2. WearerSet + per-person active-speaker auxiliary loss.
-3. WearerSet + active-speaker + pure-target addressee auxiliary loss.
+If local evidence gives a reliable event-level gain, compare candidates in
+increasing complexity:
 
-Mask group/unknown/offscreen target annotations from addressee loss. They may
-remain in the activity loss. Do not use LLM pseudo-labels as primary gold.
+1. Pooled local audio/vision.
+2. Parameter-matched temporal branch.
+3. Per-frame person set with no cross-time identity.
+4. Shared track-wise temporal encoder.
+5. Person–audio cross-attention.
+6. Optional per-person active-speaker auxiliary loss, only after selecting a
+   person-aware activity-only model.
 
-Stop after these stages unless a clear event-level gain appears. Broad Thinker
-LoRA, 7B, and ROMA experiments are diagnostics, not the core method, because
-the existing runs show negative transfer or no consistent 0.2 s benefit.
+Keep the dense EgoSpeak wearer/others target as the primary task. Target of
+speech is analysis-only and never model input. Describe only the train-side
+winner as the final method; move rejected candidates to ablations.
 
-## P4 — Run causal visual counterfactuals
+For a track-aware candidate, temporary IDs route hidden state only: no ID
+embedding, cross-recording identity, or fixed seating slot. Unit-test variable
+person count, empty sets, padding, slot permutation, global ID renaming,
+source/gap resets, and strict causal cutoffs.
 
-For the frozen best checkpoint, evaluate:
+## P4 — Run visual counterfactuals conditionally
+
+Only if a track-aware candidate is selected, evaluate one frozen checkpoint
+with:
 
 - correct person tracks;
-- wrong-person tracks from the same scene;
+- clip-consistent global renaming, which must be invariant;
+- deliberately broken cross-time identity with unchanged per-frame crops;
 - time-shifted tracks with unchanged audio;
-- removed/zero visual features;
+- removed visual features;
+- fixed-rate ID switch, box dropout, and fragmentation;
 - visibility-conditioned subsets.
 
-All controls must reuse the same model weights. Retraining each control would
-mix representation and optimization effects.
+The separately trained no-ID model belongs in the main ablation table. Do not
+retrain test-time counterfactuals.
 
-**Acceptance gate:** at matched addressed-response recall, correct vision must
-reduce third-party false entries more than wrong/time-shifted/removed vision,
-without worse response latency or wearer-yield recall. Aggregate mAP alone is
-not sufficient.
+## P5 — Statistics and evidence
 
-## P5 — Statistics and analysis
+- [ ] Three seeds, mean ± sample standard deviation.
+- [ ] Paired cluster bootstrap by session/conversation block for 95% intervals.
+- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
+- [ ] False wearer entries per multi-other opportunity minute, onset latency,
+  wearer-yield recall/latency, and original ten-horizon AP/mAP.
+- [ ] Error strata: backbone, visibility, number of visible people, overlap,
+  offscreen activity, track age/fragmentation, and transition proxy.
+- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
+  same multi-other analysis.
+- [ ] Provided-track and online-tracker conditions named separately.
 
-- [ ] Three training seeds, mean ± sample standard deviation.
-- [ ] Paired cluster bootstrap by session/conversation block for 95% confidence
-  intervals; do not bootstrap adjacent 5 Hz frames independently.
-- [ ] Operating points at addressed recall 0.7, 0.8, and 0.9; headline at 0.8.
-- [ ] Error strata: target visibility, number of visible people, overlap,
-  backchannel/interruption proxy, and offscreen target.
-- [ ] P50/P95 decision latency, throughput, peak GPU memory, and trainable
-  parameters.
-- [ ] Ego4D only as secondary aggregate activity generalization; it cannot
-  validate addressee grounding without comparable target labels.
+Qualitative cases follow a frozen selection rule and include the largest paired
+improvement, a median case, a failure, and one preserved true wearer onset.
+Examples explain rates; they never replace them.
 
 ## Paper assembly order
 
-1. Finalize the event protocol and insert counts in Experimental Setup.
-2. Produce the main matched-recall table before polishing prose.
-3. Produce the correct/wrong/time-shift/remove counterfactual table.
-4. Draw one compact method figure with causal cutoffs and the variable set.
-5. Collapse the preliminary aggregate table into the final main table if page
-   pressure is high.
-6. Add one short failure analysis, not a catalog of all earlier experiments.
-7. Replace the abstract and conclusion TODOs with exact numbers and confidence
-   intervals.
-8. Remove every visible `\todo{}` marker and verify the official ICASSP 2027
-   template/page limit once released.
+1. Freeze and audit the event protocol.
+2. Implement and causally audit all backbone readouts.
+3. Complete the compact frozen/PEFT/local matrix.
+4. Decide whether the paper can support a cross-family or only model-specific
+   representation claim.
+5. Select a local fusion candidate using training-side event metrics.
+6. Run conditional visual/track counterfactuals.
+7. Produce the cross-backbone event table and paired intervals.
+8. Draw one compact final system figure; remove rejected methods from the main
+   narrative.
+9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
 
-## Stop/go decision
+## Stop/go decisions
 
-- **Go:** correct person-level vision provides a statistically credible
-  event-level advantage over G+A, pooled-V, temporal capacity control, and
-  wrong/time-shifted vision.
-- **Reframe:** all visual variants improve similarly. Claim better local
-  temporal context, not addressee grounding.
-- **Stop this method:** aggregate mAP rises but third-party false entries do
-  not fall at matched recall, or legitimate response latency/recall degrades.
+- **General claim:** at least two backbone families show the same frozen/PEFT
+  weakness and a reliable local-evidence improvement.
+- **Model-specific claim:** the effect appears only within Qwen; say so and do
+  not generalize to all omni models.
+- **Audio-local reframe:** local audio improves but all visual variants fail;
+  claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
+- **Track-aware method:** correct continuity beats pooled vision, temporal
+  capacity, no-ID/reset, broken continuity, and time-shifted controls at matched
+  recall without worse onset/yield behavior.
+- **Stop the adapter:** aggregate mAP rises but multi-other false entries do not
+  fall, or legitimate onset/yield behavior degrades.