# ICASSP 2027 completion checklist

This is an internal execution checklist. It is not part of the submitted PDF.

## Reviewer verdict on the current draft

The potentially publishable question is now model-agnostic: **do
general-purpose omni representations expose the fine-grained evidence needed
for 200 ms egocentric floor action, and can causal local evidence reduce false
wearer entries at matched onset recall when frozen readout or PEFT is
insufficient?**

The contribution is not “the first audio-visual multiparty turn model.” VAP,
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
gap is continuous, causal floor-action projection for an invisible camera
wearer, combined with a controlled diagnosis of global omni states and a
behavior-aligned multi-other event protocol.

Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
prove a limitation shared by omni models nor establish track-consistent fusion
as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
Qwen2.5-Omni-7B, and MiniCPM-o 4.5.

## P0 — Freeze the task before the backbone sweep

- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
  of the ten 0.2 s horizons.
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
- [ ] Add competing-risk rules: a candidate event ends at the first matched
  wearer action, incompatible floor change, timeout, or invalid source boundary.
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
  detections, source discontinuities, track visibility, offscreen speakers,
  and whether target-of-speech strata are diagnostic only.
- [ ] Produce train/test event-count tables, duration, class prevalence, and
  visibility coverage before examining model scores.
- [ ] Mark all already viewed Sessions 1–3 results as exploratory. Lock the
  protocol and select thresholds, checkpoints, and hyperparameters using a
  training-side fold or cross-fitting over Sessions 4–12.

**Gate:** two independent code paths or a manual sample audit must agree on at
least 100 randomly selected events. Publish the number of unique other IDs and
multi-other opportunity minutes before the main sweep.

## P1 — Audit each backbone interface

For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:

- [ ] exact model/code revision, processor, dtype, and hardware;
- [ ] native audio/video sampling and streaming/chunk semantics;
- [ ] exact causal hidden-state readout and dimensionality;
- [ ] context limit, cache/reset behavior, and source-boundary handling;
- [ ] a cutoff/impulse test proving no future frame or sample enters the state;
- [ ] P50/P95 latency, throughput, and peak memory.

Do not force an offline or future-conditioned feature into the common table. If
a model cannot expose a comparable causal state, report that protocol as
unsupported. Raw scores are not a pure architecture ranking; emphasize
within-backbone deltas.

## P2 — Establish the compact representation matrix

Run three seeds per supported backbone with the same data, physical cutoffs,
target, prediction-head capacity, optimizer exposure, and evaluator:

1. Frozen global causal state.
2. Pre-registered PEFT/LoRA budget.
3. Global state plus recent local audio.
4. Global state plus pooled local audio and vision.

Use a common local encoder as the primary cross-backbone control. If native
local encoders are tested, report them in a separate within-model panel. Do not
mix common-encoder and native-encoder deltas in one claim.

**Gate:** baseline standard deviations are known; the evaluator reproduces
identical scores from saved predictions; parameter counts and training budgets
are reported. A single failed LoRA recipe is evidence about that recipe, not
proof that the model cannot adapt.

## P3 — Select local fusion only after P2

If local evidence gives a reliable event-level gain, compare candidates in
increasing complexity:

1. Pooled local audio/vision.
2. Parameter-matched temporal branch.
3. Per-frame person set with no cross-time identity.
4. Shared track-wise temporal encoder.
5. Person–audio cross-attention.
6. Optional per-person active-speaker auxiliary loss, only after selecting a
   person-aware activity-only model.

Keep the dense EgoSpeak wearer/others target as the primary task. Target of
speech is analysis-only and never model input. Describe only the train-side
winner as the final method; move rejected candidates to ablations.

For a track-aware candidate, temporary IDs route hidden state only: no ID
embedding, cross-recording identity, or fixed seating slot. Unit-test variable
person count, empty sets, padding, slot permutation, global ID renaming,
source/gap resets, and strict causal cutoffs.

## P4 — Run visual counterfactuals conditionally

Only if a track-aware candidate is selected, evaluate one frozen checkpoint
with:

- correct person tracks;
- clip-consistent global renaming, which must be invariant;
- deliberately broken cross-time identity with unchanged per-frame crops;
- time-shifted tracks with unchanged audio;
- removed visual features;
- fixed-rate ID switch, box dropout, and fragmentation;
- visibility-conditioned subsets.

The separately trained no-ID model belongs in the main ablation table. Do not
retrain test-time counterfactuals.

## P5 — Statistics and evidence

- [ ] Three seeds, mean ± sample standard deviation.
- [ ] Paired cluster bootstrap by session/conversation block for 95% intervals.
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
- [ ] False wearer entries per multi-other opportunity minute, onset latency,
  wearer-yield recall/latency, and original ten-horizon AP/mAP.
- [ ] Error strata: backbone, visibility, number of visible people, overlap,
  offscreen activity, track age/fragmentation, and transition proxy.
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
  same multi-other analysis.
- [ ] Provided-track and online-tracker conditions named separately.

Qualitative cases follow a frozen selection rule and include the largest paired
improvement, a median case, a failure, and one preserved true wearer onset.
Examples explain rates; they never replace them.

## Paper assembly order

1. Freeze and audit the event protocol.
2. Implement and causally audit all backbone readouts.
3. Complete the compact frozen/PEFT/local matrix.
4. Decide whether the paper can support a cross-family or only model-specific
   representation claim.
5. Select a local fusion candidate using training-side event metrics.
6. Run conditional visual/track counterfactuals.
7. Produce the cross-backbone event table and paired intervals.
8. Draw one compact final system figure; remove rejected methods from the main
   narrative.
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.

## Stop/go decisions

- **General claim:** at least two backbone families show the same frozen/PEFT
  weakness and a reliable local-evidence improvement.
- **Model-specific claim:** the effect appears only within Qwen; say so and do
  not generalize to all omni models.
- **Audio-local reframe:** local audio improves but all visual variants fail;
  claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
- **Track-aware method:** correct continuity beats pooled vision, temporal
  capacity, no-ID/reset, broken continuity, and time-shifted controls at matched
  recall without worse onset/yield behavior.
- **Stop the adapter:** aggregate mAP rises but multi-other false entries do not
  fall, or legitimate onset/yield behavior degrades.