# ICASSP 2027 completion checklist

This is an internal execution checklist. It is not part of the submitted PDF.

## Current priority override (2026-09-17)

Use the [local-evidence structure decision](../../docs/manuscript_local_evidence_structure_2026-09-17.md)
for the current draft. The older semantic/person-grounded checklist below is
a conditional extension, not authorization to describe it as implemented or
a prerequisite for this narrower activity-anticipation study.

- [x] Separate model architecture from streaming inference architecture.
- [x] Describe actual G+A fusion, optional pooled vision, and observed-tail
  recomputation; retain only one displayed method equation.
- [x] Move the provisional LLM annotation protocol/workload outside the paper
  body without relabeling it as human ground truth.
- [x] Reserve local-evidence, streaming/context, and combined error/qualitative
  analysis slots; record pending experiments without LoRA expansion in the
  [dated log](../../docs/local_evidence_analysis_experiment_log_2026-09-17.md).
- [ ] Complete a training-side-frozen A/G/G+A matrix with three seeds and common
  causal-valid pairs; audit capacity and distinguish their history ranges.
- [ ] Evaluate optional vision on/off before retaining a visual-gain claim.
- [ ] Complete tail refresh x local-audio four-group comparison with enabled
  local audio fixed at 5 Hz; report phase strata and same-decomposition replay.
- [ ] Re-extract/train the predeclared 4/8/16-s context-cap controls; add G at
  each cap before claiming local fusion reduces context sensitivity.
- [ ] Add a predicted-current-activity persistence reference and capacity/recency
  controls; do not supply GT activity as model input.
- [ ] Report scene-level activity-error counts and fixed-horizon timelines,
  including a shared failure; semantic negatives cannot override raw labels.
- [ ] Report dense AP/mAP and conditional endpoint onset/offset AP separately,
  including denominators, per-session results, and limits of the three-session
  official test. Do not claim one-event-one-vote detection.
- [ ] Measure full extraction latency/memory before claiming real-time FPS.
- [ ] Run paired G/GA backbone-transfer controls for Qwen Omni 3B/7B and
  MiniCPM-o 4.5 with the same local A; report within-backbone deltas and audit
  readout, dimensions, refresh, and actual history. MiniCPM 5-Hz rows require
  GPU tail/cache/causality verification; native 1-Hz pilots need Qwen 1-Hz
  controls and cannot substitute for matched 5-Hz rows. The dated log overrides
  the older cache-diagnostic-only scope below, not its safety checks.
- [ ] Acknowledge that already viewed Sessions 1--3 results are exploratory;
  additional seeds do not create a fresh independent test.
- [ ] Resolve all TODOs, retain only supported mechanisms, and check the
  four-technical-page limit, author kit, and IEEE AI-drafting disclosure.

## Reviewer verdict and frozen paper shape

The potentially publishable question is: **can a general-purpose Omni readout
be adapted into 200 ms, wearer-relative floor control by explicitly preserving
short-window causal audio and anonymous visible-person evidence?**

The contribution is not “the first audio-visual multiparty turn model.” VAP,
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
gap is continuous, causal floor-action projection for an invisible camera
wearer, combined with a controlled diagnosis of global omni states and a
behavior-aligned multi-other event protocol.

Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
prove a limitation shared by Omni models nor establish track-consistent fusion
as the final method. Qwen2.5-Omni-3B is primary; Qwen2.5-Omni-7B is an optional
capacity control. MiniCPM-o remains outside the core method and main comparison;
it is permitted only as a diagnostic streaming/cache ablation with a matched
Qwen refresh-rate control.

The four-page paper has only three contributions: temporal--role adaptation
mismatch, one causal local floor adapter, and causal behavioral evidence.
LoRA, the oracle, and the event protocol are respectively a baseline,
motivation, and validation mechanism.

## P0 — Freeze the task before confirmatory adaptation runs

- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
  of the ten 0.2 s horizons.
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
- [x] Implement and unit-test competing-risk rules: a response candidate ends
  before the next other onset and a yield candidate before the next other
  offset; right-censored and invalid windows are counted explicitly.
- [ ] Audit the competing-risk implementation on real timelines and freeze the
  timeout/source-boundary convention before confirmatory reporting.
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
  detections, source discontinuities, track visibility, offscreen speakers,
  and whether target-of-speech strata are diagnostic only.
- [ ] Produce train/test event-count tables, duration, class prevalence, and
  visibility coverage before examining model scores.
- [ ] Mark all already viewed Sessions 1–3 results as exploratory. Lock the
  protocol and select thresholds, checkpoints, and hyperparameters using a
  training-side fold or cross-fitting over Sessions 4–12.

**Gate:** two independent code paths or a manual sample audit must agree on at
least 100 randomly selected events. Publish the number of unique other IDs and
multi-other opportunity minutes before the main sweep.

## P1 — Audit the primary backbone interface

For Qwen2.5-Omni-3B, and for 7B only if used as the scaling control, record:

- [ ] exact model/code revision, processor, dtype, and hardware;
- [ ] native audio/video sampling and streaming/chunk semantics;
- [ ] exact causal hidden-state readout and dimensionality;
- [ ] context limit, cache/reset behavior, and source-boundary handling;
- [ ] a cutoff/impulse test proving no future frame or sample enters the state;
- [ ] P50/P95 latency, throughput, and peak memory.

Do not force an offline or future-conditioned feature into the common table. If
a model cannot expose a comparable causal state, report that protocol as
unsupported. Raw scores are not a pure architecture ranking; emphasize
within-backbone deltas.

For the diagnostic streaming/cache ablation:

- [ ] compare Qwen full causal-prefix recomputation and cache reuse at the
  same 5-Hz cutoffs, recording numerical agreement, throughput, and memory;
- [ ] map the native MiniCPM-o one-second output to 5-Hz rows by causal
  forward-hold only, never by backfilling the state into earlier rows;
- [ ] run a Qwen 1-Hz update-and-hold control with the same physical chunk
  boundaries and report the within-Qwen 5-Hz versus 1-Hz delta;
- [ ] freeze initialization before the first completed chunk and reset every
  modality/language-model cache at session and source-gap boundaries;
- [ ] before admitting a MiniCPM-o 5-Hz cached-branch row, compare it against
  full causal-prefix recomputation at sampled cutoffs under a frozen cosine and
  maximum-absolute-error tolerance;
- [ ] run future-media intervention tests at every tested cutoff and record
  throughput, peak memory, feature dimensionality, and exact code revision.

If the 5-Hz cached branch fails equivalence or leakage tests, omit that row and
report only the native 1-Hz causal-hold diagnostic. Do not present unmatched
Qwen--MiniCPM raw scores as a pure backbone comparison.

## P2 — Establish the compact adaptation matrix

Run three seeds for the primary backbone with the same data, physical cutoffs,
target, prediction-head capacity, optimizer exposure, and evaluator:

1. Frozen global causal state.
2. Pre-registered PEFT/LoRA budget.
3. Global state plus recent local audio.
4. Global state plus pooled local audio and vision.

Add a parameter- and history-matched temporal local branch before attributing
gain to audiovisual organization. Use the same local adapter for the optional
7B scaling row.

**Gate:** baseline standard deviations are known; the evaluator reproduces
identical scores from saved predictions; parameter counts and training budgets
are reported. A single failed LoRA recipe is evidence about that recipe, not
proof that the model cannot adapt.

## P3 — Train the no-ID local floor adapter

The primary method is a short-window causal encoder over recent mixed audio and
per-frame anonymous visible-person sets. It uses no participant-ID embedding or
cross-time track state. Compare it with:

1. Recent local audio only.
2. Global/full-frame visual pooling.
3. A parameter- and history-matched temporal branch.
4. Time-shifted local vision.
5. Removed local vision.
6. A same-rate control if the local path runs faster than 5 Hz.

Keep the dense EgoSpeak wearer/others target as the primary task. Target of
speech is analysis-only and never model input. Use ``dual-rate'' only if the
local input/update rate differs explicitly and defeats the same-rate control.

Unit-test variable person count, empty sets, padding, person-set permutation,
source/gap resets, and strict causal cutoffs.

## P4 — Consider track continuity only after the no-ID method wins

Only if a track-aware candidate then beats the no-ID method on training-side
data, evaluate one frozen checkpoint with:

- correct person tracks;
- clip-consistent global renaming, which must be invariant;
- deliberately broken cross-time identity with unchanged per-frame crops;
- time-shifted tracks with unchanged audio;
- removed visual features;
- fixed-rate ID switch, box dropout, and fragmentation;
- visibility-conditioned subsets.

The separately trained no-ID model belongs in the main ablation table. Do not
retrain test-time counterfactuals.

## P5 — Statistics and evidence

- [ ] Three seeds, mean ± sample standard deviation.
- [ ] Paired cluster bootstrap by session/conversation block for 95% intervals.
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
- [ ] Fixed-horizon entry/INT event AP, recall-matched hard-negative
  opportunity FPR, subtype FPR, and original ten-horizon dense AP/mAP.
- [ ] Treat one-to-one event matching and signed latency as secondary; report
  them only after freezing the online threshold, persistence, refractory
  period, matching interval, and competing-risk rule.
- [ ] Error strata: visibility, number of visible people, overlap, offscreen
  activity, and transition proxy. Add track age/fragmentation only if a
  track-aware method is selected.
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
  same multi-other analysis.
- [ ] Name the causal box source and its limitations. Add separate
  provided-track/online-tracker rows only if tracking enters the selected
  method.

Qualitative cases follow a frozen selection rule and include the largest paired
improvement, a median case, a failure, and one preserved true wearer onset.
Examples explain rates; they never replace them.

## Paper assembly order

1. Freeze and audit the event protocol.
2. Audit the Qwen2.5-Omni-3B readout and matched PEFT baseline.
3. Complete the frozen/PEFT/audio/matched-temporal/local-AV matrix.
4. Run correct/time-shift/remove-vision controls and the optional 7B scaling row.
5. Run the Qwen refresh-rate control and the scoped MiniCPM-o streaming/cache
   diagnostic; admit the 5-Hz MiniCPM-o row only after equivalence and leakage
   audits.
6. Consider track continuity only if the no-ID method already passes.
7. Produce one confirmatory event table and one mechanism-ablation table.
8. Draw one compact final system figure; remove rejected methods from the main
   narrative.
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.

## Stop/go decisions

- **Claim boundary:** with Qwen-only evidence, describe a Qwen2.5-Omni case
  study and the tested adaptation mechanism; do not generalize to all Omni
  models.
- **Streaming boundary:** call native MiniCPM-o 1-Hz plus causal hold a held-state
  baseline, not native 5-Hz inference; a 5-Hz cache result exists only after
  cached/full-prefix equivalence and future-input invariance pass.
- **Audio-local reframe:** local audio improves but all visual variants fail;
  claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
- **Track-aware method:** correct continuity beats pooled vision, temporal
  capacity, no-ID/reset, broken continuity, and time-shifted controls at matched
  recall without worse onset/yield behavior.
- **Stop the adapter:** aggregate mAP rises but multi-other false entries do not
  fall, or legitimate onset/yield behavior degrades.