142 lines
6.5 KiB
Markdown
142 lines
6.5 KiB
Markdown
# ICASSP 2027 completion checklist
|
||
|
||
This is an internal execution checklist. It is not part of the submitted PDF.
|
||
|
||
## Reviewer verdict on the current draft
|
||
|
||
The research question is potentially publishable: **at matched wearer-response
|
||
recall, can correctly aligned person-level vision reduce false wearer entries
|
||
during third-party conversation without degrading legitimate wearer responses?**
|
||
|
||
The contribution is not “the first audio-visual multiparty turn model.” VAP,
|
||
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
|
||
gap is continuous, causal floor-entry projection for an **invisible camera
|
||
wearer**, conditioned on addressee evidence from a variable participant set.
|
||
|
||
The current aggregate result (G+A+pooled-V minus G+A: +0.30 points at 0.2 s,
|
||
+0.29 mean mAP, one seed) motivates the question but does not establish social
|
||
grounding. Acceptance depends on the event protocol and identity controls.
|
||
|
||
## P0 — Freeze the task before training a new model
|
||
|
||
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
|
||
of the ten 0.2 s horizons.
|
||
- [ ] Freeze addressed-response and third-party event definitions.
|
||
- [ ] Add competing-risk rules: an addressed response ends at the first wearer
|
||
onset, unrelated-other onset, timeout, or invalid annotation boundary.
|
||
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
|
||
detections, clip boundaries, group/none/multiple/unknown targets, and
|
||
offscreen speakers.
|
||
- [ ] Produce train/test event-count tables, duration, class prevalence, and
|
||
visibility coverage before examining model scores.
|
||
- [ ] Mark all already viewed Sessions 1–3 results as exploratory. From this
|
||
point forward, lock the event protocol and select thresholds, checkpoints,
|
||
and all hyperparameters using a training-side fold or cross-fitting over
|
||
Sessions 4–12. Run the fixed final model on Sessions 1–3 only after that
|
||
lock; do not describe the historically viewed split as untouched.
|
||
|
||
**Gate:** two independent code paths or a manual sample audit must agree on at
|
||
least 100 randomly selected events. Do not start the main model sweep before
|
||
this gate passes.
|
||
|
||
## P1 — Establish matched baselines
|
||
|
||
Run three seeds with identical optimizer, history, parameter budget, and event
|
||
evaluator:
|
||
|
||
1. G: frozen streaming omni state.
|
||
2. G+A: global state plus recent audio.
|
||
3. G+A+V-pool: the current pooled visual baseline.
|
||
4. Temporal control: G+A plus a parameter-matched non-person temporal branch.
|
||
|
||
Report original ten-horizon AP/mAP and the event metrics. Confirm that changing
|
||
batching, cached features, or track padding does not change baseline outputs.
|
||
|
||
**Gate:** baseline standard deviations are known, and the event evaluator
|
||
reproduces identical scores from saved predictions.
|
||
|
||
## P2 — Implement the minimum person-set model
|
||
|
||
- [ ] Shared temporal encoder for every face track.
|
||
- [ ] Each person representation cross-attends to the same recent mixed-audio
|
||
token sequence; no separated/close-mic audio at inference.
|
||
- [ ] Masked permutation-equivariant set encoder.
|
||
- [ ] Learned OFFSCREEN/NULL element.
|
||
- [ ] Fixed floor/address/third-party role queries.
|
||
- [ ] Zero-initialized gated residual into the frozen global state.
|
||
- [ ] Unit tests for variable `N`, empty visible set, missing frames, track-slot
|
||
permutation, padding invariance, and strict causal cutoff.
|
||
|
||
**Gate:** with residual gate zero, logits are numerically identical to G+A; a
|
||
random permutation of track slots leaves logits unchanged.
|
||
|
||
## P3 — Train only the ablations that answer the claim
|
||
|
||
Use the same three seeds:
|
||
|
||
1. WearerSet with activity loss only.
|
||
2. WearerSet + per-person active-speaker auxiliary loss.
|
||
3. WearerSet + active-speaker + pure-target addressee auxiliary loss.
|
||
|
||
Mask group/unknown/offscreen target annotations from addressee loss. They may
|
||
remain in the activity loss. Do not use LLM pseudo-labels as primary gold.
|
||
|
||
Stop after these stages unless a clear event-level gain appears. Broad Thinker
|
||
LoRA, 7B, and ROMA experiments are diagnostics, not the core method, because
|
||
the existing runs show negative transfer or no consistent 0.2 s benefit.
|
||
|
||
## P4 — Run causal visual counterfactuals
|
||
|
||
For the frozen best checkpoint, evaluate:
|
||
|
||
- correct person tracks;
|
||
- wrong-person tracks from the same scene;
|
||
- time-shifted tracks with unchanged audio;
|
||
- removed/zero visual features;
|
||
- visibility-conditioned subsets.
|
||
|
||
All controls must reuse the same model weights. Retraining each control would
|
||
mix representation and optimization effects.
|
||
|
||
**Acceptance gate:** at matched addressed-response recall, correct vision must
|
||
reduce third-party false entries more than wrong/time-shifted/removed vision,
|
||
without worse response latency or wearer-yield recall. Aggregate mAP alone is
|
||
not sufficient.
|
||
|
||
## P5 — Statistics and analysis
|
||
|
||
- [ ] Three training seeds, mean ± sample standard deviation.
|
||
- [ ] Paired cluster bootstrap by session/conversation block for 95% confidence
|
||
intervals; do not bootstrap adjacent 5 Hz frames independently.
|
||
- [ ] Operating points at addressed recall 0.7, 0.8, and 0.9; headline at 0.8.
|
||
- [ ] Error strata: target visibility, number of visible people, overlap,
|
||
backchannel/interruption proxy, and offscreen target.
|
||
- [ ] P50/P95 decision latency, throughput, peak GPU memory, and trainable
|
||
parameters.
|
||
- [ ] Ego4D only as secondary aggregate activity generalization; it cannot
|
||
validate addressee grounding without comparable target labels.
|
||
|
||
## Paper assembly order
|
||
|
||
1. Finalize the event protocol and insert counts in Experimental Setup.
|
||
2. Produce the main matched-recall table before polishing prose.
|
||
3. Produce the correct/wrong/time-shift/remove counterfactual table.
|
||
4. Draw one compact method figure with causal cutoffs and the variable set.
|
||
5. Collapse the preliminary aggregate table into the final main table if page
|
||
pressure is high.
|
||
6. Add one short failure analysis, not a catalog of all earlier experiments.
|
||
7. Replace the abstract and conclusion TODOs with exact numbers and confidence
|
||
intervals.
|
||
8. Remove every visible `\todo{}` marker and verify the official ICASSP 2027
|
||
template/page limit once released.
|
||
|
||
## Stop/go decision
|
||
|
||
- **Go:** correct person-level vision provides a statistically credible
|
||
event-level advantage over G+A, pooled-V, temporal capacity control, and
|
||
wrong/time-shifted vision.
|
||
- **Reframe:** all visual variants improve similarly. Claim better local
|
||
temporal context, not addressee grounding.
|
||
- **Stop this method:** aggregate mAP rises but third-party false entries do
|
||
not fall at matched recall, or legitimate response latency/recall degrades.
|