Add competing-risk evaluation and align ICASSP story

This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
+57 -53
View File
@@ -2,13 +2,11 @@
This is an internal execution checklist. It is not part of the submitted PDF.
## Reviewer verdict on the current draft
## Reviewer verdict and frozen paper shape
The potentially publishable question is now model-agnostic: **do
general-purpose omni representations expose the fine-grained evidence needed
for 200 ms egocentric floor action, and can causal local evidence reduce false
wearer entries at matched onset recall when frozen readout or PEFT is
insufficient?**
The potentially publishable question is: **can a general-purpose Omni readout
be adapted into 200 ms, wearer-relative floor control by explicitly preserving
short-window causal audio and anonymous visible-person evidence?**
The contribution is not “the first audio-visual multiparty turn model.” VAP,
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
@@ -17,17 +15,25 @@ wearer, combined with a controlled diagnosis of global omni states and a
behavior-aligned multi-other event protocol.
Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
prove a limitation shared by omni models nor establish track-consistent fusion
as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
prove a limitation shared by Omni models nor establish track-consistent fusion
as the final method. Qwen2.5-Omni-3B is primary; Qwen2.5-Omni-7B is an optional
capacity control. MiniCPM-o and a cross-family sweep are outside the core paper.
## P0 — Freeze the task before the backbone sweep
The four-page paper has only three contributions: temporal--role adaptation
mismatch, one causal local floor adapter, and causal behavioral evidence.
LoRA, the oracle, and the event protocol are respectively a baseline,
motivation, and validation mechanism.
## P0 — Freeze the task before confirmatory adaptation runs
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
of the ten 0.2 s horizons.
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
- [ ] Add competing-risk rules: a candidate event ends at the first matched
wearer action, incompatible floor change, timeout, or invalid source boundary.
- [x] Implement and unit-test competing-risk rules: a response candidate ends
before the next other onset and a yield candidate before the next other
offset; right-censored and invalid windows are counted explicitly.
- [ ] Audit the competing-risk implementation on real timelines and freeze the
timeout/source-boundary convention before confirmatory reporting.
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
detections, source discontinuities, track visibility, offscreen speakers,
and whether target-of-speech strata are diagnostic only.
@@ -41,9 +47,9 @@ Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
least 100 randomly selected events. Publish the number of unique other IDs and
multi-other opportunity minutes before the main sweep.
## P1 — Audit each backbone interface
## P1 — Audit the primary backbone interface
For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:
For Qwen2.5-Omni-3B, and for 7B only if used as the scaling control, record:
- [ ] exact model/code revision, processor, dtype, and hardware;
- [ ] native audio/video sampling and streaming/chunk semantics;
@@ -57,9 +63,9 @@ a model cannot expose a comparable causal state, report that protocol as
unsupported. Raw scores are not a pure architecture ranking; emphasize
within-backbone deltas.
## P2 — Establish the compact representation matrix
## P2 — Establish the compact adaptation matrix
Run three seeds per supported backbone with the same data, physical cutoffs,
Run three seeds for the primary backbone with the same data, physical cutoffs,
target, prediction-head capacity, optimizer exposure, and evaluator:
1. Frozen global causal state.
@@ -67,41 +73,39 @@ target, prediction-head capacity, optimizer exposure, and evaluator:
3. Global state plus recent local audio.
4. Global state plus pooled local audio and vision.
Use a common local encoder as the primary cross-backbone control. If native
local encoders are tested, report them in a separate within-model panel. Do not
mix common-encoder and native-encoder deltas in one claim.
Add a parameter- and history-matched temporal local branch before attributing
gain to audiovisual organization. Use the same local adapter for the optional
7B scaling row.
**Gate:** baseline standard deviations are known; the evaluator reproduces
identical scores from saved predictions; parameter counts and training budgets
are reported. A single failed LoRA recipe is evidence about that recipe, not
proof that the model cannot adapt.
## P3 — Select local fusion only after P2
## P3 — Train the no-ID local floor adapter
If local evidence gives a reliable event-level gain, compare candidates in
increasing complexity:
The primary method is a short-window causal encoder over recent mixed audio and
per-frame anonymous visible-person sets. It uses no participant-ID embedding or
cross-time track state. Compare it with:
1. Pooled local audio/vision.
2. Parameter-matched temporal branch.
3. Per-frame person set with no cross-time identity.
4. Shared track-wise temporal encoder.
5. Person–audio cross-attention.
6. Optional per-person active-speaker auxiliary loss, only after selecting a
person-aware activity-only model.
1. Recent local audio only.
2. Global/full-frame visual pooling.
3. A parameter- and history-matched temporal branch.
4. Time-shifted local vision.
5. Removed local vision.
6. A same-rate control if the local path runs faster than 5 Hz.
Keep the dense EgoSpeak wearer/others target as the primary task. Target of
speech is analysis-only and never model input. Describe only the train-side
winner as the final method; move rejected candidates to ablations.
speech is analysis-only and never model input. Use ``dual-rate'' only if the
local input/update rate differs explicitly and defeats the same-rate control.
For a track-aware candidate, temporary IDs route hidden state only: no ID
embedding, cross-recording identity, or fixed seating slot. Unit-test variable
person count, empty sets, padding, slot permutation, global ID renaming,
Unit-test variable person count, empty sets, padding, person-set permutation,
source/gap resets, and strict causal cutoffs.
## P4 — Run visual counterfactuals conditionally
## P4 — Consider track continuity only after the no-ID method wins
Only if a track-aware candidate is selected, evaluate one frozen checkpoint
with:
Only if a track-aware candidate then beats the no-ID method on training-side
data, evaluate one frozen checkpoint with:
- correct person tracks;
- clip-consistent global renaming, which must be invariant;
@@ -121,11 +125,14 @@ retrain test-time counterfactuals.
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
- [ ] False wearer entries per multi-other opportunity minute, onset latency,
wearer-yield recall/latency, and original ten-horizon AP/mAP.
- [ ] Error strata: backbone, visibility, number of visible people, overlap,
offscreen activity, track age/fragmentation, and transition proxy.
- [ ] Error strata: visibility, number of visible people, overlap, offscreen
activity, and transition proxy. Add track age/fragmentation only if a
track-aware method is selected.
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
same multi-other analysis.
- [ ] Provided-track and online-tracker conditions named separately.
- [ ] Name the causal box source and its limitations. Add separate
provided-track/online-tracker rows only if tracking enters the selected
method.
Qualitative cases follow a frozen selection rule and include the largest paired
improvement, a median case, a failure, and one preserved true wearer onset.
@@ -134,23 +141,20 @@ Examples explain rates; they never replace them.
## Paper assembly order
1. Freeze and audit the event protocol.
2. Implement and causally audit all backbone readouts.
3. Complete the compact frozen/PEFT/local matrix.
4. Decide whether the paper can support a cross-family or only model-specific
representation claim.
5. Select a local fusion candidate using training-side event metrics.
6. Run conditional visual/track counterfactuals.
7. Produce the cross-backbone event table and paired intervals.
8. Draw one compact final system figure; remove rejected methods from the main
2. Audit the Qwen2.5-Omni-3B readout and matched PEFT baseline.
3. Complete the frozen/PEFT/audio/matched-temporal/local-AV matrix.
4. Run correct/time-shift/remove-vision controls and the optional 7B scaling row.
5. Consider track continuity only if the no-ID method already passes.
6. Produce one confirmatory event table and one mechanism-ablation table.
7. Draw one compact final system figure; remove rejected methods from the main
narrative.
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
8. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
## Stop/go decisions
- **General claim:** at least two backbone families show the same frozen/PEFT
weakness and a reliable local-evidence improvement.
- **Model-specific claim:** the effect appears only within Qwen; say so and do
not generalize to all omni models.
- **Claim boundary:** with Qwen-only evidence, describe a Qwen2.5-Omni case
study and the tested adaptation mechanism; do not generalize to all Omni
models.
- **Audio-local reframe:** local audio improves but all visual variants fail;
claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
- **Track-aware method:** correct continuity beats pooled vision, temporal