Add competing-risk evaluation and align ICASSP story
This commit is contained in:
@@ -2,13 +2,11 @@
|
||||
|
||||
This is an internal execution checklist. It is not part of the submitted PDF.
|
||||
|
||||
## Reviewer verdict on the current draft
|
||||
## Reviewer verdict and frozen paper shape
|
||||
|
||||
The potentially publishable question is now model-agnostic: **do
|
||||
general-purpose omni representations expose the fine-grained evidence needed
|
||||
for 200 ms egocentric floor action, and can causal local evidence reduce false
|
||||
wearer entries at matched onset recall when frozen readout or PEFT is
|
||||
insufficient?**
|
||||
The potentially publishable question is: **can a general-purpose Omni readout
|
||||
be adapted into 200 ms, wearer-relative floor control by explicitly preserving
|
||||
short-window causal audio and anonymous visible-person evidence?**
|
||||
|
||||
The contribution is not “the first audio-visual multiparty turn model.” VAP,
|
||||
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
|
||||
@@ -17,17 +15,25 @@ wearer, combined with a controlled diagnosis of global omni states and a
|
||||
behavior-aligned multi-other event protocol.
|
||||
|
||||
Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
|
||||
prove a limitation shared by omni models nor establish track-consistent fusion
|
||||
as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
|
||||
Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
|
||||
prove a limitation shared by Omni models nor establish track-consistent fusion
|
||||
as the final method. Qwen2.5-Omni-3B is primary; Qwen2.5-Omni-7B is an optional
|
||||
capacity control. MiniCPM-o and a cross-family sweep are outside the core paper.
|
||||
|
||||
## P0 — Freeze the task before the backbone sweep
|
||||
The four-page paper has only three contributions: temporal--role adaptation
|
||||
mismatch, one causal local floor adapter, and causal behavioral evidence.
|
||||
LoRA, the oracle, and the event protocol are respectively a baseline,
|
||||
motivation, and validation mechanism.
|
||||
|
||||
## P0 — Freeze the task before confirmatory adaptation runs
|
||||
|
||||
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
|
||||
of the ten 0.2 s horizons.
|
||||
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
|
||||
- [ ] Add competing-risk rules: a candidate event ends at the first matched
|
||||
wearer action, incompatible floor change, timeout, or invalid source boundary.
|
||||
- [x] Implement and unit-test competing-risk rules: a response candidate ends
|
||||
before the next other onset and a yield candidate before the next other
|
||||
offset; right-censored and invalid windows are counted explicitly.
|
||||
- [ ] Audit the competing-risk implementation on real timelines and freeze the
|
||||
timeout/source-boundary convention before confirmatory reporting.
|
||||
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
|
||||
detections, source discontinuities, track visibility, offscreen speakers,
|
||||
and whether target-of-speech strata are diagnostic only.
|
||||
@@ -41,9 +47,9 @@ Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
|
||||
least 100 randomly selected events. Publish the number of unique other IDs and
|
||||
multi-other opportunity minutes before the main sweep.
|
||||
|
||||
## P1 — Audit each backbone interface
|
||||
## P1 — Audit the primary backbone interface
|
||||
|
||||
For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:
|
||||
For Qwen2.5-Omni-3B, and for 7B only if used as the scaling control, record:
|
||||
|
||||
- [ ] exact model/code revision, processor, dtype, and hardware;
|
||||
- [ ] native audio/video sampling and streaming/chunk semantics;
|
||||
@@ -57,9 +63,9 @@ a model cannot expose a comparable causal state, report that protocol as
|
||||
unsupported. Raw scores are not a pure architecture ranking; emphasize
|
||||
within-backbone deltas.
|
||||
|
||||
## P2 — Establish the compact representation matrix
|
||||
## P2 — Establish the compact adaptation matrix
|
||||
|
||||
Run three seeds per supported backbone with the same data, physical cutoffs,
|
||||
Run three seeds for the primary backbone with the same data, physical cutoffs,
|
||||
target, prediction-head capacity, optimizer exposure, and evaluator:
|
||||
|
||||
1. Frozen global causal state.
|
||||
@@ -67,41 +73,39 @@ target, prediction-head capacity, optimizer exposure, and evaluator:
|
||||
3. Global state plus recent local audio.
|
||||
4. Global state plus pooled local audio and vision.
|
||||
|
||||
Use a common local encoder as the primary cross-backbone control. If native
|
||||
local encoders are tested, report them in a separate within-model panel. Do not
|
||||
mix common-encoder and native-encoder deltas in one claim.
|
||||
Add a parameter- and history-matched temporal local branch before attributing
|
||||
gain to audiovisual organization. Use the same local adapter for the optional
|
||||
7B scaling row.
|
||||
|
||||
**Gate:** baseline standard deviations are known; the evaluator reproduces
|
||||
identical scores from saved predictions; parameter counts and training budgets
|
||||
are reported. A single failed LoRA recipe is evidence about that recipe, not
|
||||
proof that the model cannot adapt.
|
||||
|
||||
## P3 — Select local fusion only after P2
|
||||
## P3 — Train the no-ID local floor adapter
|
||||
|
||||
If local evidence gives a reliable event-level gain, compare candidates in
|
||||
increasing complexity:
|
||||
The primary method is a short-window causal encoder over recent mixed audio and
|
||||
per-frame anonymous visible-person sets. It uses no participant-ID embedding or
|
||||
cross-time track state. Compare it with:
|
||||
|
||||
1. Pooled local audio/vision.
|
||||
2. Parameter-matched temporal branch.
|
||||
3. Per-frame person set with no cross-time identity.
|
||||
4. Shared track-wise temporal encoder.
|
||||
5. Person–audio cross-attention.
|
||||
6. Optional per-person active-speaker auxiliary loss, only after selecting a
|
||||
person-aware activity-only model.
|
||||
1. Recent local audio only.
|
||||
2. Global/full-frame visual pooling.
|
||||
3. A parameter- and history-matched temporal branch.
|
||||
4. Time-shifted local vision.
|
||||
5. Removed local vision.
|
||||
6. A same-rate control if the local path runs faster than 5 Hz.
|
||||
|
||||
Keep the dense EgoSpeak wearer/others target as the primary task. Target of
|
||||
speech is analysis-only and never model input. Describe only the train-side
|
||||
winner as the final method; move rejected candidates to ablations.
|
||||
speech is analysis-only and never model input. Use ``dual-rate'' only if the
|
||||
local input/update rate differs explicitly and defeats the same-rate control.
|
||||
|
||||
For a track-aware candidate, temporary IDs route hidden state only: no ID
|
||||
embedding, cross-recording identity, or fixed seating slot. Unit-test variable
|
||||
person count, empty sets, padding, slot permutation, global ID renaming,
|
||||
Unit-test variable person count, empty sets, padding, person-set permutation,
|
||||
source/gap resets, and strict causal cutoffs.
|
||||
|
||||
## P4 — Run visual counterfactuals conditionally
|
||||
## P4 — Consider track continuity only after the no-ID method wins
|
||||
|
||||
Only if a track-aware candidate is selected, evaluate one frozen checkpoint
|
||||
with:
|
||||
Only if a track-aware candidate then beats the no-ID method on training-side
|
||||
data, evaluate one frozen checkpoint with:
|
||||
|
||||
- correct person tracks;
|
||||
- clip-consistent global renaming, which must be invariant;
|
||||
@@ -121,11 +125,14 @@ retrain test-time counterfactuals.
|
||||
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
|
||||
- [ ] False wearer entries per multi-other opportunity minute, onset latency,
|
||||
wearer-yield recall/latency, and original ten-horizon AP/mAP.
|
||||
- [ ] Error strata: backbone, visibility, number of visible people, overlap,
|
||||
offscreen activity, track age/fragmentation, and transition proxy.
|
||||
- [ ] Error strata: visibility, number of visible people, overlap, offscreen
|
||||
activity, and transition proxy. Add track age/fragmentation only if a
|
||||
track-aware method is selected.
|
||||
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
|
||||
same multi-other analysis.
|
||||
- [ ] Provided-track and online-tracker conditions named separately.
|
||||
- [ ] Name the causal box source and its limitations. Add separate
|
||||
provided-track/online-tracker rows only if tracking enters the selected
|
||||
method.
|
||||
|
||||
Qualitative cases follow a frozen selection rule and include the largest paired
|
||||
improvement, a median case, a failure, and one preserved true wearer onset.
|
||||
@@ -134,23 +141,20 @@ Examples explain rates; they never replace them.
|
||||
## Paper assembly order
|
||||
|
||||
1. Freeze and audit the event protocol.
|
||||
2. Implement and causally audit all backbone readouts.
|
||||
3. Complete the compact frozen/PEFT/local matrix.
|
||||
4. Decide whether the paper can support a cross-family or only model-specific
|
||||
representation claim.
|
||||
5. Select a local fusion candidate using training-side event metrics.
|
||||
6. Run conditional visual/track counterfactuals.
|
||||
7. Produce the cross-backbone event table and paired intervals.
|
||||
8. Draw one compact final system figure; remove rejected methods from the main
|
||||
2. Audit the Qwen2.5-Omni-3B readout and matched PEFT baseline.
|
||||
3. Complete the frozen/PEFT/audio/matched-temporal/local-AV matrix.
|
||||
4. Run correct/time-shift/remove-vision controls and the optional 7B scaling row.
|
||||
5. Consider track continuity only if the no-ID method already passes.
|
||||
6. Produce one confirmatory event table and one mechanism-ablation table.
|
||||
7. Draw one compact final system figure; remove rejected methods from the main
|
||||
narrative.
|
||||
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
|
||||
8. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
|
||||
|
||||
## Stop/go decisions
|
||||
|
||||
- **General claim:** at least two backbone families show the same frozen/PEFT
|
||||
weakness and a reliable local-evidence improvement.
|
||||
- **Model-specific claim:** the effect appears only within Qwen; say so and do
|
||||
not generalize to all omni models.
|
||||
- **Claim boundary:** with Qwen-only evidence, describe a Qwen2.5-Omni case
|
||||
study and the tested adaptation mechanism; do not generalize to all Omni
|
||||
models.
|
||||
- **Audio-local reframe:** local audio improves but all visual variants fail;
|
||||
claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
|
||||
- **Track-aware method:** correct continuity beats pooled vision, temporal
|
||||
|
||||
Reference in New Issue
Block a user