wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add competing-risk evaluation and align ICASSP story

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reports/omni_turn_research_direction_2026-08-26.md
+353
View File
@@ -0,0 +1,353 @@
# Frozen Omni turn anticipation: evidence, trends, and next direction
Date: **2026-08-26**
This memo combines the current EasyCom experiments, a read-only audit of the
local EasyCom/Ego4D/EgoSpeak copies under `/Volumes/MX500B/Code`, and recent
work on streaming multimodal interaction. Nothing on the NTFS volume was
modified.
## Executive decision
The most defensible ICASSP direction is **not** to continue scaling the
Qwen2.5-Omni backbone or to fine-tune more of it. The evidence instead supports:
> **A frozen causal Omni backbone with a fine-grained, transition-aware
> multimodal decision readout for egocentric future-speaker anticipation.**
The current 4-epoch last-four-layer concat-MLP reaches `60.87144` mean mAP,
slightly above EgoSpeak's reported EasyCom GRU A+V result (`60.6`), while using
the Omni Transformer itself for temporal modeling and no additional RNN. It is
not yet a clean paper result: it is single-seed and the epoch budget was chosen
after looking at Sessions 1--3. Its main scientific value is the diagnosis:
1. useful turn information is distributed across the upper Omni layers;
2. a final-layer query alone under-exposes that information;
3. unconstrained high-capacity readout training overfits the nine training
conversations very quickly;
4. larger 7B features and Thinker LoRA do not solve the problem.
The next contribution should therefore target **how to expose and supervise
the pretrained representation**, not how to make the backbone larger.
## Current result ladder
Selected EasyCom full-timeline results on Sessions 1--3:
| System | 0.2s mAP | Mean mAP | Interpretation |
|---|---:|---:|---|
| EgoSpeak Transformer A+V | 73.8 | 58.7 | best reported EgoSpeak 0.2s result |
| EgoSpeak GRU A+V | 73.5 | 60.6 | best reported EgoSpeak mean result |
| Frozen Omni-3B, final-layer query | 71.99768 | 58.51937 | initial 8s reference |
| Frozen Omni-3B, last-4 scalar mixture | **73.45085** | 60.53423 | recovers the short-horizon deficit |
| Frozen Omni-3B, last-4 concat-MLP, 4 epochs | 73.42020 | **60.87144** | exploratory project best |
| Same concat-MLP, 12 epochs | 72.44507 | 57.98809 | clear overfitting |
| Same concat-MLP, 50 epochs | 66.54645 | 53.03515 | severe overfitting |
| Frozen Omni-7B, reported run | 69.80133 | 57.67367 | larger backbone is worse |
The concat-MLP result is archived in
[`results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/`](../results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/README.md).
Compared with the scalar mixture, concat-MLP gains only `+0.33721` mean mAP
and loses `0.03065` at 0.2 seconds. The added capacity mainly helps around
1.2--1.6 seconds. This is useful but too small to justify a 16.8M-parameter
fusion module without a cleaner validation protocol and stronger regularization.
## What the local data actually imply
### EasyCom: many frames, few independent conversations
The EgoSpeak EasyCom targets contain:
| Split | Sessions | 5Hz steps | Duration | Background | Wearer | Others |
|---|---:|---:|---:|---:|---:|---:|
| Train (4--12) | 9 | 72,405 | 4.0225h | 22.57% | 24.93% | 60.50% |
| Test (1--3) | 3 | 23,048 | 1.2804h | 22.91% | 26.74% | 58.71% |
| Total | 12 | 95,453 | 5.3029h | -- | -- | -- |
Wearer and other-speaker labels overlap in about `8.08%` of all steps, so this
is intrinsically a multi-label conversational-state problem even though the
EgoSpeak implementation applies a three-way softmax objective.
This exposes an immediate objective mismatch. The observed target rows contain
exactly four conversational states:
1. `[1, 0, 0]`: silence/background;
2. `[0, 1, 0]`: wearer only;
3. `[0, 0, 1]`: other people only;
4. `[0, 1, 1]`: wearer and others overlap.
A three-way softmax cannot assign high probability to both active-speaker
classes. In the overlap state, the LSTR loss used by the current project sums
`-log p_wearer - log p_others`, forcing the two positive classes to compete.
This is especially relevant because wearer AP is the class that collapses most
under overfitting.
A structured four-state head is a cleaner formulation: predict a softmax over
`silence / wearer-only / others-only / overlap`, then expose evaluator scores
as `p_background=p_silence`, `p_wearer=p_wearer-only+p_overlap`, and
`p_others=p_others-only+p_overlap`. This retains a normalized state model while
matching the multi-positive EgoSpeak labels exactly.
The important limitation is the effective sample size. Adjacent 200ms anchors
are strongly correlated, and there are only nine independent training
sessions. Session-level variation is large:
- wearer prevalence ranges from `10.37%` to `37.70%`;
- other-speaker prevalence ranges from `47.71%` to `77.17%`;
- overlap ranges from `1.42%` to `20.35%`;
- label-change frequency ranges from about `26` to `35` events per minute.
This explains why a 16.8M concat projection can reduce training loss from
`0.806973` to `0.300277` while test mean mAP falls by `7.83629` points. The head
can learn conversation/session priors rather than a transferable switching
rule.
The data are nevertheless rich in transitions. Depending on the session,
roughly `54%`--`69%` of anchors have at least one label change in their next
two seconds. The next method should not simply oversample all transition-near
frames; it should explicitly model **persistence versus switching**.
### Existing annotations support stronger auxiliary supervision
The raw EasyCom copy contains 323 transcript JSON files with:
- 6,981 utterances and about 52,171 transcribed words;
- speaker ID and frame-accurate start/end timestamps;
- Target-of-Speech labels on 6,963 utterances;
- 6DoF tracked participant pose at the original 20 FPS;
- participant-linked head and face boxes.
These annotations permit auxiliary targets without new manual labeling:
- current conversational state;
- keep/switch state between adjacent 200ms steps;
- time-to-next-change bins;
- utterance completion/onset proximity;
- addressee type (group versus individual);
- head-orientation change and target-of-speech prediction.
Transcript text should be used as **training-only privileged information** or a
distillation target if the final online model must consume raw audio/video only.
That would demonstrate a semantic advantage without giving Omni unavailable
future text at test time.
### Ego4D is the more convincing generalization test
The split used by EgoSpeak's reported experiment contains 346 train and 87
test clips, almost all 5 minutes long:
| Split | Clips | 5Hz steps | Duration | Background | Wearer | Others | Overlap |
|---|---:|---:|---:|---:|---:|---:|---:|
| Train | 346 | 518,881 | 28.83h | 43.31% | 23.24% | 40.53% | 5.58% |
| Test | 87 | 130,472 | 7.25h | 42.41% | 26.30% | 38.81% | 5.63% |
This provides far more independent clips and substantially less split shift
than EasyCom at the session level. A method that improves only EasyCom may be
learning its fixed around-table conversational geometry. A method that also
improves Ego4D supports the stronger claim that the readout exposes general
Omni interaction cues.
## Research trend and what transfers to this project
### 1. Explicit decision heads are replacing implicit generation triggers
[ROMA](https://arxiv.org/abs/2601.10323) processes aligned streaming
audio-video units, preserves 40ms audio temporal position IDs, and places a
lightweight speak head parallel to the LM head. It forms the head input as a
learned mixture of the last four layers. This strongly validates the project's
last-four-layer query experiment, while also suggesting that the low-capacity
scalar mixture is a more natural primary model than concat-MLP.
[Proact-VL](https://arxiv.org/abs/2603.03447) inserts a dedicated FLAG token,
reads it with a small gated MLP, and separates the decision of when to respond
from response generation. [JoyAI-VL-Interaction](https://arxiv.org/abs/2606.14777)
similarly makes silence an explicit learned action rather than an external
heuristic. The shared trend is a small, explicit decision interface on top of
a general multimodal backbone.
### 2. Transition supervision is becoming central
Proact-VL argues that the important imbalance is not merely class frequency,
but **state transition versus state persistence**. It upweights switch steps
and adds two stability terms: local probability consistency during persistent
states and global speaking-rate calibration. This maps directly onto the
EasyCom structure and is the highest-priority idea for improving 0.2-second
prediction.
The [BadCat unit-based agent](https://arxiv.org/abs/2601.20230) reaches a
similar conclusion from a systems perspective: it represents dialogue control
as `keep-listen`, `listen-to-speak`, `keep-speak`, and `speak-to-listen` actions.
That is a strong conceptual justification for adding a keep/switch auxiliary
head rather than another generic hidden layer.
### 3. Frozen backbones plus task-specific fusion are credible
[MM-When2Speak](https://arxiv.org/abs/2505.14654) formulates timing as dense
response-type classification and uses adaptive attention over visual, audio,
and textual tokens. Its response fusion stage freezes the LLM and modality
encoders and optimizes a lightweight attention module. This is closely aligned
with our empirical finding that frozen Omni is more stable than Thinker LoRA.
The important next ablation is therefore not another larger query MLP. It is a
comparison between:
1. final query state;
2. learned last-four query mixture;
3. recent 200ms audio-token pooling;
4. recent 200ms audio-video token attention;
5. query plus recent-token pool.
For token pooling, only payload tokens in the causal interval should be used,
selected by TMRoPE time IDs and excluding padding/template tokens.
### 4. Fine temporal cadence is a differentiator
Most proactive video systems make decisions at one-second cadence. EgoSpeak
and this project operate at 200ms. That is not merely an implementation detail:
it creates a credible ICASSP contribution around fine-grained conversational
switch anticipation, especially because Omni audio tokens retain finer 40ms
temporal positions.
The paper should emphasize that a two-second multimodal ingestion unit does
not force a two-second decision rate. Causal readout at each 200ms anchor can
still exploit fine-grained audio timing, provided the state/readout does not
collapse the current-tail tokens.
## Recommended paper contribution
Working framing:
> **Fine-Grained Causal Readout of Frozen Omni Models for Egocentric Turn
> Anticipation**
Possible three-part contribution:
1. **Streaming formulation.** Raw egocentric audio/video is encoded by a
frozen Omni Transformer with bounded causal memory, producing decisions at
5Hz without an external RNN.
2. **Hierarchical temporal readout.** A parameter-efficient upper-layer query
mixture is fused with recent TMRoPE-localized audio/video payload tokens,
exposing both semantic context and immediate prosodic/nonverbal cues.
3. **Structured transition-aware decoding.** A four-state overlap-aware output
is marginalized to EgoSpeak's three AP classes, while existing frame labels
generate current, keep/switch, and time-to-change supervision with
persistence consistency and session-aware validation.
The negative results become useful evidence rather than discarded runs:
- 7B is worse than 3B;
- Thinker LoRA causes representation drift and worse test mAP;
- concat-MLP helps briefly but collapses with longer training;
- a tiny last-four mixture recovers almost all of the useful gain.
Together they support the thesis that **the bottleneck is task-aligned causal
readout and supervision, not foundation-model capacity**.
## Prioritized experiment plan
### P0: make the current gain publishable
1. Train on Sessions 4--11 and use Session 12 only as development data.
2. Use one fixed 12-epoch schedule and save every epoch. Do not compare runs
whose cosine schedule changes with the requested epoch count.
3. Select epoch and threshold using a predeclared development metric.
4. Retrain on Sessions 4--12 for the selected epoch count, then evaluate
Sessions 1--3 once.
5. Run at least three seeds; EgoSpeak reports five-seed summary statistics.
6. Match single-query, fixed mean, scalar mixture, and concat-MLP schedules.
Primary selection metric should be `0.5 * mAP@0.2s + 0.5 * mean_mAP`, or both
metrics should be reported as a Pareto pair. Selecting only mean mAP can hide
the exact responsiveness regression that matters most to this project.
Before changing the representation, run an identical-output-objective
ablation on the frozen final-layer and scalar-mixture features:
| Output formulation | Training loss | Evaluation scores |
|---|---|---|
| Official control | current three-logit LSTR MCE | three-way softmax |
| Independent multi-label | BCE on wearer/others | sigmoid scores; background derived from neither |
| Structured state | four-way CE on silence/wearer/others/overlap | three marginals |
The structured state variant is preferred because it represents overlap
without abandoning a coherent normalized conversational state. If it improves
the scalar-mixture model, carry it into every later experiment.
### P1: low-capacity temporal readout
Run these in order:
| Readout | Approx. new fusion parameters | Purpose |
|---|---:|---|
| Last-4 scalar mixture | 4 | robust reference |
| Feature-wise last-4 gate | 8,192 | per-channel layer choice |
| Low-rank concat, rank 128 | about 1.31M | cross-layer interaction |
| Low-rank concat, rank 256 | about 2.62M | capacity ablation |
| Current concat-MLP | about 16.8M | upper-capacity control |
Initialize feature-wise/low-rank fusion as a residual around the scalar mean,
so the first optimizer step preserves the strong pretrained readout. Use
weight decay, dropout, and dev early stopping, but do not make generic
regularization the main novelty.
### P1: transition-aware supervision
Convert each target vector to the corresponding four-state variable `q_t`,
then derive:
- `current_t = q_t`;
- `switch_{t,h} = 1[q_{t+h} != q_t]` for short horizons;
- `time_to_change_t` capped at 2 seconds;
- `persistence_t = 1 - switch_{t,1}`.
Train the existing ten-horizon head jointly with current/switch/time-to-change
heads. Apply larger weight only to true switch steps, plus a consistency loss
when the ground-truth state persists. This should be evaluated first on the
camera-wearer AP at 0.2 seconds, where representation drift and head
overfitting have repeatedly caused the largest damage.
### P2: expose recent token evidence
Add a readout query that attends only to payload tokens from `[t-0.2s, t]`,
identified through TMRoPE positions. Compare audio-only recent tokens, AV
recent tokens, and query-plus-token fusion. Keep the backbone frozen.
This is the most distinctive method contribution, but it should follow P0/P1
because token masking and timestamp correctness are easier to get wrong than
layer fusion.
### P2: use existing semantics without test leakage
Use EasyCom transcription and Target-of-Speech only during training:
- distill utterance-completion likelihood into the AV state;
- predict group-versus-individual addressee;
- predict a coarse semantic continuation state such as incomplete/complete;
- drop the privileged text branch at inference.
This makes the claimed "Omni semantic advantage" testable. Merely replacing
wav2vec/ResNet with an Omni hidden state does not demonstrate semantic use.
### P3: generalization and ablations
- EasyCom audio-only, video-only, and AV under identical readout capacity;
- Ego4D 346/87 split with the same method and no EasyCom-specific tuning;
- 4s, 8s, and 20s memory with fixed 5Hz output;
- 2 FPS versus 5 FPS video ablation;
- runtime, memory, and causal verification;
- per-phase and transition-conditioned AP in addition to EgoSpeak Table 2.
## Stop conditions
Do not spend another full training cycle on:
- full Thinker LoRA on EasyCom alone;
- Qwen2.5-Omni-7B without a readout change;
- 25--50 epoch high-capacity heads without development early stopping;
- more overlapping-window extraction as the main architecture;
- prompt wording as the primary method.
Those directions are weakly supported by the current evidence and make the
paper harder to interpret. The clean next experiment is the matched P0 readout
ablation followed by transition-aware supervision.
reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md
+280
View File
@@ -0,0 +1,280 @@
# ROMA negative transfer and the next ICASSP direction
Date: **2026-08-27**
This memo combines the newly archived frozen-ROMA EasyCom results with recent
work on predictive turn-taking and proactive streaming multimodal agents. The
main conclusion is deliberately narrow:
> The next useful experiment is not another backbone or a larger readout. It
> is a **low-capacity multi-timescale causal readout plus structured future
> floor projection** at 5 Hz.
## Evidence from the current project
| System | 0.2s mAP | Mean mAP | What it establishes |
|---|---:|---:|---|
| Frozen Qwen-3B, final query, 8s | 71.99768 | 58.51937 | strong simple baseline |
| Frozen Qwen-3B, last-4 scalar mixture | 73.45085 | 60.53423 | upper-layer information is useful |
| Frozen Qwen-3B, last-4 concat, 4ep | 73.42020 | 60.87144 | small longer-horizon gain, high overfit risk |
| Frozen Qwen-7B, reported run | 69.80133 | 57.67367 | capacity alone is not the answer |
| Frozen ROMA, last-4 fixed mean, 4ep | 69.21863 | 58.90402 | best ROMA control |
| Frozen ROMA, last-4 concat, 4ep | 68.65486 | 56.52425 | wide fusion overfits strongly |
With the same last-4 concat design and epoch count, ROMA loses `4.76534` at
0.2 seconds and `4.34719` in mean mAP relative to Qwen-3B. This is a strong
diagnostic but not a parameter-matched attribution test: ROMA's wider hidden
state makes its concat fusion 51.4M parameters instead of about 16.8M. The
fixed mean removes that fusion-capacity confound and the worst overfitting, but
an exact Qwen-3B fixed-mean control is still missing.
This does **not** prove that ROMA is a worse streaming model. ROMA is adapted
for one-second multimodal units and proactive alert/narration supervision,
whereas this project extracts it with an eight-second Qwen-style replay and
asks for 200 ms future-speaker anticipation. The result instead shows a task
alignment mismatch: stronger generic proactive behavior does not guarantee a
better representation for fine-grained floor transfer.
## Strongly related work and what transfers
### EgoSpeak: the exact benchmark contract
[EgoSpeak (Findings of NAACL 2025)](https://github.com/jun297/EgoSpeak)
defines the exact problem and supplies LSTR and MiniROAD baselines. Its main
strength is the 5 Hz future-speaker evaluation on EasyCom and Ego4D. Its
limitation for the present question is architectural: pretrained ResNet and
wav2vec features are followed by a task-specific temporal model, so it does
not test whether an Omni Transformer already contains the necessary causal
state.
**Transfer:** preserve its split, ten horizons, target alignment, causality,
and AP evaluator exactly. The paper contribution must lie above this unchanged
contract.
### MM-VAP and MuVAP: model the future activity trajectory, not ten isolated labels
[MM-VAP (Findings of ACL 2025)](https://aclanthology.org/2025.findings-acl.12/)
adds facial expression, head pose, and gaze to an audio VAP model and reports
84% versus 79% hold/shift accuracy. Its ablation identifies facial expression
as the most useful visual cue.
[MuVAP (arXiv 2026)](https://arxiv.org/abs/2606.16731) is even closer to this
project: it is causal, multimodal, multiparty, single-camera, and single-audio
stream. It predicts a near-future joint voice-activity window and introduces
**Role-Relative Projection**, mapping arbitrary speakers to current versus
next floor-holder roles to avoid combinatorial growth.
The original VAP insight is crucial: near-future speaking states are jointly
structured. A backchannel, overlap, hold, and switch are different trajectories
through the next two seconds, not ten unrelated classification targets.
**Transfer:** replace the independent `10 x 3` output with a structured future
floor decoder. EasyCom naturally supplies four states at each horizon:
`silence`, `wearer-only`, `others-only`, and `overlap`. A small temporal decoder
can model their dependencies and marginalize back to EgoSpeak's three AP
scores. No new annotation is required.
### Interspeech 2025 gaze study: global video tokens may hide the useful cue
[Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations
(Interspeech 2025)](https://www.isca-archive.org/interspeech_2025/heo25_interspeech.html)
shows that gaze from one participant already improves prediction, and multiple
participants improve it further when gaze is grounded with speaker location.
**Transfer:** EasyCom already provides face/head boxes and tracked poses. Use
them to build participant- or role-conditioned visual pooling rather than
expecting a global video query state to preserve small facial and gaze changes.
This is more defensible than increasing the global MLP.
### ROMA: aligned streaming units and a small explicit decision head
[ROMA](https://arxiv.org/abs/2601.10323) uses one-second synchronized
audio-video units, chunked TMRoPE, 40 ms audio temporal positions, and a
two-layer speak head fed by a learned mixture of the final four layers. It also
uses a two-stage curriculum: streaming-format alignment followed by proactive
timing training.
**Transfer:** keep the explicit lightweight decision interface and last-layer
mixture. More importantly, retain the 40 ms audio time IDs when selecting the
recent 200 ms payload. The present negative result says that importing ROMA's
checkpoint alone is insufficient; its useful idea is the interface and
alignment structure.
### Proact-VL: transition imbalance is not ordinary class imbalance
[Proact-VL (2026 preprint)](https://arxiv.org/abs/2603.03447) reads a dedicated
`FLAG` token with a small gated MLP. It weights rare speaking/silence state
transitions more heavily, smooths probabilities only during persistent states,
and regularizes the global speaking rate.
**Transfer:** the project's previous transition auxiliary head is not the same
idea. It asks an extra branch to predict a transition but leaves the primary
future-activity probabilities unconstrained. A stronger experiment should:
1. upweight the main state loss at true hold/switch boundaries;
2. penalize probability jitter only inside persistence regions;
3. calibrate predicted per-class activity rates on the training split.
This directly shapes the output trajectory and is more likely to improve the
0.2-second boundary behavior than another auxiliary classifier.
### VideoLLM-online and Dispider: separate high-rate perception/decision from generation
[VideoLLM-online (CVPR 2024)](https://showlab.github.io/videollm-online/)
trains temporally aligned streaming dialogue and parallelizes video encoding,
LLM forwarding, and generation; it reports 5--10 FPS on an RTX 3090 and
10--15 FPS on an A100.
[Dispider (CVPR 2025)](https://github.com/Mark12Ding/Dispider) disentangles
perception, decision, and reaction. A compact module continuously decides
`wait/respond`, while generation runs asynchronously and does not block new
perception.
**Transfer:** for the benchmark, keep only frozen Omni perception plus the 5 Hz
floor controller. For a later interactive demo, run text/speech generation
asynchronously after a trigger. Do not train the heavy generator merely to
improve a 30-logit anticipation benchmark.
## Proposed method: Multi-timescale Future Floor Projection
At every 200 ms prediction time `t`, form two causal summaries:
1. **semantic context** `q_t`: a fixed or learned low-capacity mixture of the
final four query states over the bounded eight-second stream;
2. **immediate interaction cue** `r_t`: attention pooling over only audio/video
payload tokens whose TMRoPE timestamps fall in `(t-0.2s, t]`.
Exclude template tokens, padding, and tokens later than `t`. Keep audio and
video pools separate first, then fuse them through a small gate:
```text
q_t (long semantic state) -----------+
+--> gated low-rank fusion --> future-floor decoder
a_t (recent 200 ms audio tokens) ----+
v_t (current/role visual tokens) ----+
```
The future-floor decoder predicts a sequence of ten four-state distributions,
not ten independent three-class logits. A minimal version is:
```text
fused state -> 10 learned horizon queries -> 2-layer temporal Transformer
-> [10, 4] state logits
```
Convert it to the official evaluator scores with:
```text
p(background) = p(silence)
p(wearer) = p(wearer-only) + p(overlap)
p(others) = p(others-only) + p(overlap)
```
This design joins three complementary signals:
- Omni upper layers provide long semantic context;
- TMRoPE-local token pooling exposes recent prosody and motion;
- VAP-style structured projection models persistence, handover, overlap, and
backchannel trajectories across the two-second future.
## Highest-value experiment sequence
### P0: close attribution gaps before adding a method
1. Run Qwen-3B last-4 **fixed mean**, four epochs, same three seeds and schedule
as the existing readouts.
2. Run original Qwen2.5-Omni-7B versus ROMA with the same last-four fixed-mean
features and head. This isolates ROMA adaptation from model size.
3. Stop ROMA if the matched comparison is not positive. Do not spend more
budget tuning its 51.4M concat head.
### P1: test the central readout hypothesis
Use frozen Qwen-3B and the same dense 5 Hz anchors:
| Variant | Long state | Recent 200ms payload | Future decoder |
|---|---|---|---|
| A | final query | none | current `10 x 3` |
| B | last-4 mixture | none | current `10 x 3` |
| C | last-4 mixture | audio pool | current `10 x 3` |
| D | last-4 mixture | audio + role visual pool | current `10 x 3` |
| E | last-4 mixture | audio + role visual pool | structured `[10,4]` |
Variant C is the most informative first implementation. If C cannot improve
0.2-second wearer/background AP, the hypothesis that the single query hides
recent prosody is weakened before more complicated visual work.
### P2: add trajectory-aware loss only after P1
For E, compare:
- ordinary four-state CE;
- boundary-weighted CE;
- boundary-weighted CE plus persistence-only smoothness;
- the above plus class activity-rate calibration.
Keep loss weights small and predeclare them on a development session. Report
the main activity mAP and separate diagnostics on transition-near versus
persistence anchors.
### P3: use EasyCom annotations without new manual labels
Participant head/face boxes and tracked poses can supervise role-conditioned
visual pooling. Speech transcripts and Target-of-Speech labels should be
training-only privileged signals or distillation targets, not test-time input.
This gives a principled semantic/visual benefit while preserving the raw AV
online setting.
### P4: require Ego4D confirmation
EasyCom has only nine training conversations. A method selected on EasyCom
must be tested on the much larger Ego4D split. A gain on both datasets supports
a representation/readout claim; a gain only on EasyCom may reflect its fixed
around-table geometry.
## Evaluation contract for a publishable result
- Primary: 0.2-second mAP, because this is the current deficit and the claimed
fine-grained contribution.
- Secondary: mean mAP over all ten horizons.
- Diagnostics: per-class AP, transition-near/persistence mAP, overlap-state AP,
calibration, and end-to-end 5 Hz latency.
- Selection: Sessions 4--11 train and Session 12 development, then retrain on
Sessions 4--12 for the fixed budget and evaluate Sessions 1--3 once.
- Statistics: at least three seeds; five if matching EgoSpeak's reporting.
- Causality: no token, frame, transcript, or normalization statistic after
prediction time `t`.
## Paper angle
A defensible ICASSP framing is:
> **Fine-Grained Multimodal Future Floor Projection from Frozen Omni Models**
Possible contributions:
1. a strictly causal 5 Hz egocentric audio-video anticipation formulation on
top of a frozen Omni Transformer;
2. a multi-timescale readout that combines upper-layer semantic context with
TMRoPE-local recent payload cues;
3. an overlap-aware structured future-floor decoder with transition/persistence
supervision;
4. controlled evidence that larger backbones, ROMA transfer, and Thinker LoRA
do not solve the fine-grained task by themselves.
This is stronger than claiming a new head. The scientific claim is that
general streaming Omni states must be **read and supervised at the temporal
structure of conversational floor transfer**.
## Stop conditions
- Stop backbone scaling unless the exact same readout/schedule improves both
0.2-second and mean mAP.
- Stop high-capacity concat fusion; the current data already show severe
session-level overfitting.
- Stop transition auxiliary branches that do not change the primary activity
trajectory.
- Stop tuning on Sessions 1--3; use them once after the configuration is fixed.
The archived ROMA tables and provenance are in
[`results/easycom_roma_frozen_last4_2026-08-27/`](../results/easycom_roma_frozen_last4_2026-08-27/README.md).