icassp2027-experiment/reports/omni_turn_research_direction_2026-08-26.md

# Frozen Omni turn anticipation: evidence, trends, and next direction

Date: **2026-08-26**

This memo combines the current EasyCom experiments, a read-only audit of the
local EasyCom/Ego4D/EgoSpeak copies under `/Volumes/MX500B/Code`, and recent
work on streaming multimodal interaction. Nothing on the NTFS volume was
modified.

## Executive decision

The most defensible ICASSP direction is **not** to continue scaling the
Qwen2.5-Omni backbone or to fine-tune more of it. The evidence instead supports:

> **A frozen causal Omni backbone with a fine-grained, transition-aware
> multimodal decision readout for egocentric future-speaker anticipation.**

The current 4-epoch last-four-layer concat-MLP reaches `60.87144` mean mAP,
slightly above EgoSpeak's reported EasyCom GRU A+V result (`60.6`), while using
the Omni Transformer itself for temporal modeling and no additional RNN. It is
not yet a clean paper result: it is single-seed and the epoch budget was chosen
after looking at Sessions 1--3. Its main scientific value is the diagnosis:

1. useful turn information is distributed across the upper Omni layers;
2. a final-layer query alone under-exposes that information;
3. unconstrained high-capacity readout training overfits the nine training
   conversations very quickly;
4. larger 7B features and Thinker LoRA do not solve the problem.

The next contribution should therefore target **how to expose and supervise
the pretrained representation**, not how to make the backbone larger.

## Current result ladder

Selected EasyCom full-timeline results on Sessions 1--3:

| System | 0.2s mAP | Mean mAP | Interpretation |
|---|---:|---:|---|
| EgoSpeak Transformer A+V | 73.8 | 58.7 | best reported EgoSpeak 0.2s result |
| EgoSpeak GRU A+V | 73.5 | 60.6 | best reported EgoSpeak mean result |
| Frozen Omni-3B, final-layer query | 71.99768 | 58.51937 | initial 8s reference |
| Frozen Omni-3B, last-4 scalar mixture | **73.45085** | 60.53423 | recovers the short-horizon deficit |
| Frozen Omni-3B, last-4 concat-MLP, 4 epochs | 73.42020 | **60.87144** | exploratory project best |
| Same concat-MLP, 12 epochs | 72.44507 | 57.98809 | clear overfitting |
| Same concat-MLP, 50 epochs | 66.54645 | 53.03515 | severe overfitting |
| Frozen Omni-7B, reported run | 69.80133 | 57.67367 | larger backbone is worse |

The concat-MLP result is archived in
[`results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/`](../results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/README.md).

Compared with the scalar mixture, concat-MLP gains only `+0.33721` mean mAP
and loses `0.03065` at 0.2 seconds. The added capacity mainly helps around
1.2--1.6 seconds. This is useful but too small to justify a 16.8M-parameter
fusion module without a cleaner validation protocol and stronger regularization.

## What the local data actually imply

### EasyCom: many frames, few independent conversations

The EgoSpeak EasyCom targets contain:

| Split | Sessions | 5Hz steps | Duration | Background | Wearer | Others |
|---|---:|---:|---:|---:|---:|---:|
| Train (4--12) | 9 | 72,405 | 4.0225h | 22.57% | 24.93% | 60.50% |
| Test (1--3) | 3 | 23,048 | 1.2804h | 22.91% | 26.74% | 58.71% |
| Total | 12 | 95,453 | 5.3029h | -- | -- | -- |

Wearer and other-speaker labels overlap in about `8.08%` of all steps, so this
is intrinsically a multi-label conversational-state problem even though the
EgoSpeak implementation applies a three-way softmax objective.

This exposes an immediate objective mismatch. The observed target rows contain
exactly four conversational states:

1. `[1, 0, 0]`: silence/background;
2. `[0, 1, 0]`: wearer only;
3. `[0, 0, 1]`: other people only;
4. `[0, 1, 1]`: wearer and others overlap.

A three-way softmax cannot assign high probability to both active-speaker
classes. In the overlap state, the LSTR loss used by the current project sums
`-log p_wearer - log p_others`, forcing the two positive classes to compete.
This is especially relevant because wearer AP is the class that collapses most
under overfitting.

A structured four-state head is a cleaner formulation: predict a softmax over
`silence / wearer-only / others-only / overlap`, then expose evaluator scores
as `p_background=p_silence`, `p_wearer=p_wearer-only+p_overlap`, and
`p_others=p_others-only+p_overlap`. This retains a normalized state model while
matching the multi-positive EgoSpeak labels exactly.

The important limitation is the effective sample size. Adjacent 200ms anchors
are strongly correlated, and there are only nine independent training
sessions. Session-level variation is large:

- wearer prevalence ranges from `10.37%` to `37.70%`;
- other-speaker prevalence ranges from `47.71%` to `77.17%`;
- overlap ranges from `1.42%` to `20.35%`;
- label-change frequency ranges from about `26` to `35` events per minute.

This explains why a 16.8M concat projection can reduce training loss from
`0.806973` to `0.300277` while test mean mAP falls by `7.83629` points. The head
can learn conversation/session priors rather than a transferable switching
rule.

The data are nevertheless rich in transitions. Depending on the session,
roughly `54%`--`69%` of anchors have at least one label change in their next
two seconds. The next method should not simply oversample all transition-near
frames; it should explicitly model **persistence versus switching**.

### Existing annotations support stronger auxiliary supervision

The raw EasyCom copy contains 323 transcript JSON files with:

- 6,981 utterances and about 52,171 transcribed words;
- speaker ID and frame-accurate start/end timestamps;
- Target-of-Speech labels on 6,963 utterances;
- 6DoF tracked participant pose at the original 20 FPS;
- participant-linked head and face boxes.

These annotations permit auxiliary targets without new manual labeling:

- current conversational state;
- keep/switch state between adjacent 200ms steps;
- time-to-next-change bins;
- utterance completion/onset proximity;
- addressee type (group versus individual);
- head-orientation change and target-of-speech prediction.

Transcript text should be used as **training-only privileged information** or a
distillation target if the final online model must consume raw audio/video only.
That would demonstrate a semantic advantage without giving Omni unavailable
future text at test time.

### Ego4D is the more convincing generalization test

The split used by EgoSpeak's reported experiment contains 346 train and 87
test clips, almost all 5 minutes long:

| Split | Clips | 5Hz steps | Duration | Background | Wearer | Others | Overlap |
|---|---:|---:|---:|---:|---:|---:|---:|
| Train | 346 | 518,881 | 28.83h | 43.31% | 23.24% | 40.53% | 5.58% |
| Test | 87 | 130,472 | 7.25h | 42.41% | 26.30% | 38.81% | 5.63% |

This provides far more independent clips and substantially less split shift
than EasyCom at the session level. A method that improves only EasyCom may be
learning its fixed around-table conversational geometry. A method that also
improves Ego4D supports the stronger claim that the readout exposes general
Omni interaction cues.

## Research trend and what transfers to this project

### 1. Explicit decision heads are replacing implicit generation triggers

[ROMA](https://arxiv.org/abs/2601.10323) processes aligned streaming
audio-video units, preserves 40ms audio temporal position IDs, and places a
lightweight speak head parallel to the LM head. It forms the head input as a
learned mixture of the last four layers. This strongly validates the project's
last-four-layer query experiment, while also suggesting that the low-capacity
scalar mixture is a more natural primary model than concat-MLP.

[Proact-VL](https://arxiv.org/abs/2603.03447) inserts a dedicated FLAG token,
reads it with a small gated MLP, and separates the decision of when to respond
from response generation. [JoyAI-VL-Interaction](https://arxiv.org/abs/2606.14777)
similarly makes silence an explicit learned action rather than an external
heuristic. The shared trend is a small, explicit decision interface on top of
a general multimodal backbone.

### 2. Transition supervision is becoming central

Proact-VL argues that the important imbalance is not merely class frequency,
but **state transition versus state persistence**. It upweights switch steps
and adds two stability terms: local probability consistency during persistent
states and global speaking-rate calibration. This maps directly onto the
EasyCom structure and is the highest-priority idea for improving 0.2-second
prediction.

The [BadCat unit-based agent](https://arxiv.org/abs/2601.20230) reaches a
similar conclusion from a systems perspective: it represents dialogue control
as `keep-listen`, `listen-to-speak`, `keep-speak`, and `speak-to-listen` actions.
That is a strong conceptual justification for adding a keep/switch auxiliary
head rather than another generic hidden layer.

### 3. Frozen backbones plus task-specific fusion are credible

[MM-When2Speak](https://arxiv.org/abs/2505.14654) formulates timing as dense
response-type classification and uses adaptive attention over visual, audio,
and textual tokens. Its response fusion stage freezes the LLM and modality
encoders and optimizes a lightweight attention module. This is closely aligned
with our empirical finding that frozen Omni is more stable than Thinker LoRA.

The important next ablation is therefore not another larger query MLP. It is a
comparison between:

1. final query state;
2. learned last-four query mixture;
3. recent 200ms audio-token pooling;
4. recent 200ms audio-video token attention;
5. query plus recent-token pool.

For token pooling, only payload tokens in the causal interval should be used,
selected by TMRoPE time IDs and excluding padding/template tokens.

### 4. Fine temporal cadence is a differentiator

Most proactive video systems make decisions at one-second cadence. EgoSpeak
and this project operate at 200ms. That is not merely an implementation detail:
it creates a credible ICASSP contribution around fine-grained conversational
switch anticipation, especially because Omni audio tokens retain finer 40ms
temporal positions.

The paper should emphasize that a two-second multimodal ingestion unit does
not force a two-second decision rate. Causal readout at each 200ms anchor can
still exploit fine-grained audio timing, provided the state/readout does not
collapse the current-tail tokens.

## Recommended paper contribution

Working framing:

> **Fine-Grained Causal Readout of Frozen Omni Models for Egocentric Turn
> Anticipation**

Possible three-part contribution:

1. **Streaming formulation.** Raw egocentric audio/video is encoded by a
   frozen Omni Transformer with bounded causal memory, producing decisions at
   5Hz without an external RNN.
2. **Hierarchical temporal readout.** A parameter-efficient upper-layer query
   mixture is fused with recent TMRoPE-localized audio/video payload tokens,
   exposing both semantic context and immediate prosodic/nonverbal cues.
3. **Structured transition-aware decoding.** A four-state overlap-aware output
   is marginalized to EgoSpeak's three AP classes, while existing frame labels
   generate current, keep/switch, and time-to-change supervision with
   persistence consistency and session-aware validation.

The negative results become useful evidence rather than discarded runs:

- 7B is worse than 3B;
- Thinker LoRA causes representation drift and worse test mAP;
- concat-MLP helps briefly but collapses with longer training;
- a tiny last-four mixture recovers almost all of the useful gain.

Together they support the thesis that **the bottleneck is task-aligned causal
readout and supervision, not foundation-model capacity**.

## Prioritized experiment plan

### P0: make the current gain publishable

1. Train on Sessions 4--11 and use Session 12 only as development data.
2. Use one fixed 12-epoch schedule and save every epoch. Do not compare runs
   whose cosine schedule changes with the requested epoch count.
3. Select epoch and threshold using a predeclared development metric.
4. Retrain on Sessions 4--12 for the selected epoch count, then evaluate
   Sessions 1--3 once.
5. Run at least three seeds; EgoSpeak reports five-seed summary statistics.
6. Match single-query, fixed mean, scalar mixture, and concat-MLP schedules.

Primary selection metric should be `0.5 * mAP@0.2s + 0.5 * mean_mAP`, or both
metrics should be reported as a Pareto pair. Selecting only mean mAP can hide
the exact responsiveness regression that matters most to this project.

Before changing the representation, run an identical-output-objective
ablation on the frozen final-layer and scalar-mixture features:

| Output formulation | Training loss | Evaluation scores |
|---|---|---|
| Official control | current three-logit LSTR MCE | three-way softmax |
| Independent multi-label | BCE on wearer/others | sigmoid scores; background derived from neither |
| Structured state | four-way CE on silence/wearer/others/overlap | three marginals |

The structured state variant is preferred because it represents overlap
without abandoning a coherent normalized conversational state. If it improves
the scalar-mixture model, carry it into every later experiment.

### P1: low-capacity temporal readout

Run these in order:

| Readout | Approx. new fusion parameters | Purpose |
|---|---:|---|
| Last-4 scalar mixture | 4 | robust reference |
| Feature-wise last-4 gate | 8,192 | per-channel layer choice |
| Low-rank concat, rank 128 | about 1.31M | cross-layer interaction |
| Low-rank concat, rank 256 | about 2.62M | capacity ablation |
| Current concat-MLP | about 16.8M | upper-capacity control |

Initialize feature-wise/low-rank fusion as a residual around the scalar mean,
so the first optimizer step preserves the strong pretrained readout. Use
weight decay, dropout, and dev early stopping, but do not make generic
regularization the main novelty.

### P1: transition-aware supervision

Convert each target vector to the corresponding four-state variable `q_t`,
then derive:

- `current_t = q_t`;
- `switch_{t,h} = 1[q_{t+h} != q_t]` for short horizons;
- `time_to_change_t` capped at 2 seconds;
- `persistence_t = 1 - switch_{t,1}`.

Train the existing ten-horizon head jointly with current/switch/time-to-change
heads. Apply larger weight only to true switch steps, plus a consistency loss
when the ground-truth state persists. This should be evaluated first on the
camera-wearer AP at 0.2 seconds, where representation drift and head
overfitting have repeatedly caused the largest damage.

### P2: expose recent token evidence

Add a readout query that attends only to payload tokens from `[t-0.2s, t]`,
identified through TMRoPE positions. Compare audio-only recent tokens, AV
recent tokens, and query-plus-token fusion. Keep the backbone frozen.

This is the most distinctive method contribution, but it should follow P0/P1
because token masking and timestamp correctness are easier to get wrong than
layer fusion.

### P2: use existing semantics without test leakage

Use EasyCom transcription and Target-of-Speech only during training:

- distill utterance-completion likelihood into the AV state;
- predict group-versus-individual addressee;
- predict a coarse semantic continuation state such as incomplete/complete;
- drop the privileged text branch at inference.

This makes the claimed "Omni semantic advantage" testable. Merely replacing
wav2vec/ResNet with an Omni hidden state does not demonstrate semantic use.

### P3: generalization and ablations

- EasyCom audio-only, video-only, and AV under identical readout capacity;
- Ego4D 346/87 split with the same method and no EasyCom-specific tuning;
- 4s, 8s, and 20s memory with fixed 5Hz output;
- 2 FPS versus 5 FPS video ablation;
- runtime, memory, and causal verification;
- per-phase and transition-conditioned AP in addition to EgoSpeak Table 2.

## Stop conditions

Do not spend another full training cycle on:

- full Thinker LoRA on EasyCom alone;
- Qwen2.5-Omni-7B without a readout change;
- 25--50 epoch high-capacity heads without development early stopping;
- more overlapping-window extraction as the main architecture;
- prompt wording as the primary method.

Those directions are weakly supported by the current evidence and make the
paper harder to interpret. The clean next experiment is the matched P0 readout
ablation followed by transition-aware supervision.