icassp2027-experiment/PROJECT_CONTEXT.md

AI Project Context

This is the short, authoritative repository map for humans and AI agents. The long README.md is an operational history and command reference; it is not the experiment scheduler.

One-sentence objective

Determine whether strictly causal, speaker-grounded audiovisual evidence from Qwen2.5-Omni can improve an invisible egocentric wearer's floor-entry and floor-yield behavior in natural multiparty conversation.

The paper is not simply “replace EgoSpeak's encoders with a newer backbone.” The decisive test is whether correct person-aligned vision reduces false wearer entries during other-to-other conversation at matched wearer-onset recall, without hurting addressed responses or yield behavior.

Source-of-truth order

Need	Authoritative file
Progressive feature-fusion implementation route	docs/progressive_feature_fusion_route_2026-09-02.md
Current hypotheses, experiment gates and stopping rules	docs/omni_speaker_floor_experiment_plan_2026-08-30.md
Novelty boundary and claim restrictions	docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md
Code/data map and current blocker	This file
Runnable commands and implementation history	README.md and docs/README.md
Archived measured results	results/README.md
Literature summaries	literature_notes/README.md

When files disagree, prefer the newest scoped document and verify behavior in code/tests. A planned experiment is not an implemented feature or a measured result.

Current state

Verified assets

• Strictly causal 5 Hz Qwen2.5-Omni global state extraction on EasyCom.
• Prepared-media and direct-original-media extraction paths.
• Frozen head training/inference with EgoSpeak-compatible 10-horizon AP/mAP.
• EasyCom local Qwen visual crops and recent mixed-audio sidecars.
• Encoder-only A/V/AV and G, G+A, G+V, G+A+V baselines.
• Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
• Timing-derived wearer floor-state and event diagnostics.

Implemented, awaiting remote runtime verification

• F1 Frame-set Attention loader/head/train/infer path. It uses fused G+A as one query over the latest half-open causal person set, retains direct A in the head, and dynamically pads people per batch. Do not report it as a measured method until the L20 smoke and full controls pass.
• Two current-sidecar screening controls: audio-person-frame-set adds a zero-gated audio-to-person read before the F1 query, and gated-person-state-adapter adds a zero-gated person residual after the frozen Thinker readout. They are explicitly recorded as F5/F6 screening, not full history-conditioned F5 or an internal Thinker F6 adapter.

Current blocker

evaluate_floor_state_changes.py counts a wearer transition anywhere in a fixed response window. Before this becomes the paper event protocol, add competing-risk censoring:

• an other-offset response window ends at the next other onset;
• an other-onset yield window ends at the next other offset.

Until then, EOT/HOLD, response and yield results are proxy diagnostics, not semantic event ground truth.

Next experiment gates

1. P0: fix/freeze event attribution, splits and scenario definitions.
2. P1: probe aggregate activity and per-person active-speaker grounding; compare correct, wrong-person and time-shifted visual inputs.
3. P2: re-evaluate G, G+A, G+A+pooled-V under the frozen event protocol.
4. P3: begin the progressive person-fusion route with Frame-set Attention, then add spatial or track-temporal structure only after its controls pass.
5. P4: attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.

The detailed architecture, controls and stopping gates for P3 are frozen in the progressive feature-fusion route.

Stable data contracts

Time and labels

• Sampling rate: 5 Hz (STEP_SECONDS = 0.2).
• State row j uses media no later than (j+1)/5 seconds.
• Future horizons: 10, corresponding to 0.2, 0.4, ..., 2.0 seconds.
• Class order: background, camera_wearer_speaking, other_people_speaking.
• EasyCom targets are dense [T, 3]; predictions are [target_time, class, future_horizon].
• EasyCom split: train Session 4--12; test Session 1--3.

Global feature sidecar

Produced by streaming_turn_model.py or streaming_turn_model_directly.py as a .pt dictionary. Required training fields include:

• states: [T, 2048] or multi-layer [T, L, 2048];
• times: [0.2, 0.4, ...];
• label_indices: dense zero-origin indices into the target timeline;
• extraction mode, configuration and source metadata.

Never silently accept non-zero origins, missing label indices or mismatched time grids.

Local feature sidecar

Produced by extract_turn_local_features.py, validated by local_turn_features.py, and stored as qwen-local-pooled-v1. It contains native 2048D visual/audio vectors, validity masks, times, label indices, source metadata and the exact extraction config. A partial smoke sidecar must not be reused as a full-session feature file.

Inference result

infer_turn_head.py writes:

RESULT_DIR/
  metrics.json
  pred/<sample>.npy       # [T, 3, 10]
  valid/<sample>.npy      # [T, 10]
  transition_pred/        # only when the auxiliary head exists

evaluate_floor_state_changes.py consumes pred/ and valid/; it does not run a GPU model.

Canonical pipelines

Direct global features (preferred semantic path)

original EasyCom/Ego4D media
  -> streaming_turn_model_directly.py
  -> global feature .pt
  -> train_turn_head.py
  -> checkpoint .pth + .json
  -> infer_turn_head.py
  -> pred/ + valid/ + metrics.json
  -> evaluate_floor_state_changes.py

This path samples source video causally in Python and avoids requiring a re-encoded fixed-FPS MP4.

Prepared-media compatibility path

prepare_easycom_session.py / prepare_ego4d_2fps.py
  -> strict 2 FPS media + metadata
  -> streaming_turn_model.py
  -> the same training/inference pipeline

Keep it for reproducibility, cache-replay verification and comparisons with already extracted features.

Local evidence path

raw media + person boxes + existing global reference .pt
  -> extract_turn_local_features.py
     (or extract_ego4d_local_multigpu.py supervisor)
  -> local .pt sidecar
  -> train_turn_head.py --local-feature-root ...
  -> infer_turn_head.py --local-feature-root ...

The current local visual feature is pooled. It does not by itself prove speaker identity, active-speaker alignment or addressee understanding.

Root-code map

Root scripts remain flat because remote L20 commands call them directly. Do not reorganize them physically without compatibility wrappers.

Shared libraries

File	Responsibility
easycom_head.py	Dataset constants, feature loading, heads, losses and AP metrics
local_turn_features.py	Local-sidecar contract, alignment, crops and pooling
omni_cached_av.py	Cached AV structures used by Omni extraction/LoRA paths

Global feature extraction

File	Responsibility
streaming_turn_model_directly.py	Preferred direct raw-media causal extraction
streaming_turn_model.py	Prepared-media causal-prefix/open-stream extraction and replay checks
qwen_omni_features.py	Single-window feature inspection
cache_omni_av_pieces.py	Build reusable AV pieces for LoRA experiments
extract_ego4d_batch.py	Persistent-model Ego4D global batch extraction
analyze_block_phase.py	Streaming block-phase diagnostics
profile_streaming_bottleneck_gpu7.py	GPU extraction profiling

Data preparation and audit

File	Responsibility
easycom_manifest.py	EasyCom clip inventory and dense-timeline validation
prepare_easycom_session.py	Optional strict-grid EasyCom media preparation
prepare_ego4d_2fps.py	Optional strict-grid Ego4D media preparation
prepare_encoder_only_features.py	Convert local sidecars into encoder-only references
audit_egospeak_features.py	Audit released EgoSpeak feature layout
derive_easycom_interaction_events.py	Deterministic transcript/timing proxies
adjudicate_easycom_interaction_events.py	Optional ambiguous-event adjudication
audit_easycom_floor_events.py	Count EOT/takeover/backchannel candidates

Local features

File	Responsibility
extract_turn_local_features.py	Qwen person-crop visual and recent-audio sidecars
extract_ego4d_local_multigpu.py	Foreground supervisor for disjoint multi-GPU workers

Training, inference and evaluation

File	Responsibility
train_turn_head.py	Frozen-backbone head training, optional local fusion/auxiliary losses
infer_turn_head.py	Save EgoSpeak-compatible predictions and AP/mAP
evaluate_floor_state_changes.py	CPU floor-state/event proxy evaluation
summarize_local_baselines.py	Aggregate local baseline metrics.json files
train_omni_turn_lora.py	Thinker LoRA training from cached AV
eval_omni_turn_lora.py	Thinker LoRA evaluation

Launchers and tests

• run_*.sh and run_*.py are reproducible launchers, not libraries.
• smoke_test.py is the model/environment smoke check.
• test_*.py are CPU-oriented regression tests colocated at root for current import and command compatibility.

Terminology restrictions

• Global (G): a Thinker readout that may mix audio, video and language context; do not describe it as audio-free.
• Local audio (A): recent mixed audio encoded outside the Thinker.
• Local visual (V): person-box visual features; current pooling discards stable person structure.
• Encoder-only: Qwen modality encoder output without Thinker states.
• Speaker grounding: evidence that the correct visible person is linked to the correct speech/time. AV improvement alone is insufficient.
• Addressee grounding: evidence about whom speech targets; aggregate others labels cannot supervise it.
• Floor entry/yield: observed wearer OFF→ON / ON→OFF behavior under a defined causal event window; not automatically semantic response or interruption intent.

Environment boundaries

• The open-stream replay implementation is pinned to transformers==4.52.3 because it uses version-specific internals.
• The local encoder path has been tested with 4.52.3 and 4.57.6; sidecars from one experiment must share the same recorded contract/version.
• Remote paths differ (/data/... and /data1/...). Commands should expose dataset/model roots as variables rather than hard-code one machine.
• Large models, datasets, checkpoints and extracted features live outside Git.