From 12430361adb0080554766b5dcd40027920c7c2ed Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Mon, 31 Aug 2026 23:39:15 +0800 Subject: [PATCH] docs: generalize manuscript across omni backbones --- manuscript/main/README.md | 26 ++ manuscript/main/REVIEWER_CHECKLIST.md | 207 ++++++----- manuscript/main/main.tex | 508 ++++++++++++++++---------- manuscript/main/references.bib | 7 + 4 files changed, 459 insertions(+), 289 deletions(-) diff --git a/manuscript/main/README.md b/manuscript/main/README.md index 50ccf9c..30e7f2b 100644 --- a/manuscript/main/README.md +++ b/manuscript/main/README.md @@ -28,3 +28,29 @@ main/ The author/affiliation, final results, Method, Experimental Setup, Results and Conclusion are still explicit TODOs. Do not turn planned experiments into reported findings. + +The current paper thesis is model-agnostic and deliberately narrower than +addressee prediction. It asks whether general-purpose omni representations +expose the fine-grained evidence needed for 200 ms wearer floor-action +projection, whether PEFT can recover it, and whether causal local audio/person +evidence provides a reliable complement. The initial planned backbone set is: + +- Qwen2.5-Omni-3B; +- Qwen2.5-Omni-7B; +- MiniCPM-o 4.5. + +The backbone and final fusion method are **not frozen yet**. Current Qwen-3B +numbers are exploratory motivation, not evidence that every omni model has the +same limitation. Compare frozen, PEFT, and local-evidence protocols within each +backbone; report common local encoders separately from model-native local +encoders. Raw cross-model scores are secondary because the models expose +different streaming/readout interfaces. + +Track-consistent person fusion remains one candidate after pooled and +capacity-matched controls. If selected, dataset participant IDs route +short-term causal state only; they are not semantic inputs. The publication +gate is an event-level reduction in multi-other false entries at matched +wearer-onset recall without degrading onset latency or wearer-yield behavior. +A general omni claim requires a consistent result across at least two backbone +families; otherwise the paper must state a model-specific finding. Selected +examples are explanatory evidence only. diff --git a/manuscript/main/REVIEWER_CHECKLIST.md b/manuscript/main/REVIEWER_CHECKLIST.md index 87eb6af..1377914 100644 --- a/manuscript/main/REVIEWER_CHECKLIST.md +++ b/manuscript/main/REVIEWER_CHECKLIST.md @@ -4,138 +4,157 @@ This is an internal execution checklist. It is not part of the submitted PDF. ## Reviewer verdict on the current draft -The research question is potentially publishable: **at matched wearer-response -recall, can correctly aligned person-level vision reduce false wearer entries -during third-party conversation without degrading legitimate wearer responses?** +The potentially publishable question is now model-agnostic: **do +general-purpose omni representations expose the fine-grained evidence needed +for 200 ms egocentric floor action, and can causal local evidence reduce false +wearer entries at matched onset recall when frozen readout or PEFT is +insufficient?** The contribution is not “the first audio-visual multiparty turn model.” VAP, EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible -gap is continuous, causal floor-entry projection for an **invisible camera -wearer**, conditioned on addressee evidence from a variable participant set. +gap is continuous, causal floor-action projection for an invisible camera +wearer, combined with a controlled diagnosis of global omni states and a +behavior-aligned multi-other event protocol. -The current aggregate result (G+A+pooled-V minus G+A: +0.30 points at 0.2 s, -+0.29 mean mAP, one seed) motivates the question but does not establish social -grounding. Acceptance depends on the event protocol and identity controls. +Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither +prove a limitation shared by omni models nor establish track-consistent fusion +as the final method. The planned initial backbone set is Qwen2.5-Omni-3B, +Qwen2.5-Omni-7B, and MiniCPM-o 4.5. -## P0 — Freeze the task before training a new model +## P0 — Freeze the task before the backbone sweep - [ ] Define the exact 5 Hz causal cutoff and physical target interval for each of the ten 0.2 s horizons. -- [ ] Freeze addressed-response and third-party event definitions. -- [ ] Add competing-risk rules: an addressed response ends at the first wearer - onset, unrelated-other onset, timeout, or invalid annotation boundary. +- [ ] Freeze wearer-onset/yield and multi-other exchange definitions. +- [ ] Add competing-risk rules: a candidate event ends at the first matched + wearer action, incompatible floor change, timeout, or invalid source boundary. - [ ] Specify overlap, minimum silence, maximum response delay, duplicate - detections, clip boundaries, group/none/multiple/unknown targets, and - offscreen speakers. + detections, source discontinuities, track visibility, offscreen speakers, + and whether target-of-speech strata are diagnostic only. - [ ] Produce train/test event-count tables, duration, class prevalence, and visibility coverage before examining model scores. -- [ ] Mark all already viewed Sessions 1–3 results as exploratory. From this - point forward, lock the event protocol and select thresholds, checkpoints, - and all hyperparameters using a training-side fold or cross-fitting over - Sessions 4–12. Run the fixed final model on Sessions 1–3 only after that - lock; do not describe the historically viewed split as untouched. +- [ ] Mark all already viewed Sessions 1–3 results as exploratory. Lock the + protocol and select thresholds, checkpoints, and hyperparameters using a + training-side fold or cross-fitting over Sessions 4–12. **Gate:** two independent code paths or a manual sample audit must agree on at -least 100 randomly selected events. Do not start the main model sweep before -this gate passes. +least 100 randomly selected events. Publish the number of unique other IDs and +multi-other opportunity minutes before the main sweep. -## P1 — Establish matched baselines +## P1 — Audit each backbone interface -Run three seeds with identical optimizer, history, parameter budget, and event -evaluator: +For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record: -1. G: frozen streaming omni state. -2. G+A: global state plus recent audio. -3. G+A+V-pool: the current pooled visual baseline. -4. Temporal control: G+A plus a parameter-matched non-person temporal branch. +- [ ] exact model/code revision, processor, dtype, and hardware; +- [ ] native audio/video sampling and streaming/chunk semantics; +- [ ] exact causal hidden-state readout and dimensionality; +- [ ] context limit, cache/reset behavior, and source-boundary handling; +- [ ] a cutoff/impulse test proving no future frame or sample enters the state; +- [ ] P50/P95 latency, throughput, and peak memory. -Report original ten-horizon AP/mAP and the event metrics. Confirm that changing -batching, cached features, or track padding does not change baseline outputs. +Do not force an offline or future-conditioned feature into the common table. If +a model cannot expose a comparable causal state, report that protocol as +unsupported. Raw scores are not a pure architecture ranking; emphasize +within-backbone deltas. -**Gate:** baseline standard deviations are known, and the event evaluator -reproduces identical scores from saved predictions. +## P2 — Establish the compact representation matrix -## P2 — Implement the minimum person-set model +Run three seeds per supported backbone with the same data, physical cutoffs, +target, prediction-head capacity, optimizer exposure, and evaluator: -- [ ] Shared temporal encoder for every face track. -- [ ] Each person representation cross-attends to the same recent mixed-audio - token sequence; no separated/close-mic audio at inference. -- [ ] Masked permutation-equivariant set encoder. -- [ ] Learned OFFSCREEN/NULL element. -- [ ] Fixed floor/address/third-party role queries. -- [ ] Zero-initialized gated residual into the frozen global state. -- [ ] Unit tests for variable `N`, empty visible set, missing frames, track-slot - permutation, padding invariance, and strict causal cutoff. +1. Frozen global causal state. +2. Pre-registered PEFT/LoRA budget. +3. Global state plus recent local audio. +4. Global state plus pooled local audio and vision. -**Gate:** with residual gate zero, logits are numerically identical to G+A; a -random permutation of track slots leaves logits unchanged. +Use a common local encoder as the primary cross-backbone control. If native +local encoders are tested, report them in a separate within-model panel. Do not +mix common-encoder and native-encoder deltas in one claim. -## P3 — Train only the ablations that answer the claim +**Gate:** baseline standard deviations are known; the evaluator reproduces +identical scores from saved predictions; parameter counts and training budgets +are reported. A single failed LoRA recipe is evidence about that recipe, not +proof that the model cannot adapt. -Use the same three seeds: +## P3 — Select local fusion only after P2 -1. WearerSet with activity loss only. -2. WearerSet + per-person active-speaker auxiliary loss. -3. WearerSet + active-speaker + pure-target addressee auxiliary loss. +If local evidence gives a reliable event-level gain, compare candidates in +increasing complexity: -Mask group/unknown/offscreen target annotations from addressee loss. They may -remain in the activity loss. Do not use LLM pseudo-labels as primary gold. +1. Pooled local audio/vision. +2. Parameter-matched temporal branch. +3. Per-frame person set with no cross-time identity. +4. Shared track-wise temporal encoder. +5. Person–audio cross-attention. +6. Optional per-person active-speaker auxiliary loss, only after selecting a + person-aware activity-only model. -Stop after these stages unless a clear event-level gain appears. Broad Thinker -LoRA, 7B, and ROMA experiments are diagnostics, not the core method, because -the existing runs show negative transfer or no consistent 0.2 s benefit. +Keep the dense EgoSpeak wearer/others target as the primary task. Target of +speech is analysis-only and never model input. Describe only the train-side +winner as the final method; move rejected candidates to ablations. -## P4 — Run causal visual counterfactuals +For a track-aware candidate, temporary IDs route hidden state only: no ID +embedding, cross-recording identity, or fixed seating slot. Unit-test variable +person count, empty sets, padding, slot permutation, global ID renaming, +source/gap resets, and strict causal cutoffs. -For the frozen best checkpoint, evaluate: +## P4 — Run visual counterfactuals conditionally + +Only if a track-aware candidate is selected, evaluate one frozen checkpoint +with: - correct person tracks; -- wrong-person tracks from the same scene; +- clip-consistent global renaming, which must be invariant; +- deliberately broken cross-time identity with unchanged per-frame crops; - time-shifted tracks with unchanged audio; -- removed/zero visual features; +- removed visual features; +- fixed-rate ID switch, box dropout, and fragmentation; - visibility-conditioned subsets. -All controls must reuse the same model weights. Retraining each control would -mix representation and optimization effects. +The separately trained no-ID model belongs in the main ablation table. Do not +retrain test-time counterfactuals. -**Acceptance gate:** at matched addressed-response recall, correct vision must -reduce third-party false entries more than wrong/time-shifted/removed vision, -without worse response latency or wearer-yield recall. Aggregate mAP alone is -not sufficient. +## P5 — Statistics and evidence -## P5 — Statistics and analysis +- [ ] Three seeds, mean ± sample standard deviation. +- [ ] Paired cluster bootstrap by session/conversation block for 95% intervals. +- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8. +- [ ] False wearer entries per multi-other opportunity minute, onset latency, + wearer-yield recall/latency, and original ten-horizon AP/mAP. +- [ ] Error strata: backbone, visibility, number of visible people, overlap, + offscreen activity, track age/fragmentation, and transition proxy. +- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the + same multi-other analysis. +- [ ] Provided-track and online-tracker conditions named separately. -- [ ] Three training seeds, mean ± sample standard deviation. -- [ ] Paired cluster bootstrap by session/conversation block for 95% confidence - intervals; do not bootstrap adjacent 5 Hz frames independently. -- [ ] Operating points at addressed recall 0.7, 0.8, and 0.9; headline at 0.8. -- [ ] Error strata: target visibility, number of visible people, overlap, - backchannel/interruption proxy, and offscreen target. -- [ ] P50/P95 decision latency, throughput, peak GPU memory, and trainable - parameters. -- [ ] Ego4D only as secondary aggregate activity generalization; it cannot - validate addressee grounding without comparable target labels. +Qualitative cases follow a frozen selection rule and include the largest paired +improvement, a median case, a failure, and one preserved true wearer onset. +Examples explain rates; they never replace them. ## Paper assembly order -1. Finalize the event protocol and insert counts in Experimental Setup. -2. Produce the main matched-recall table before polishing prose. -3. Produce the correct/wrong/time-shift/remove counterfactual table. -4. Draw one compact method figure with causal cutoffs and the variable set. -5. Collapse the preliminary aggregate table into the final main table if page - pressure is high. -6. Add one short failure analysis, not a catalog of all earlier experiments. -7. Replace the abstract and conclusion TODOs with exact numbers and confidence - intervals. -8. Remove every visible `\todo{}` marker and verify the official ICASSP 2027 - template/page limit once released. +1. Freeze and audit the event protocol. +2. Implement and causally audit all backbone readouts. +3. Complete the compact frozen/PEFT/local matrix. +4. Decide whether the paper can support a cross-family or only model-specific + representation claim. +5. Select a local fusion candidate using training-side event metrics. +6. Run conditional visual/track counterfactuals. +7. Produce the cross-backbone event table and paired intervals. +8. Draw one compact final system figure; remove rejected methods from the main + narrative. +9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit. -## Stop/go decision +## Stop/go decisions -- **Go:** correct person-level vision provides a statistically credible - event-level advantage over G+A, pooled-V, temporal capacity control, and - wrong/time-shifted vision. -- **Reframe:** all visual variants improve similarly. Claim better local - temporal context, not addressee grounding. -- **Stop this method:** aggregate mAP rises but third-party false entries do - not fall at matched recall, or legitimate response latency/recall degrades. +- **General claim:** at least two backbone families show the same frozen/PEFT + weakness and a reliable local-evidence improvement. +- **Model-specific claim:** the effect appears only within Qwen; say so and do + not generalize to all omni models. +- **Audio-local reframe:** local audio improves but all visual variants fail; + claim a fine-grained acoustic readout bottleneck, not social visual reasoning. +- **Track-aware method:** correct continuity beats pooled vision, temporal + capacity, no-ID/reset, broken continuity, and time-shifted controls at matched + recall without worse onset/yield behavior. +- **Stop the adapter:** aggregate mAP rises but multi-other false entries do not + fall, or legitimate onset/yield behavior degrades. diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index 388301e..3f6a550 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -7,11 +7,11 @@ \usepackage{amsmath,amssymb,graphicx,hyperref,xcolor} \newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}} -\newcommand{\method}{WearerSet} \newcommand{\vect}[1]{\boldsymbol{#1}} +\setlength{\emergencystretch}{2em} -\title{NOT EVERY OPEN FLOOR IS MINE: ADDRESSEE-CONDITIONED FLOOR-ENTRY -PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION} +\title{WHEN SHOULD AN EGOCENTRIC AGENT SPEAK? DIAGNOSING AND ADAPTING +OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION} % TODO: Insert the final author list and affiliations after the ICASSP 2027 % review policy and author order are finalized. @@ -23,55 +23,59 @@ PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION} \maketitle \begin{abstract} -An egocentric conversational agent must decide not only when a speaking -opportunity arises, but also whether that opportunity belongs to it. Existing -speech-activity projection collapses all non-wearer speakers, allowing strong -frame-level accuracy to coexist with intrusive entries during conversations -between bystanders. We formulate addressee-conditioned floor-entry projection: -dense causal forecasts of wearer and non-wearer activity at 200-ms resolution, -conditioned on a variable-cardinality set of visible participants. Our model -augments a frozen streaming omni-model with person-aligned audio--visual -representations and permutation-invariant role queries for floor state, -wearer address, and third-party interaction. We further introduce an EasyCom -event protocol that separates addressed-to-wearer responses from third-party -exchanges and reports false entries at matched response recall and decision -latency. Correct-, wrong-person, time-shifted, and removed-video controls test -whether a gain follows social grounding rather than model capacity. -\todo{Replace this sentence with the main result: FP/min reduction at 80\% -response recall, response latency/recall non-degradation, and 95\% confidence -interval.} +Before generating a response, an egocentric conversational agent must decide +whether to speak or keep listening in a scene that may contain several other +people. General-purpose omni models offer broad streaming audio--visual +context, but it is unclear whether their global representations expose the +fine-grained local cues required for a decision every 200~ms. We study this +question across representative omni backbones and adaptation regimes: frozen +global readout, parameter-efficient tuning, and causal augmentation with +recent audio and visible-person histories. The human camera wearer serves as a +behavioral proxy for the agent, while the dense EgoSpeak target remains the +common output. An EasyCom event protocol reports false wearer entries during +multi-other exchanges at matched wearer-onset recall, and track/alignment +controls test whether visual gains follow person continuity rather than added +capacity. Exploratory Qwen2.5-Omni-3B results motivate the study but do not yet +establish a limitation shared by omni models or determine the final adapter. +\todo{After the train-side study is locked, name the selected backbones and +adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction +at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.} \end{abstract} \begin{keywords} -turn-taking, speech initiation, addressee detection, egocentric video, -audio-visual learning +turn-taking, speech initiation, egocentric video, omni models, +audio-visual adaptation \end{keywords} \section{Introduction} \label{sec:intro} -Conversational agents must continuously decide whether to speak or keep -listening. In a dyadic exchange, a pause or an apparent end of turn is often -treated as an invitation to respond. This assumption fails in multiparty -interaction: one participant may finish an utterance addressed to another -participant, and an egocentric agent that reacts only to acoustic availability -will enter a floor that was never offered to it. Such an error is socially more -disruptive than a small framewise prediction error. A useful agent must -therefore distinguish \emph{when the floor may change} from \emph{whether the -change is directed to the camera wearer}. +Consider an embodied conversational agent that observes the scene through an +egocentric camera and hears mixed environmental audio. Before response +generation and speech synthesis, a causal floor controller must decide every +200~ms whether the agent should continue listening, begin speaking, or yield. +In a dyadic exchange, a pause or an apparent end of turn may be an invitation +to respond. This assumption fails in multiparty interaction: one participant +may hand the floor to another visible participant, and an agent that reacts +only to acoustic availability may interrupt an exchange that does not involve +it. We study this floor-action controller; response content generation and +closed-loop conversational effects are outside our scope. The observed human +camera wearer is a behavioral proxy rather than an equivalent autonomous +agent: its activity supplies human demonstrations of floor timing under the +same first-person observation geometry. Voice Activity Projection (VAP) models future joint speech activity and derives turn-taking events without manually labeling each event \cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal, untrimmed egocentric video, forecasting wearer and aggregate non-wearer activity over the next two seconds \cite{kim2025egospeak}. Its role-relative -output is attractive for a wearable agent, but an ``others'' channel cannot -distinguish an utterance addressed to the wearer from an exchange between -bystanders. Conventional per-frame average precision can consequently be -dominated by ordinary activity while hiding precisely the false-entry behavior -that visual social cues should prevent. Event-oriented evaluation likewise -emphasizes recall, false alarms, and latency rather than a single frame-level -score \cite{jiang2026turnbench}. +output is attractive for a wearable agent, but an ``others'' channel discards +whether the same or different visible people produced the recent activity and +how their behavior evolved across time. Conventional per-frame average +precision can consequently be dominated by ordinary activity while hiding +precisely the false-entry behavior that person-consistent visual cues may +prevent. Event-oriented evaluation likewise emphasizes recall, false alarms, +and latency rather than a single frame-level score \cite{jiang2026turnbench}. Related work supplies complementary parts of the problem. Egocentric talking-to-me detection predicts whether a completed utterance addresses the @@ -79,46 +83,44 @@ camera wearer and benefits from long-term social context \cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}. The former is generally utterance-level rather than continuous anticipation; -the latter does not ask whether an invisible camera wearer should enter. Our -gap is therefore narrower than ``audio--visual multiparty turn taking'': we -study continuous, wearer-relative floor action when the candidate responder is -behind the camera and the number of visible participants changes over time. +the latter predicts visible participants rather than the floor action of an +invisible camera wearer. Our gap is therefore narrower than ``audio--visual +multiparty turn taking'': we study whether short-term anonymous participant +continuity helps direct, continuous wearer-action projection when the candidate +responder is behind the camera and the visible participant set changes. -We call this task \emph{addressee-conditioned egocentric floor-entry -projection}. A global streaming audio--video representation supplies broad -context, while a set branch preserves individual visual histories and their -alignment to recent mixed audio. Fixed semantic role queries compress a -variable participant set into wearer-address and third-party evidence without -assuming a fixed seating order. EasyCom is suitable because it provides -synchronized egocentric audio--video, per-person tracks, voice activity, and -target-of-speech annotations \cite{donley2021easycom}. Target annotations are -used only for training-side supervision and evaluation strata, never as input -at test time. +We call this task \emph{egocentric floor-action projection}. The study asks +four questions: (RQ1) how much wearer activity is linearly or shallowly +readable from a frozen global omni state; (RQ2) whether standard +parameter-efficient tuning recovers missing performance; (RQ3) whether causal +local audio and person-level visual evidence provide complementary information; +and (RQ4) whether any aggregate gain reduces false agent entries without +making the controller globally passive. EasyCom is suitable because it +provides synchronized egocentric audio--video, per-person tracks, and voice +activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are +analysis strata only and are never model input or required output. -Our contributions are threefold. (1) We formulate continuous speech -initiation as recipient-aware floor-entry projection for an invisible camera -wearer. (2) We propose a causal set-based fusion module that grounds mixed -audio in a variable number of visible participants and preserves the original -dense EgoSpeak target. (3) We introduce an event evaluation that reports -third-party false entries at matched addressed-response recall, accompanied by -latency and visual counterfactual controls. This design makes the scientific -claim falsifiable: visual fusion is useful only if \emph{correctly aligned} -vision improves the operating trade-off without suppressing legitimate wearer -responses. +The intended contributions are therefore a controlled cross-backbone +representation diagnosis, a lightweight local-evidence adaptation selected +from pre-registered candidates, and a behavior-aligned event evaluation. The +current Qwen2.5-Omni observations are motivation, not a claim that all omni +models intrinsically fail. Likewise, track-consistent cross-attention becomes +the final method only if it beats pooled, no-ID, and capacity-matched controls. \begin{figure}[t] \centering \fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering -\todo{Draw the final method figure: causal omni state $\vect g_t$; recent -mixed-audio tokens; a masked set of face-track histories; shared per-person -audio--visual grounding; three role queries; zero-initialized residual; and -the $10\times3$ activity head. Mark all causal cutoffs and masks.}}} -\caption{Proposed wearer-relative set fusion. The number and ordering of -visible participants may change, while the output remains fixed-size.} +\todo{After model selection, draw the final system figure. For now it must show +the shared experiment interface: raw causal AV, backbone-specific global +readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous +person histories, and the common $10\times3$ head. Mark causal cutoffs, +trainable modules, parameter budgets, and source-boundary resets.}}} +\caption{Planned model-agnostic floor-action study. The local adapter remains a +candidate family until the controlled train-side comparison is complete.} \label{fig:method} \end{figure} -\section{Task and Method} +\section{Methods} \label{sec:method} \subsection{Wearer-relative activity projection} @@ -141,74 +143,109 @@ windows; elsewhere it remains a future-activity probability. This retains the EgoSpeak target and evaluation while exposing a deployable 5-Hz decision signal. -\subsection{Frozen global stream and local participant set} +\subsection{Omni backbones and representation protocols} -We obtain a causal global state $\vect{g}_t$ from the streaming Thinker of -Qwen2.5-Omni \cite{xu2025qwenomni}. The backbone is frozen in the main -experiment because preliminary LoRA adaptation caused negative transfer -(Sec.~\ref{sec:results}). From the most recent $L_a$ seconds, a shared audio -encoder produces token sequence $\vect{A}_t$. For each visible track -$i\in\{1,\ldots,N_t\}$, a shared temporal visual encoder produces -$\vect{v}_{t,i}$ from its recent face history. Person-aligned evidence is +Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a +backbone-specific causal global readout at the same physical cutoff $t$. +Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family; +Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5 +\cite{cui2026minicpmo45} form the planned initial comparison. These models have +different native streaming APIs, temporal tokenization, hidden dimensions, and +adaptation interfaces. We therefore emphasize within-backbone deltas rather +than treating raw cross-model scores as a pure architecture ranking. + +Every supported backbone is evaluated under three representation protocols: +(i) a frozen global readout with a dimension-matched prediction head; +(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the +same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak +targets, context duration, training examples, head capacity, and evaluation +are held fixed whenever the backbone permits it. A failure of one LoRA recipe +is reported as that recipe's result, not as proof that the backbone cannot +adapt. + +\todo{Implement and audit one adapter for each planned backbone. For every +model, document model revision, processor, native audio/video rates, exact +hidden-state readout, causal/streaming semantics, context limit, dtype, +trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency. +Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot +expose a comparable causal state, report it as unsupported rather than inventing +a non-equivalent readout. Add future backbones only before the experiment +protocol is locked.} + +\subsection{Candidate local-evidence adaptation} + +The current hypothesis is that global omni states preserve broad semantic +context but may make recent prosody, activity boundaries, and individual +visual motion difficult for a shallow floor head to recover. From the most +recent $L_a$ seconds, a local encoder produces audio evidence $\vect A_t$ and, +optionally, an anonymous visible-person set $\mathcal P_t$. A generic candidate +adapter is \begin{equation} - \vect{p}_{t,i}=\vect{v}_{t,i}+ - \mathrm{CA}(\vect{v}_{t,i},\vect{A}_t,\vect{A}_t), - \label{eq:person} + \vect z^{(m)}_t=\vect g^{(m)}_t+ + \gamma_t H_\phi(\vect g^{(m)}_t,\vect A_t,\mathcal P_t), + \qquad \gamma_t\vert_{\mathrm{init}}=0, + \label{eq:adapter} \end{equation} -so every participant queries the same mixed audio rather than requiring -unavailable separated microphones. A masked set encoder models interactions -among $\mathcal{P}_t=\{\vect{p}_{t,i}\}_{i=1}^{N_t}$ and is permutation -equivariant \cite{lee2019settransformer}. Missing detections use a learned -\textsc{offscreen/null} element. In minibatches, sets are padded to -$N_{\max}$ with an attention mask; no slot-ID embedding is used, and track -slots are randomly permuted during training. +followed by the common prediction head. Local evidence is derived from the +same causal raw media, not close microphones, transcripts, speaker labels, or +future frames. -\todo{Implement and verify the shared track encoder, mixed-audio -cross-attention, OFFSCREEN token, padding mask, and numerical invariance under -track-slot permutation. Freeze $L_a$, visual crop history, $N_{\max}$, and -face-track confidence filtering from train-side experiments only.} +We evaluate candidates in increasing structural complexity: pooled recent +audio/vision; a capacity-matched temporal branch; a per-frame person set with +no cross-time identity; a shared track-wise temporal encoder; and person--audio +cross-attention. Only a candidate that improves the pre-registered event metric +over simpler and capacity-matched controls becomes the final method. If +person continuity is selected, temporary IDs route hidden state only---there +is no ID embedding, name recognition, or cross-recording identity---and the set +operator is permutation equivariant \cite{lee2019settransformer}. -\subsection{Role-query fusion} - -Mean pooling can erase which participant is speaking to whom. Instead, three -learned semantic queries represent floor state, address-to-wearer evidence, -and third-party interaction. Each query is conditioned on $\vect{g}_t$ and -cross-attends to the participant set: -\begin{align} - \vect{Q}_t &= [\vect{q}_{\mathrm{floor}}; - \vect{q}_{\mathrm{addr}}; - \vect{q}_{\mathrm{third}}]+W_g\vect{g}_t,\\ - \vect{R}_t &= \mathrm{MHA}(\vect{Q}_t,\widetilde{\mathcal P}_t, - \widetilde{\mathcal P}_t),\\ - \vect{z}_t &= \vect{g}_t+\tanh(\alpha)W_r\mathrm{vec}(\vect{R}_t), - \label{eq:fusion} -\end{align} -where $\alpha$ is initialized to zero. The zero-initialized residual makes the -starting model identical to the frozen global baseline and allows the visual -branch to be removed exactly. A two-layer prediction head maps $\vect{z}_t$ to -Eq.~\eqref{eq:task}. +\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable +parameters and effective local history. Compare a shared local encoder across +backbones (cross-model control) with each backbone's native local encoder +(within-model control); do not mix these protocols in one claimed delta. +For track candidates, specify visibility/age masks, gap/source resets, +OFFSCREEN handling, and global-rename invariance. Describe only the selected +architecture as the final method; move rejected candidates to ablations.} \subsection{Training objectives} -The main loss is class-balanced binary cross entropy over all horizons and -activity channels. When annotations are available, auxiliary heads predict -per-track active-speaker state and whether the current source addresses the -wearer: +The common main loss is class-balanced binary cross entropy over all horizons +and activity channels. This dense EgoSpeak target is the only supervision +required by the task. If a person-aware candidate survives the activity-only +comparison, an optional auxiliary head predicts current active-speaker state: \begin{equation} \mathcal{L}=\mathcal{L}_{\mathrm{act}}+ - \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}+ - \lambda_{\mathrm{addr}}\mathcal{L}_{\mathrm{addr}}. + \lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}. \label{eq:loss} \end{equation} -Ambiguous, group-addressed, and offscreen-target samples are masked from -$\mathcal{L}_{\mathrm{addr}}$, but remain in activity training. No language -model or pseudo-label is used to define primary test events. +Participant ID, target-of-speech, transcript, close-microphone audio, and +future frames are never model inputs. Target-of-speech may stratify qualitative +analysis but is not a predicted variable or a requirement for the primary +multi-other event definition. -\todo{Run the staged ablation: activity only; +ASD; +addressee. Select -$\lambda$ values and early stopping on a training-side calibration fold. Do -not inspect Sessions 1--3 during selection.} +\todo{First compare all representation/adaptation protocols with activity loss +only. Then ablate +ASD for the selected person-aware candidate. Add an +event-specific wearer onset/yield loss only as a separately reported extension. +Select all coefficients and early stopping on a training-side calibration +fold; do not inspect Sessions 1--3 during selection.} -\section{Experimental Setup} +\subsection{Track source and deployment contract} + +Person-aware candidates first use dataset-provided boxes and anonymous IDs to +isolate the value of local visual structure. A deployable version instead +obtains causal tracklets from a face/head detector followed by an online +multi-object tracker; no future-frame smoothing or cross-session recognition +is permitted. EasyCom IDs are continued only across genuinely adjacent source +segments and reset across redacted gaps. Ego4D IDs are local to a clip. + +\todo{If a person-aware candidate is selected, state precisely whether face or +head boxes are used. Report provided-track +coverage, mean/maximum people, occlusion gaps, track fragments, and source-gap +resets for both datasets. Add either one online detector/tracker experiment or +controlled box jitter, dropout, fragmentation, and ID-switch curves; otherwise +limit every deployment claim to ``given causal participant tracklets.''} + +\section{Experiments} \label{sec:setup} \textbf{Data and splits.} @@ -216,45 +253,83 @@ We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and Sessions 1--3 for final reporting. All architecture, checkpoint, threshold, and hyperparameter choices use training-side folds only. Inputs contain the camera stream and shared array audio available to the wearer; close-talking -microphones and target-of-speech labels are supervision only. We also report aggregate -activity forecasting on the EgoSpeak Ego4D split \cite{grauman2022ego4d}, but -do not claim addressee grounding there because comparable target annotations -are absent. +microphones, per-person voice labels, and target-of-speech labels are not model +inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D +split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies +clip-local anonymous participant tracks; its per-person voice coverage on the +433-clip EgoSpeak split must be audited before reporting the same multi-other +event analysis. + +\textbf{Backbone comparison.} +The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and +MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state +$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence +augmentation $G_m+L$. The primary diagnostic is the within-backbone change; +cross-backbone absolute scores are secondary because native tokenization, +pretraining, streaming interfaces, and hidden-state semantics differ. Common +local encoders test whether adaptation transfers across global backbones, +whereas native local encoders are a separate, model-specific protocol. All +rows share physical input cutoffs, targets, data splits, and evaluation. +Unsupported causal readouts are disclosed rather than replaced by offline or +future-conditioned features. + +\todo{Implement the three backbone adapters and validate state timestamps with +an impulse/cutoff test. Pre-register one matched PEFT budget and one common +local branch. Report native-vs-common local encoders in separate table panels. +Keep future model additions outside the confirmatory comparison once this +matrix is locked.} \textbf{Event construction.} -An \emph{addressed response} starts from a pure other-to-wearer utterance and -requires a wearer onset before an unrelated other onset. A \emph{third-party -exchange} starts from a pure other-to-other target while the wearer remains -silent. Group, none, multiple, and unknown targets form a separate diagnostic -stratum. Threshold $\tau_r$ is selected on the development split to obtain -addressed-response recall $r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary -metric is false wearer entries per minute of third-party opportunity at -matched recall; the headline operating point is $r=0.8$. We also report -addressed-response recall, signed detection latency, wearer yield recall, and -the original ten-horizon per-frame mAP. +We derive \emph{wearer onset} events directly from the wearer activity target. +A \emph{multi-other exchange} is a wearer-silent opportunity interval in which +at least two distinct non-wearer participant IDs contribute temporally ordered +speech activity. It is an observable interaction proxy, not a claim about the +speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the +development split to obtain wearer-onset recall +$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer +entries per minute of multi-other opportunity at matched onset recall; the +headline operating point is $r=0.8$. We also report signed onset latency, +wearer-yield recall/latency, and the original ten-horizon per-frame mAP. -\todo{Freeze the event pairing and competing-risk rules, including pre/post -windows, minimum silence, maximum response delay, overlap handling, and one -detection per event. Publish counts for train/test positive and negative -events, visibility, duration, and excluded target types. Treat all previously -viewed Sessions 1--3 scores as exploratory; lock the final protocol and model -before the confirmatory three-seed run.} +\todo{Freeze the event pairing and competing-risk rules, including the +multi-other alternation window, minimum speech/silence, overlap, maximum onset +delay, duplicate detections, one detection per event, track visibility, source +boundaries, and group/offscreen cases. Publish train/test onset counts, +multi-other events/minutes, number of distinct visible speakers, and exclusion +counts. Manually audit at least 100 events. Treat all previously viewed +Sessions 1--3 scores as exploratory; lock the protocol and model before the +confirmatory three-seed run.} \textbf{Baselines and controls.} -We compare: global omni state (G); G plus recent audio (G+A); G+A plus pooled -visual context (G+A+V-pool); a parameter-matched temporal branch without -person identity; and the proposed set branch. Required counterfactuals replace -correct tracks with (i) wrong participants from the same scene, (ii) -time-shifted tracks, and (iii) zeros. These controls share parameter count, -audio history, and optimization budget. +For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$ +plus recent local audio; and $G_m$ plus pooled local audio--visual evidence. +This compact matrix tests the representation hypothesis before investing in a +complex fusion module. If local vision survives three seeds and the event +gate, we then compare a parameter-matched temporal branch without person +continuity, a separately trained per-frame/no-ID set branch, and a +track-consistent branch. On a frozen track-aware checkpoint, a clip-consistent +global renaming must leave outputs unchanged, whereas \emph{broken +continuity} periodically remaps IDs while preserving every crop, timestamp, +audio sample, and target. Time-shifted and removed vision provide additional +alignment controls. + +\todo{Define the broken-continuity corruption before evaluation: remap only at +pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not +call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops, +frame counts, audio, labels, and visibility masks. Train the no-ID/reset +baseline from scratch with matched parameters; evaluate all test-time +counterfactuals using one frozen checkpoint and one frozen threshold.} \textbf{Implementation and statistics.} -All models run at 5~Hz with an 8-s causal memory. The 3B Thinker is frozen and -local audio/face histories are encoded causally. We report three seeds and -mean$\pm$standard deviation. Confidence intervals for event metrics use paired -cluster bootstrap over sessions/conversation blocks rather than treating -adjacent frames as independent. Runtime includes P50/P95 decision latency, -peak memory, and trainable parameter count. +All prediction heads run at 5~Hz and use an 8-s causal observation window; +model-specific deviations required by a native streaming interface are stated +explicitly. In the frozen protocol the omni backbone does not update. In PEFT +and local protocols, trainable parameters and optimizer exposure are reported +separately. Every temporal state resets at non-contiguous source boundaries. +We report three seeds and mean$\pm$standard deviation. Confidence intervals for +event metrics use paired cluster bootstrap over sessions/conversation blocks +rather than treating adjacent frames as independent. Runtime includes P50/P95 +decision latency, throughput, peak memory, and trainable parameter count. \todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions, $L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter @@ -264,95 +339,138 @@ count, hardware, and real-time latency after the architecture is frozen.} \label{sec:results} \begin{table}[t] -\caption{Current aggregate evidence on EasyCom test Sessions 1--3. Values are -percentages from one seed and are motivation, not the final main result.} +\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test +Sessions 1--3. Values are percentages from one seed, not a cross-backbone or +confirmatory result.} \label{tab:preliminary} \centering \small +\resizebox{\columnwidth}{!}{% \begin{tabular}{lccc} \hline Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\ \hline -G+A & 81.72 & 85.07 & 65.11\\ -G+A+V-pool & 82.03 & 85.09 & 65.40\\ +Qwen-3B $G$+local A & 81.72 & 85.07 & 65.11\\ +Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\ \hline $\Delta$ visual & +0.30 & +0.02 & +0.29\\ \hline -\end{tabular} +\end{tabular}} \end{table} Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient. Pooled vision gives a small gain, while immediate wearer AP is nearly unchanged. This is compatible with either weak but useful social grounding or generic visual regularization; only matched-recall event evaluation and -counterfactual identity controls can distinguish them. +track-continuity controls can distinguish them. +A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen +readout and 56.20 for one LoRA recipe. Because that diagnostic used a different +feature protocol, it is not combined numerically with +Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT +study remains necessary. \begin{table}[t] -\caption{Primary EasyCom event evaluation. Lower $\mathrm{FP}_{\mathrm{3p}}$ -is better; all rows use the same development-selected operating protocol.} +\caption{Planned cross-backbone EasyCom event evaluation. Lower +$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same +development-selected operating protocol. ``Local'' denotes the train-side +selected common local-evidence candidate.} \label{tab:main} \centering \small \resizebox{\columnwidth}{!}{% \begin{tabular}{lccccc} \hline -Model & $\mathrm{FP}_{\mathrm{3p}}$/min@$R=.8$ & $R_{\mathrm{addr}}$ & -Latency & Yield R & Mean mAP\\ +Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R & +Onset latency & Yield R & Mean mAP\\ \hline -G & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -G+A & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -G+A+V-pool & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -Temporal control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -\method{} & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -\method{}+ASD & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ -\method{}+Addr & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen-3B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen-3B PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen-3B + local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen-7B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +Qwen-7B PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +MiniCPM-o frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ +MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ \hline \end{tabular}} \end{table} \todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three -seeds, plus 95\% paired cluster-bootstrap confidence intervals for the -proposed-vs-G+A difference. The paper's main claim is supported only if -third-party FP/min decreases at matched recall without worse response latency -or wearer-yield recall.} +seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs +PEFT and frozen vs local within each backbone. A general omni-representation +claim requires a consistent pattern across at least two backbone families; an +effect confined to Qwen must be stated as Qwen-specific. The behavior claim is +supported only if multi-other FP/min decreases at matched wearer-onset recall +without worse onset latency or wearer-yield recall. If the selected local +candidate uses tracks, also report its paired intervals against no-ID and +temporal-capacity controls.} \begin{table}[t] -\caption{Visual grounding counterfactual on the frozen best model.} +\caption{Conditional track-continuity counterfactual on the best track-aware +model. This table is retained only if train-side selection chooses such a +candidate; all rows reuse one frozen checkpoint.} \label{tab:counterfactual} \centering \small +\resizebox{\columnwidth}{!}{% \begin{tabular}{lccc} \hline -Visual evidence & $\mathrm{FP}_{\mathrm{3p}}$/min & Addr. recall & 0.2-s mAP\\ +Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\ \hline Correct tracks & \todo{--} & \todo{--} & \todo{--}\\ -Wrong person & \todo{--} & \todo{--} & \todo{--}\\ +Global ID rename & \todo{--} & \todo{--} & \todo{--}\\ +Broken continuity & \todo{--} & \todo{--} & \todo{--}\\ Time shifted & \todo{--} & \todo{--} & \todo{--}\\ Removed & \todo{--} & \todo{--} & \todo{--}\\ \hline -\end{tabular} +\end{tabular}} \end{table} -Table~\ref{tab:counterfactual} is the identity test. A true social-grounding -effect should be largest for correct tracks and should collapse for wrong or -time-shifted people. If every visual variant improves equally, the defensible -conclusion is additional capacity/history, not addressee grounding. +If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the +continuity test. Global renaming must be +numerically invariant because IDs carry no semantics. If the model exploits +person history, correct tracks should outperform broken continuity even though +both contain exactly the same per-frame people. If broken, time-shifted, and +removed variants perform similarly to correct tracks, the defensible +conclusion is additional capacity or generic visual context, not useful +person-continuous reasoning. -\todo{Add one compact error analysis split by target visibility, number of -visible people, overlap, and backchannel/interruption proxy. Include event -counts with every rate. Move negative LoRA/7B/ROMA diagnostics to supplementary -material unless they directly explain the final method.} +\todo{Add one compact error analysis split by model family, track visibility, +number of visible people, overlap, track age/fragmentation, and +backchannel/interruption proxy. Include event counts with every rate. Move +rejected PEFT recipes and fusion candidates to supplementary material unless +they directly explain the selected method.} + +\textbf{Qualitative evidence.} +For a multi-other exchange, visualize per-person speech activity and track IDs, +wearer ground truth, the frozen-global and selected-local wearer-entry scores, +and the single frozen threshold on one physical time axis. If the selected +candidate is track-aware, also show its broken-continuity score. A useful +example shows the frozen baseline rising at an other-speaker boundary while +the selected adapter remains below threshold; a companion positive example +must show that a real wearer onset is not suppressed. + +\todo{Do not hand-pick only successful clips. Before viewing the final test +plots, freeze a deterministic selection rule and show: largest paired +improvement, near-median improvement, one failure, and one correctly retained +wearer onset. Include full local timelines and source frame times, not cropped +screenshots without context. State that examples illustrate the aggregate +event result and cannot establish it alone.} \section{Conclusion} \label{sec:conclusion} -We study when an egocentric agent should enter a multiparty floor, not merely -whether someone will speak. The proposed formulation retains dense future -activity prediction while grounding wearer entry in a variable set of visible -participants. Its event protocol tests the intended behavior directly: -reducing intrusions into third-party exchanges at matched legitimate-response -recall. \todo{Replace with one quantitative conclusion and one limitation; -do not claim success from aggregate mAP alone.} +We study when an egocentric agent should enter or yield a multiparty floor, not +merely whether someone will speak. The central question is model-agnostic: +whether frozen or parameter-efficiently adapted omni representations expose +the fine-grained floor cues required by a 5-Hz controller, and whether causal +local evidence closes any gap. Dense activity forecasting supplies a common +target, while matched-recall event evaluation tests whether an apparent gain +actually prevents intrusions during multi-other exchanges. Current Qwen-3B +results are exploratory motivation only. \todo{After the controlled study, +state which backbone/adapter was selected, whether the representation pattern +generalized beyond Qwen, the quantitative false-entry and onset/yield results, +and the main deployment limitation. Do not infer success from aggregate mAP or +selected examples alone.} \bibliographystyle{IEEEbib} \bibliography{references} diff --git a/manuscript/main/references.bib b/manuscript/main/references.bib index a61f3e6..06483e2 100644 --- a/manuscript/main/references.bib +++ b/manuscript/main/references.bib @@ -48,6 +48,13 @@ year = {2025} } +@article{cui2026minicpmo45, + author = {Junbo Cui and Bokai Xu and Chongyi Wang and Tianyu Yu and Weiyue Sun and Yingjing Xu and Tianran Wang and Zhihui He and Wenshuo Ma and Tianchi Cai and others}, + title = {{MiniCPM-o 4.5}: Towards Real-Time Full-Duplex Omni-Modal Interaction}, + journal = {arXiv preprint arXiv:2604.27393}, + year = {2026} +} + @inproceedings{lee2019settransformer, author = {Juho Lee and Yoonho Lee and Jungtaek Kim and Adam Kosiorek and Seungjin Choi and Yee Whye Teh}, title = {Set Transformer: A Framework for Attention-Based Permutation-Invariant Neural Networks},