docs: generalize manuscript across omni backbones
This commit is contained in:
Xin Wang
@@ -28,3 +28,29 @@ main/
|
|||||||
The author/affiliation, final results, Method, Experimental Setup, Results and
|
The author/affiliation, final results, Method, Experimental Setup, Results and
|
||||||
Conclusion are still explicit TODOs. Do not turn planned experiments into
|
Conclusion are still explicit TODOs. Do not turn planned experiments into
|
||||||
reported findings.
|
reported findings.
|
||||||
|
|
||||||
|
The current paper thesis is model-agnostic and deliberately narrower than
|
||||||
|
addressee prediction. It asks whether general-purpose omni representations
|
||||||
|
expose the fine-grained evidence needed for 200 ms wearer floor-action
|
||||||
|
projection, whether PEFT can recover it, and whether causal local audio/person
|
||||||
|
evidence provides a reliable complement. The initial planned backbone set is:
|
||||||
|
|
||||||
|
- Qwen2.5-Omni-3B;
|
||||||
|
- Qwen2.5-Omni-7B;
|
||||||
|
- MiniCPM-o 4.5.
|
||||||
|
|
||||||
|
The backbone and final fusion method are **not frozen yet**. Current Qwen-3B
|
||||||
|
numbers are exploratory motivation, not evidence that every omni model has the
|
||||||
|
same limitation. Compare frozen, PEFT, and local-evidence protocols within each
|
||||||
|
backbone; report common local encoders separately from model-native local
|
||||||
|
encoders. Raw cross-model scores are secondary because the models expose
|
||||||
|
different streaming/readout interfaces.
|
||||||
|
|
||||||
|
Track-consistent person fusion remains one candidate after pooled and
|
||||||
|
capacity-matched controls. If selected, dataset participant IDs route
|
||||||
|
short-term causal state only; they are not semantic inputs. The publication
|
||||||
|
gate is an event-level reduction in multi-other false entries at matched
|
||||||
|
wearer-onset recall without degrading onset latency or wearer-yield behavior.
|
||||||
|
A general omni claim requires a consistent result across at least two backbone
|
||||||
|
families; otherwise the paper must state a model-specific finding. Selected
|
||||||
|
examples are explanatory evidence only.
|
||||||
|
|||||||
@@ -4,138 +4,157 @@ This is an internal execution checklist. It is not part of the submitted PDF.
|
|||||||
|
|
||||||
## Reviewer verdict on the current draft
|
## Reviewer verdict on the current draft
|
||||||
|
|
||||||
The research question is potentially publishable: **at matched wearer-response
|
The potentially publishable question is now model-agnostic: **do
|
||||||
recall, can correctly aligned person-level vision reduce false wearer entries
|
general-purpose omni representations expose the fine-grained evidence needed
|
||||||
during third-party conversation without degrading legitimate wearer responses?**
|
for 200 ms egocentric floor action, and can causal local evidence reduce false
|
||||||
|
wearer entries at matched onset recall when frozen readout or PEFT is
|
||||||
|
insufficient?**
|
||||||
|
|
||||||
The contribution is not “the first audio-visual multiparty turn model.” VAP,
|
The contribution is not “the first audio-visual multiparty turn model.” VAP,
|
||||||
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
|
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
|
||||||
gap is continuous, causal floor-entry projection for an **invisible camera
|
gap is continuous, causal floor-action projection for an invisible camera
|
||||||
wearer**, conditioned on addressee evidence from a variable participant set.
|
wearer, combined with a controlled diagnosis of global omni states and a
|
||||||
|
behavior-aligned multi-other event protocol.
|
||||||
|
|
||||||
The current aggregate result (G+A+pooled-V minus G+A: +0.30 points at 0.2 s,
|
Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
|
||||||
+0.29 mean mAP, one seed) motivates the question but does not establish social
|
prove a limitation shared by omni models nor establish track-consistent fusion
|
||||||
grounding. Acceptance depends on the event protocol and identity controls.
|
as the final method. The planned initial backbone set is Qwen2.5-Omni-3B,
|
||||||
|
Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
|
||||||
|
|
||||||
## P0 — Freeze the task before training a new model
|
## P0 — Freeze the task before the backbone sweep
|
||||||
|
|
||||||
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
|
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
|
||||||
of the ten 0.2 s horizons.
|
of the ten 0.2 s horizons.
|
||||||
- [ ] Freeze addressed-response and third-party event definitions.
|
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
|
||||||
- [ ] Add competing-risk rules: an addressed response ends at the first wearer
|
- [ ] Add competing-risk rules: a candidate event ends at the first matched
|
||||||
onset, unrelated-other onset, timeout, or invalid annotation boundary.
|
wearer action, incompatible floor change, timeout, or invalid source boundary.
|
||||||
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
|
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate
|
||||||
detections, clip boundaries, group/none/multiple/unknown targets, and
|
detections, source discontinuities, track visibility, offscreen speakers,
|
||||||
offscreen speakers.
|
and whether target-of-speech strata are diagnostic only.
|
||||||
- [ ] Produce train/test event-count tables, duration, class prevalence, and
|
- [ ] Produce train/test event-count tables, duration, class prevalence, and
|
||||||
visibility coverage before examining model scores.
|
visibility coverage before examining model scores.
|
||||||
- [ ] Mark all already viewed Sessions 1–3 results as exploratory. From this
|
- [ ] Mark all already viewed Sessions 1–3 results as exploratory. Lock the
|
||||||
point forward, lock the event protocol and select thresholds, checkpoints,
|
protocol and select thresholds, checkpoints, and hyperparameters using a
|
||||||
and all hyperparameters using a training-side fold or cross-fitting over
|
training-side fold or cross-fitting over Sessions 4–12.
|
||||||
Sessions 4–12. Run the fixed final model on Sessions 1–3 only after that
|
|
||||||
lock; do not describe the historically viewed split as untouched.
|
|
||||||
|
|
||||||
**Gate:** two independent code paths or a manual sample audit must agree on at
|
**Gate:** two independent code paths or a manual sample audit must agree on at
|
||||||
least 100 randomly selected events. Do not start the main model sweep before
|
least 100 randomly selected events. Publish the number of unique other IDs and
|
||||||
this gate passes.
|
multi-other opportunity minutes before the main sweep.
|
||||||
|
|
||||||
## P1 — Establish matched baselines
|
## P1 — Audit each backbone interface
|
||||||
|
|
||||||
Run three seeds with identical optimizer, history, parameter budget, and event
|
For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record:
|
||||||
evaluator:
|
|
||||||
|
|
||||||
1. G: frozen streaming omni state.
|
- [ ] exact model/code revision, processor, dtype, and hardware;
|
||||||
2. G+A: global state plus recent audio.
|
- [ ] native audio/video sampling and streaming/chunk semantics;
|
||||||
3. G+A+V-pool: the current pooled visual baseline.
|
- [ ] exact causal hidden-state readout and dimensionality;
|
||||||
4. Temporal control: G+A plus a parameter-matched non-person temporal branch.
|
- [ ] context limit, cache/reset behavior, and source-boundary handling;
|
||||||
|
- [ ] a cutoff/impulse test proving no future frame or sample enters the state;
|
||||||
|
- [ ] P50/P95 latency, throughput, and peak memory.
|
||||||
|
|
||||||
Report original ten-horizon AP/mAP and the event metrics. Confirm that changing
|
Do not force an offline or future-conditioned feature into the common table. If
|
||||||
batching, cached features, or track padding does not change baseline outputs.
|
a model cannot expose a comparable causal state, report that protocol as
|
||||||
|
unsupported. Raw scores are not a pure architecture ranking; emphasize
|
||||||
|
within-backbone deltas.
|
||||||
|
|
||||||
**Gate:** baseline standard deviations are known, and the event evaluator
|
## P2 — Establish the compact representation matrix
|
||||||
reproduces identical scores from saved predictions.
|
|
||||||
|
|
||||||
## P2 — Implement the minimum person-set model
|
Run three seeds per supported backbone with the same data, physical cutoffs,
|
||||||
|
target, prediction-head capacity, optimizer exposure, and evaluator:
|
||||||
|
|
||||||
- [ ] Shared temporal encoder for every face track.
|
1. Frozen global causal state.
|
||||||
- [ ] Each person representation cross-attends to the same recent mixed-audio
|
2. Pre-registered PEFT/LoRA budget.
|
||||||
token sequence; no separated/close-mic audio at inference.
|
3. Global state plus recent local audio.
|
||||||
- [ ] Masked permutation-equivariant set encoder.
|
4. Global state plus pooled local audio and vision.
|
||||||
- [ ] Learned OFFSCREEN/NULL element.
|
|
||||||
- [ ] Fixed floor/address/third-party role queries.
|
|
||||||
- [ ] Zero-initialized gated residual into the frozen global state.
|
|
||||||
- [ ] Unit tests for variable `N`, empty visible set, missing frames, track-slot
|
|
||||||
permutation, padding invariance, and strict causal cutoff.
|
|
||||||
|
|
||||||
**Gate:** with residual gate zero, logits are numerically identical to G+A; a
|
Use a common local encoder as the primary cross-backbone control. If native
|
||||||
random permutation of track slots leaves logits unchanged.
|
local encoders are tested, report them in a separate within-model panel. Do not
|
||||||
|
mix common-encoder and native-encoder deltas in one claim.
|
||||||
|
|
||||||
## P3 — Train only the ablations that answer the claim
|
**Gate:** baseline standard deviations are known; the evaluator reproduces
|
||||||
|
identical scores from saved predictions; parameter counts and training budgets
|
||||||
|
are reported. A single failed LoRA recipe is evidence about that recipe, not
|
||||||
|
proof that the model cannot adapt.
|
||||||
|
|
||||||
Use the same three seeds:
|
## P3 — Select local fusion only after P2
|
||||||
|
|
||||||
1. WearerSet with activity loss only.
|
If local evidence gives a reliable event-level gain, compare candidates in
|
||||||
2. WearerSet + per-person active-speaker auxiliary loss.
|
increasing complexity:
|
||||||
3. WearerSet + active-speaker + pure-target addressee auxiliary loss.
|
|
||||||
|
|
||||||
Mask group/unknown/offscreen target annotations from addressee loss. They may
|
1. Pooled local audio/vision.
|
||||||
remain in the activity loss. Do not use LLM pseudo-labels as primary gold.
|
2. Parameter-matched temporal branch.
|
||||||
|
3. Per-frame person set with no cross-time identity.
|
||||||
|
4. Shared track-wise temporal encoder.
|
||||||
|
5. Person–audio cross-attention.
|
||||||
|
6. Optional per-person active-speaker auxiliary loss, only after selecting a
|
||||||
|
person-aware activity-only model.
|
||||||
|
|
||||||
Stop after these stages unless a clear event-level gain appears. Broad Thinker
|
Keep the dense EgoSpeak wearer/others target as the primary task. Target of
|
||||||
LoRA, 7B, and ROMA experiments are diagnostics, not the core method, because
|
speech is analysis-only and never model input. Describe only the train-side
|
||||||
the existing runs show negative transfer or no consistent 0.2 s benefit.
|
winner as the final method; move rejected candidates to ablations.
|
||||||
|
|
||||||
## P4 — Run causal visual counterfactuals
|
For a track-aware candidate, temporary IDs route hidden state only: no ID
|
||||||
|
embedding, cross-recording identity, or fixed seating slot. Unit-test variable
|
||||||
|
person count, empty sets, padding, slot permutation, global ID renaming,
|
||||||
|
source/gap resets, and strict causal cutoffs.
|
||||||
|
|
||||||
For the frozen best checkpoint, evaluate:
|
## P4 — Run visual counterfactuals conditionally
|
||||||
|
|
||||||
|
Only if a track-aware candidate is selected, evaluate one frozen checkpoint
|
||||||
|
with:
|
||||||
|
|
||||||
- correct person tracks;
|
- correct person tracks;
|
||||||
- wrong-person tracks from the same scene;
|
- clip-consistent global renaming, which must be invariant;
|
||||||
|
- deliberately broken cross-time identity with unchanged per-frame crops;
|
||||||
- time-shifted tracks with unchanged audio;
|
- time-shifted tracks with unchanged audio;
|
||||||
- removed/zero visual features;
|
- removed visual features;
|
||||||
|
- fixed-rate ID switch, box dropout, and fragmentation;
|
||||||
- visibility-conditioned subsets.
|
- visibility-conditioned subsets.
|
||||||
|
|
||||||
All controls must reuse the same model weights. Retraining each control would
|
The separately trained no-ID model belongs in the main ablation table. Do not
|
||||||
mix representation and optimization effects.
|
retrain test-time counterfactuals.
|
||||||
|
|
||||||
**Acceptance gate:** at matched addressed-response recall, correct vision must
|
## P5 — Statistics and evidence
|
||||||
reduce third-party false entries more than wrong/time-shifted/removed vision,
|
|
||||||
without worse response latency or wearer-yield recall. Aggregate mAP alone is
|
|
||||||
not sufficient.
|
|
||||||
|
|
||||||
## P5 — Statistics and analysis
|
- [ ] Three seeds, mean ± sample standard deviation.
|
||||||
|
- [ ] Paired cluster bootstrap by session/conversation block for 95% intervals.
|
||||||
|
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
|
||||||
|
- [ ] False wearer entries per multi-other opportunity minute, onset latency,
|
||||||
|
wearer-yield recall/latency, and original ten-horizon AP/mAP.
|
||||||
|
- [ ] Error strata: backbone, visibility, number of visible people, overlap,
|
||||||
|
offscreen activity, track age/fragmentation, and transition proxy.
|
||||||
|
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
|
||||||
|
same multi-other analysis.
|
||||||
|
- [ ] Provided-track and online-tracker conditions named separately.
|
||||||
|
|
||||||
- [ ] Three training seeds, mean ± sample standard deviation.
|
Qualitative cases follow a frozen selection rule and include the largest paired
|
||||||
- [ ] Paired cluster bootstrap by session/conversation block for 95% confidence
|
improvement, a median case, a failure, and one preserved true wearer onset.
|
||||||
intervals; do not bootstrap adjacent 5 Hz frames independently.
|
Examples explain rates; they never replace them.
|
||||||
- [ ] Operating points at addressed recall 0.7, 0.8, and 0.9; headline at 0.8.
|
|
||||||
- [ ] Error strata: target visibility, number of visible people, overlap,
|
|
||||||
backchannel/interruption proxy, and offscreen target.
|
|
||||||
- [ ] P50/P95 decision latency, throughput, peak GPU memory, and trainable
|
|
||||||
parameters.
|
|
||||||
- [ ] Ego4D only as secondary aggregate activity generalization; it cannot
|
|
||||||
validate addressee grounding without comparable target labels.
|
|
||||||
|
|
||||||
## Paper assembly order
|
## Paper assembly order
|
||||||
|
|
||||||
1. Finalize the event protocol and insert counts in Experimental Setup.
|
1. Freeze and audit the event protocol.
|
||||||
2. Produce the main matched-recall table before polishing prose.
|
2. Implement and causally audit all backbone readouts.
|
||||||
3. Produce the correct/wrong/time-shift/remove counterfactual table.
|
3. Complete the compact frozen/PEFT/local matrix.
|
||||||
4. Draw one compact method figure with causal cutoffs and the variable set.
|
4. Decide whether the paper can support a cross-family or only model-specific
|
||||||
5. Collapse the preliminary aggregate table into the final main table if page
|
representation claim.
|
||||||
pressure is high.
|
5. Select a local fusion candidate using training-side event metrics.
|
||||||
6. Add one short failure analysis, not a catalog of all earlier experiments.
|
6. Run conditional visual/track counterfactuals.
|
||||||
7. Replace the abstract and conclusion TODOs with exact numbers and confidence
|
7. Produce the cross-backbone event table and paired intervals.
|
||||||
intervals.
|
8. Draw one compact final system figure; remove rejected methods from the main
|
||||||
8. Remove every visible `\todo{}` marker and verify the official ICASSP 2027
|
narrative.
|
||||||
template/page limit once released.
|
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
|
||||||
|
|
||||||
## Stop/go decision
|
## Stop/go decisions
|
||||||
|
|
||||||
- **Go:** correct person-level vision provides a statistically credible
|
- **General claim:** at least two backbone families show the same frozen/PEFT
|
||||||
event-level advantage over G+A, pooled-V, temporal capacity control, and
|
weakness and a reliable local-evidence improvement.
|
||||||
wrong/time-shifted vision.
|
- **Model-specific claim:** the effect appears only within Qwen; say so and do
|
||||||
- **Reframe:** all visual variants improve similarly. Claim better local
|
not generalize to all omni models.
|
||||||
temporal context, not addressee grounding.
|
- **Audio-local reframe:** local audio improves but all visual variants fail;
|
||||||
- **Stop this method:** aggregate mAP rises but third-party false entries do
|
claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
|
||||||
not fall at matched recall, or legitimate response latency/recall degrades.
|
- **Track-aware method:** correct continuity beats pooled vision, temporal
|
||||||
|
capacity, no-ID/reset, broken continuity, and time-shifted controls at matched
|
||||||
|
recall without worse onset/yield behavior.
|
||||||
|
- **Stop the adapter:** aggregate mAP rises but multi-other false entries do not
|
||||||
|
fall, or legitimate onset/yield behavior degrades.
|
||||||
|
|||||||
+313
-195
@@ -7,11 +7,11 @@
|
|||||||
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
|
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
|
||||||
|
|
||||||
\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
|
\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO: #1]}}}
|
||||||
\newcommand{\method}{WearerSet}
|
|
||||||
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
\newcommand{\vect}[1]{\boldsymbol{#1}}
|
||||||
|
\setlength{\emergencystretch}{2em}
|
||||||
|
|
||||||
\title{NOT EVERY OPEN FLOOR IS MINE: ADDRESSEE-CONDITIONED FLOOR-ENTRY
|
\title{WHEN SHOULD AN EGOCENTRIC AGENT SPEAK? DIAGNOSING AND ADAPTING
|
||||||
PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
|
OMNI REPRESENTATIONS FOR MULTIPARTY FLOOR ACTION}
|
||||||
|
|
||||||
% TODO: Insert the final author list and affiliations after the ICASSP 2027
|
% TODO: Insert the final author list and affiliations after the ICASSP 2027
|
||||||
% review policy and author order are finalized.
|
% review policy and author order are finalized.
|
||||||
@@ -23,55 +23,59 @@ PROJECTION IN EGOCENTRIC MULTIPARTY CONVERSATION}
|
|||||||
\maketitle
|
\maketitle
|
||||||
|
|
||||||
\begin{abstract}
|
\begin{abstract}
|
||||||
An egocentric conversational agent must decide not only when a speaking
|
Before generating a response, an egocentric conversational agent must decide
|
||||||
opportunity arises, but also whether that opportunity belongs to it. Existing
|
whether to speak or keep listening in a scene that may contain several other
|
||||||
speech-activity projection collapses all non-wearer speakers, allowing strong
|
people. General-purpose omni models offer broad streaming audio--visual
|
||||||
frame-level accuracy to coexist with intrusive entries during conversations
|
context, but it is unclear whether their global representations expose the
|
||||||
between bystanders. We formulate addressee-conditioned floor-entry projection:
|
fine-grained local cues required for a decision every 200~ms. We study this
|
||||||
dense causal forecasts of wearer and non-wearer activity at 200-ms resolution,
|
question across representative omni backbones and adaptation regimes: frozen
|
||||||
conditioned on a variable-cardinality set of visible participants. Our model
|
global readout, parameter-efficient tuning, and causal augmentation with
|
||||||
augments a frozen streaming omni-model with person-aligned audio--visual
|
recent audio and visible-person histories. The human camera wearer serves as a
|
||||||
representations and permutation-invariant role queries for floor state,
|
behavioral proxy for the agent, while the dense EgoSpeak target remains the
|
||||||
wearer address, and third-party interaction. We further introduce an EasyCom
|
common output. An EasyCom event protocol reports false wearer entries during
|
||||||
event protocol that separates addressed-to-wearer responses from third-party
|
multi-other exchanges at matched wearer-onset recall, and track/alignment
|
||||||
exchanges and reports false entries at matched response recall and decision
|
controls test whether visual gains follow person continuity rather than added
|
||||||
latency. Correct-, wrong-person, time-shifted, and removed-video controls test
|
capacity. Exploratory Qwen2.5-Omni-3B results motivate the study but do not yet
|
||||||
whether a gain follows social grounding rather than model capacity.
|
establish a limitation shared by omni models or determine the final adapter.
|
||||||
\todo{Replace this sentence with the main result: FP/min reduction at 80\%
|
\todo{After the train-side study is locked, name the selected backbones and
|
||||||
response recall, response latency/recall non-degradation, and 95\% confidence
|
adapter and insert: frozen/PEFT/local deltas, multi-other false-entry reduction
|
||||||
interval.}
|
at 80\% onset recall, onset/yield non-degradation, and 95\% intervals.}
|
||||||
\end{abstract}
|
\end{abstract}
|
||||||
|
|
||||||
\begin{keywords}
|
\begin{keywords}
|
||||||
turn-taking, speech initiation, addressee detection, egocentric video,
|
turn-taking, speech initiation, egocentric video, omni models,
|
||||||
audio-visual learning
|
audio-visual adaptation
|
||||||
\end{keywords}
|
\end{keywords}
|
||||||
|
|
||||||
\section{Introduction}
|
\section{Introduction}
|
||||||
\label{sec:intro}
|
\label{sec:intro}
|
||||||
|
|
||||||
Conversational agents must continuously decide whether to speak or keep
|
Consider an embodied conversational agent that observes the scene through an
|
||||||
listening. In a dyadic exchange, a pause or an apparent end of turn is often
|
egocentric camera and hears mixed environmental audio. Before response
|
||||||
treated as an invitation to respond. This assumption fails in multiparty
|
generation and speech synthesis, a causal floor controller must decide every
|
||||||
interaction: one participant may finish an utterance addressed to another
|
200~ms whether the agent should continue listening, begin speaking, or yield.
|
||||||
participant, and an egocentric agent that reacts only to acoustic availability
|
In a dyadic exchange, a pause or an apparent end of turn may be an invitation
|
||||||
will enter a floor that was never offered to it. Such an error is socially more
|
to respond. This assumption fails in multiparty interaction: one participant
|
||||||
disruptive than a small framewise prediction error. A useful agent must
|
may hand the floor to another visible participant, and an agent that reacts
|
||||||
therefore distinguish \emph{when the floor may change} from \emph{whether the
|
only to acoustic availability may interrupt an exchange that does not involve
|
||||||
change is directed to the camera wearer}.
|
it. We study this floor-action controller; response content generation and
|
||||||
|
closed-loop conversational effects are outside our scope. The observed human
|
||||||
|
camera wearer is a behavioral proxy rather than an equivalent autonomous
|
||||||
|
agent: its activity supplies human demonstrations of floor timing under the
|
||||||
|
same first-person observation geometry.
|
||||||
|
|
||||||
Voice Activity Projection (VAP) models future joint speech activity and
|
Voice Activity Projection (VAP) models future joint speech activity and
|
||||||
derives turn-taking events without manually labeling each event
|
derives turn-taking events without manually labeling each event
|
||||||
\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
|
\cite{ekstedt2022vap}. EgoSpeak extends future activity prediction to causal,
|
||||||
untrimmed egocentric video, forecasting wearer and aggregate non-wearer
|
untrimmed egocentric video, forecasting wearer and aggregate non-wearer
|
||||||
activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
|
activity over the next two seconds \cite{kim2025egospeak}. Its role-relative
|
||||||
output is attractive for a wearable agent, but an ``others'' channel cannot
|
output is attractive for a wearable agent, but an ``others'' channel discards
|
||||||
distinguish an utterance addressed to the wearer from an exchange between
|
whether the same or different visible people produced the recent activity and
|
||||||
bystanders. Conventional per-frame average precision can consequently be
|
how their behavior evolved across time. Conventional per-frame average
|
||||||
dominated by ordinary activity while hiding precisely the false-entry behavior
|
precision can consequently be dominated by ordinary activity while hiding
|
||||||
that visual social cues should prevent. Event-oriented evaluation likewise
|
precisely the false-entry behavior that person-consistent visual cues may
|
||||||
emphasizes recall, false alarms, and latency rather than a single frame-level
|
prevent. Event-oriented evaluation likewise emphasizes recall, false alarms,
|
||||||
score \cite{jiang2026turnbench}.
|
and latency rather than a single frame-level score \cite{jiang2026turnbench}.
|
||||||
|
|
||||||
Related work supplies complementary parts of the problem. Egocentric
|
Related work supplies complementary parts of the problem. Egocentric
|
||||||
talking-to-me detection predicts whether a completed utterance addresses the
|
talking-to-me detection predicts whether a completed utterance addresses the
|
||||||
@@ -79,46 +83,44 @@ camera wearer and benefits from long-term social context
|
|||||||
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
|
\cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face
|
||||||
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
|
tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}.
|
||||||
The former is generally utterance-level rather than continuous anticipation;
|
The former is generally utterance-level rather than continuous anticipation;
|
||||||
the latter does not ask whether an invisible camera wearer should enter. Our
|
the latter predicts visible participants rather than the floor action of an
|
||||||
gap is therefore narrower than ``audio--visual multiparty turn taking'': we
|
invisible camera wearer. Our gap is therefore narrower than ``audio--visual
|
||||||
study continuous, wearer-relative floor action when the candidate responder is
|
multiparty turn taking'': we study whether short-term anonymous participant
|
||||||
behind the camera and the number of visible participants changes over time.
|
continuity helps direct, continuous wearer-action projection when the candidate
|
||||||
|
responder is behind the camera and the visible participant set changes.
|
||||||
|
|
||||||
We call this task \emph{addressee-conditioned egocentric floor-entry
|
We call this task \emph{egocentric floor-action projection}. The study asks
|
||||||
projection}. A global streaming audio--video representation supplies broad
|
four questions: (RQ1) how much wearer activity is linearly or shallowly
|
||||||
context, while a set branch preserves individual visual histories and their
|
readable from a frozen global omni state; (RQ2) whether standard
|
||||||
alignment to recent mixed audio. Fixed semantic role queries compress a
|
parameter-efficient tuning recovers missing performance; (RQ3) whether causal
|
||||||
variable participant set into wearer-address and third-party evidence without
|
local audio and person-level visual evidence provide complementary information;
|
||||||
assuming a fixed seating order. EasyCom is suitable because it provides
|
and (RQ4) whether any aggregate gain reduces false agent entries without
|
||||||
synchronized egocentric audio--video, per-person tracks, voice activity, and
|
making the controller globally passive. EasyCom is suitable because it
|
||||||
target-of-speech annotations \cite{donley2021easycom}. Target annotations are
|
provides synchronized egocentric audio--video, per-person tracks, and voice
|
||||||
used only for training-side supervision and evaluation strata, never as input
|
activity \cite{donley2021easycom}. Target-of-speech annotations, when used, are
|
||||||
at test time.
|
analysis strata only and are never model input or required output.
|
||||||
|
|
||||||
Our contributions are threefold. (1) We formulate continuous speech
|
The intended contributions are therefore a controlled cross-backbone
|
||||||
initiation as recipient-aware floor-entry projection for an invisible camera
|
representation diagnosis, a lightweight local-evidence adaptation selected
|
||||||
wearer. (2) We propose a causal set-based fusion module that grounds mixed
|
from pre-registered candidates, and a behavior-aligned event evaluation. The
|
||||||
audio in a variable number of visible participants and preserves the original
|
current Qwen2.5-Omni observations are motivation, not a claim that all omni
|
||||||
dense EgoSpeak target. (3) We introduce an event evaluation that reports
|
models intrinsically fail. Likewise, track-consistent cross-attention becomes
|
||||||
third-party false entries at matched addressed-response recall, accompanied by
|
the final method only if it beats pooled, no-ID, and capacity-matched controls.
|
||||||
latency and visual counterfactual controls. This design makes the scientific
|
|
||||||
claim falsifiable: visual fusion is useful only if \emph{correctly aligned}
|
|
||||||
vision improves the operating trade-off without suppressing legitimate wearer
|
|
||||||
responses.
|
|
||||||
|
|
||||||
\begin{figure}[t]
|
\begin{figure}[t]
|
||||||
\centering
|
\centering
|
||||||
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
|
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering
|
||||||
\todo{Draw the final method figure: causal omni state $\vect g_t$; recent
|
\todo{After model selection, draw the final system figure. For now it must show
|
||||||
mixed-audio tokens; a masked set of face-track histories; shared per-person
|
the shared experiment interface: raw causal AV, backbone-specific global
|
||||||
audio--visual grounding; three role queries; zero-initialized residual; and
|
readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous
|
||||||
the $10\times3$ activity head. Mark all causal cutoffs and masks.}}}
|
person histories, and the common $10\times3$ head. Mark causal cutoffs,
|
||||||
\caption{Proposed wearer-relative set fusion. The number and ordering of
|
trainable modules, parameter budgets, and source-boundary resets.}}}
|
||||||
visible participants may change, while the output remains fixed-size.}
|
\caption{Planned model-agnostic floor-action study. The local adapter remains a
|
||||||
|
candidate family until the controlled train-side comparison is complete.}
|
||||||
\label{fig:method}
|
\label{fig:method}
|
||||||
\end{figure}
|
\end{figure}
|
||||||
|
|
||||||
\section{Task and Method}
|
\section{Methods}
|
||||||
\label{sec:method}
|
\label{sec:method}
|
||||||
|
|
||||||
\subsection{Wearer-relative activity projection}
|
\subsection{Wearer-relative activity projection}
|
||||||
@@ -141,74 +143,109 @@ windows; elsewhere it remains a future-activity probability. This retains the
|
|||||||
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
|
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision
|
||||||
signal.
|
signal.
|
||||||
|
|
||||||
\subsection{Frozen global stream and local participant set}
|
\subsection{Omni backbones and representation protocols}
|
||||||
|
|
||||||
We obtain a causal global state $\vect{g}_t$ from the streaming Thinker of
|
Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a
|
||||||
Qwen2.5-Omni \cite{xu2025qwenomni}. The backbone is frozen in the main
|
backbone-specific causal global readout at the same physical cutoff $t$.
|
||||||
experiment because preliminary LoRA adaptation caused negative transfer
|
Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family;
|
||||||
(Sec.~\ref{sec:results}). From the most recent $L_a$ seconds, a shared audio
|
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5
|
||||||
encoder produces token sequence $\vect{A}_t$. For each visible track
|
\cite{cui2026minicpmo45} form the planned initial comparison. These models have
|
||||||
$i\in\{1,\ldots,N_t\}$, a shared temporal visual encoder produces
|
different native streaming APIs, temporal tokenization, hidden dimensions, and
|
||||||
$\vect{v}_{t,i}$ from its recent face history. Person-aligned evidence is
|
adaptation interfaces. We therefore emphasize within-backbone deltas rather
|
||||||
|
than treating raw cross-model scores as a pure architecture ranking.
|
||||||
|
|
||||||
|
Every supported backbone is evaluated under three representation protocols:
|
||||||
|
(i) a frozen global readout with a dimension-matched prediction head;
|
||||||
|
(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the
|
||||||
|
same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak
|
||||||
|
targets, context duration, training examples, head capacity, and evaluation
|
||||||
|
are held fixed whenever the backbone permits it. A failure of one LoRA recipe
|
||||||
|
is reported as that recipe's result, not as proof that the backbone cannot
|
||||||
|
adapt.
|
||||||
|
|
||||||
|
\todo{Implement and audit one adapter for each planned backbone. For every
|
||||||
|
model, document model revision, processor, native audio/video rates, exact
|
||||||
|
hidden-state readout, causal/streaming semantics, context limit, dtype,
|
||||||
|
trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
|
||||||
|
Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
|
||||||
|
expose a comparable causal state, report it as unsupported rather than inventing
|
||||||
|
a non-equivalent readout. Add future backbones only before the experiment
|
||||||
|
protocol is locked.}
|
||||||
|
|
||||||
|
\subsection{Candidate local-evidence adaptation}
|
||||||
|
|
||||||
|
The current hypothesis is that global omni states preserve broad semantic
|
||||||
|
context but may make recent prosody, activity boundaries, and individual
|
||||||
|
visual motion difficult for a shallow floor head to recover. From the most
|
||||||
|
recent $L_a$ seconds, a local encoder produces audio evidence $\vect A_t$ and,
|
||||||
|
optionally, an anonymous visible-person set $\mathcal P_t$. A generic candidate
|
||||||
|
adapter is
|
||||||
\begin{equation}
|
\begin{equation}
|
||||||
\vect{p}_{t,i}=\vect{v}_{t,i}+
|
\vect z^{(m)}_t=\vect g^{(m)}_t+
|
||||||
\mathrm{CA}(\vect{v}_{t,i},\vect{A}_t,\vect{A}_t),
|
\gamma_t H_\phi(\vect g^{(m)}_t,\vect A_t,\mathcal P_t),
|
||||||
\label{eq:person}
|
\qquad \gamma_t\vert_{\mathrm{init}}=0,
|
||||||
|
\label{eq:adapter}
|
||||||
\end{equation}
|
\end{equation}
|
||||||
so every participant queries the same mixed audio rather than requiring
|
followed by the common prediction head. Local evidence is derived from the
|
||||||
unavailable separated microphones. A masked set encoder models interactions
|
same causal raw media, not close microphones, transcripts, speaker labels, or
|
||||||
among $\mathcal{P}_t=\{\vect{p}_{t,i}\}_{i=1}^{N_t}$ and is permutation
|
future frames.
|
||||||
equivariant \cite{lee2019settransformer}. Missing detections use a learned
|
|
||||||
\textsc{offscreen/null} element. In minibatches, sets are padded to
|
|
||||||
$N_{\max}$ with an attention mask; no slot-ID embedding is used, and track
|
|
||||||
slots are randomly permuted during training.
|
|
||||||
|
|
||||||
\todo{Implement and verify the shared track encoder, mixed-audio
|
We evaluate candidates in increasing structural complexity: pooled recent
|
||||||
cross-attention, OFFSCREEN token, padding mask, and numerical invariance under
|
audio/vision; a capacity-matched temporal branch; a per-frame person set with
|
||||||
track-slot permutation. Freeze $L_a$, visual crop history, $N_{\max}$, and
|
no cross-time identity; a shared track-wise temporal encoder; and person--audio
|
||||||
face-track confidence filtering from train-side experiments only.}
|
cross-attention. Only a candidate that improves the pre-registered event metric
|
||||||
|
over simpler and capacity-matched controls becomes the final method. If
|
||||||
|
person continuity is selected, temporary IDs route hidden state only---there
|
||||||
|
is no ID embedding, name recognition, or cross-recording identity---and the set
|
||||||
|
operator is permutation equivariant \cite{lee2019settransformer}.
|
||||||
|
|
||||||
\subsection{Role-query fusion}
|
\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
|
||||||
|
parameters and effective local history. Compare a shared local encoder across
|
||||||
Mean pooling can erase which participant is speaking to whom. Instead, three
|
backbones (cross-model control) with each backbone's native local encoder
|
||||||
learned semantic queries represent floor state, address-to-wearer evidence,
|
(within-model control); do not mix these protocols in one claimed delta.
|
||||||
and third-party interaction. Each query is conditioned on $\vect{g}_t$ and
|
For track candidates, specify visibility/age masks, gap/source resets,
|
||||||
cross-attends to the participant set:
|
OFFSCREEN handling, and global-rename invariance. Describe only the selected
|
||||||
\begin{align}
|
architecture as the final method; move rejected candidates to ablations.}
|
||||||
\vect{Q}_t &= [\vect{q}_{\mathrm{floor}};
|
|
||||||
\vect{q}_{\mathrm{addr}};
|
|
||||||
\vect{q}_{\mathrm{third}}]+W_g\vect{g}_t,\\
|
|
||||||
\vect{R}_t &= \mathrm{MHA}(\vect{Q}_t,\widetilde{\mathcal P}_t,
|
|
||||||
\widetilde{\mathcal P}_t),\\
|
|
||||||
\vect{z}_t &= \vect{g}_t+\tanh(\alpha)W_r\mathrm{vec}(\vect{R}_t),
|
|
||||||
\label{eq:fusion}
|
|
||||||
\end{align}
|
|
||||||
where $\alpha$ is initialized to zero. The zero-initialized residual makes the
|
|
||||||
starting model identical to the frozen global baseline and allows the visual
|
|
||||||
branch to be removed exactly. A two-layer prediction head maps $\vect{z}_t$ to
|
|
||||||
Eq.~\eqref{eq:task}.
|
|
||||||
|
|
||||||
\subsection{Training objectives}
|
\subsection{Training objectives}
|
||||||
|
|
||||||
The main loss is class-balanced binary cross entropy over all horizons and
|
The common main loss is class-balanced binary cross entropy over all horizons
|
||||||
activity channels. When annotations are available, auxiliary heads predict
|
and activity channels. This dense EgoSpeak target is the only supervision
|
||||||
per-track active-speaker state and whether the current source addresses the
|
required by the task. If a person-aware candidate survives the activity-only
|
||||||
wearer:
|
comparison, an optional auxiliary head predicts current active-speaker state:
|
||||||
\begin{equation}
|
\begin{equation}
|
||||||
\mathcal{L}=\mathcal{L}_{\mathrm{act}}+
|
\mathcal{L}=\mathcal{L}_{\mathrm{act}}+
|
||||||
\lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}+
|
\lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}.
|
||||||
\lambda_{\mathrm{addr}}\mathcal{L}_{\mathrm{addr}}.
|
|
||||||
\label{eq:loss}
|
\label{eq:loss}
|
||||||
\end{equation}
|
\end{equation}
|
||||||
Ambiguous, group-addressed, and offscreen-target samples are masked from
|
Participant ID, target-of-speech, transcript, close-microphone audio, and
|
||||||
$\mathcal{L}_{\mathrm{addr}}$, but remain in activity training. No language
|
future frames are never model inputs. Target-of-speech may stratify qualitative
|
||||||
model or pseudo-label is used to define primary test events.
|
analysis but is not a predicted variable or a requirement for the primary
|
||||||
|
multi-other event definition.
|
||||||
|
|
||||||
\todo{Run the staged ablation: activity only; +ASD; +addressee. Select
|
\todo{First compare all representation/adaptation protocols with activity loss
|
||||||
$\lambda$ values and early stopping on a training-side calibration fold. Do
|
only. Then ablate +ASD for the selected person-aware candidate. Add an
|
||||||
not inspect Sessions 1--3 during selection.}
|
event-specific wearer onset/yield loss only as a separately reported extension.
|
||||||
|
Select all coefficients and early stopping on a training-side calibration
|
||||||
|
fold; do not inspect Sessions 1--3 during selection.}
|
||||||
|
|
||||||
\section{Experimental Setup}
|
\subsection{Track source and deployment contract}
|
||||||
|
|
||||||
|
Person-aware candidates first use dataset-provided boxes and anonymous IDs to
|
||||||
|
isolate the value of local visual structure. A deployable version instead
|
||||||
|
obtains causal tracklets from a face/head detector followed by an online
|
||||||
|
multi-object tracker; no future-frame smoothing or cross-session recognition
|
||||||
|
is permitted. EasyCom IDs are continued only across genuinely adjacent source
|
||||||
|
segments and reset across redacted gaps. Ego4D IDs are local to a clip.
|
||||||
|
|
||||||
|
\todo{If a person-aware candidate is selected, state precisely whether face or
|
||||||
|
head boxes are used. Report provided-track
|
||||||
|
coverage, mean/maximum people, occlusion gaps, track fragments, and source-gap
|
||||||
|
resets for both datasets. Add either one online detector/tracker experiment or
|
||||||
|
controlled box jitter, dropout, fragmentation, and ID-switch curves; otherwise
|
||||||
|
limit every deployment claim to ``given causal participant tracklets.''}
|
||||||
|
|
||||||
|
\section{Experiments}
|
||||||
\label{sec:setup}
|
\label{sec:setup}
|
||||||
|
|
||||||
\textbf{Data and splits.}
|
\textbf{Data and splits.}
|
||||||
@@ -216,45 +253,83 @@ We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and
|
|||||||
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
|
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold,
|
||||||
and hyperparameter choices use training-side folds only. Inputs contain the
|
and hyperparameter choices use training-side folds only. Inputs contain the
|
||||||
camera stream and shared array audio available to the wearer; close-talking
|
camera stream and shared array audio available to the wearer; close-talking
|
||||||
microphones and target-of-speech labels are supervision only. We also report aggregate
|
microphones, per-person voice labels, and target-of-speech labels are not model
|
||||||
activity forecasting on the EgoSpeak Ego4D split \cite{grauman2022ego4d}, but
|
inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D
|
||||||
do not claim addressee grounding there because comparable target annotations
|
split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies
|
||||||
are absent.
|
clip-local anonymous participant tracks; its per-person voice coverage on the
|
||||||
|
433-clip EgoSpeak split must be audited before reporting the same multi-other
|
||||||
|
event analysis.
|
||||||
|
|
||||||
|
\textbf{Backbone comparison.}
|
||||||
|
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and
|
||||||
|
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state
|
||||||
|
$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence
|
||||||
|
augmentation $G_m+L$. The primary diagnostic is the within-backbone change;
|
||||||
|
cross-backbone absolute scores are secondary because native tokenization,
|
||||||
|
pretraining, streaming interfaces, and hidden-state semantics differ. Common
|
||||||
|
local encoders test whether adaptation transfers across global backbones,
|
||||||
|
whereas native local encoders are a separate, model-specific protocol. All
|
||||||
|
rows share physical input cutoffs, targets, data splits, and evaluation.
|
||||||
|
Unsupported causal readouts are disclosed rather than replaced by offline or
|
||||||
|
future-conditioned features.
|
||||||
|
|
||||||
|
\todo{Implement the three backbone adapters and validate state timestamps with
|
||||||
|
an impulse/cutoff test. Pre-register one matched PEFT budget and one common
|
||||||
|
local branch. Report native-vs-common local encoders in separate table panels.
|
||||||
|
Keep future model additions outside the confirmatory comparison once this
|
||||||
|
matrix is locked.}
|
||||||
|
|
||||||
\textbf{Event construction.}
|
\textbf{Event construction.}
|
||||||
An \emph{addressed response} starts from a pure other-to-wearer utterance and
|
We derive \emph{wearer onset} events directly from the wearer activity target.
|
||||||
requires a wearer onset before an unrelated other onset. A \emph{third-party
|
A \emph{multi-other exchange} is a wearer-silent opportunity interval in which
|
||||||
exchange} starts from a pure other-to-other target while the wearer remains
|
at least two distinct non-wearer participant IDs contribute temporally ordered
|
||||||
silent. Group, none, multiple, and unknown targets form a separate diagnostic
|
speech activity. It is an observable interaction proxy, not a claim about the
|
||||||
stratum. Threshold $\tau_r$ is selected on the development split to obtain
|
speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the
|
||||||
addressed-response recall $r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary
|
development split to obtain wearer-onset recall
|
||||||
metric is false wearer entries per minute of third-party opportunity at
|
$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer
|
||||||
matched recall; the headline operating point is $r=0.8$. We also report
|
entries per minute of multi-other opportunity at matched onset recall; the
|
||||||
addressed-response recall, signed detection latency, wearer yield recall, and
|
headline operating point is $r=0.8$. We also report signed onset latency,
|
||||||
the original ten-horizon per-frame mAP.
|
wearer-yield recall/latency, and the original ten-horizon per-frame mAP.
|
||||||
|
|
||||||
\todo{Freeze the event pairing and competing-risk rules, including pre/post
|
\todo{Freeze the event pairing and competing-risk rules, including the
|
||||||
windows, minimum silence, maximum response delay, overlap handling, and one
|
multi-other alternation window, minimum speech/silence, overlap, maximum onset
|
||||||
detection per event. Publish counts for train/test positive and negative
|
delay, duplicate detections, one detection per event, track visibility, source
|
||||||
events, visibility, duration, and excluded target types. Treat all previously
|
boundaries, and group/offscreen cases. Publish train/test onset counts,
|
||||||
viewed Sessions 1--3 scores as exploratory; lock the final protocol and model
|
multi-other events/minutes, number of distinct visible speakers, and exclusion
|
||||||
before the confirmatory three-seed run.}
|
counts. Manually audit at least 100 events. Treat all previously viewed
|
||||||
|
Sessions 1--3 scores as exploratory; lock the protocol and model before the
|
||||||
|
confirmatory three-seed run.}
|
||||||
|
|
||||||
\textbf{Baselines and controls.}
|
\textbf{Baselines and controls.}
|
||||||
We compare: global omni state (G); G plus recent audio (G+A); G+A plus pooled
|
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
|
||||||
visual context (G+A+V-pool); a parameter-matched temporal branch without
|
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
|
||||||
person identity; and the proposed set branch. Required counterfactuals replace
|
This compact matrix tests the representation hypothesis before investing in a
|
||||||
correct tracks with (i) wrong participants from the same scene, (ii)
|
complex fusion module. If local vision survives three seeds and the event
|
||||||
time-shifted tracks, and (iii) zeros. These controls share parameter count,
|
gate, we then compare a parameter-matched temporal branch without person
|
||||||
audio history, and optimization budget.
|
continuity, a separately trained per-frame/no-ID set branch, and a
|
||||||
|
track-consistent branch. On a frozen track-aware checkpoint, a clip-consistent
|
||||||
|
global renaming must leave outputs unchanged, whereas \emph{broken
|
||||||
|
continuity} periodically remaps IDs while preserving every crop, timestamp,
|
||||||
|
audio sample, and target. Time-shifted and removed vision provide additional
|
||||||
|
alignment controls.
|
||||||
|
|
||||||
|
\todo{Define the broken-continuity corruption before evaluation: remap only at
|
||||||
|
pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
|
||||||
|
call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
|
||||||
|
frame counts, audio, labels, and visibility masks. Train the no-ID/reset
|
||||||
|
baseline from scratch with matched parameters; evaluate all test-time
|
||||||
|
counterfactuals using one frozen checkpoint and one frozen threshold.}
|
||||||
|
|
||||||
\textbf{Implementation and statistics.}
|
\textbf{Implementation and statistics.}
|
||||||
All models run at 5~Hz with an 8-s causal memory. The 3B Thinker is frozen and
|
All prediction heads run at 5~Hz and use an 8-s causal observation window;
|
||||||
local audio/face histories are encoded causally. We report three seeds and
|
model-specific deviations required by a native streaming interface are stated
|
||||||
mean$\pm$standard deviation. Confidence intervals for event metrics use paired
|
explicitly. In the frozen protocol the omni backbone does not update. In PEFT
|
||||||
cluster bootstrap over sessions/conversation blocks rather than treating
|
and local protocols, trainable parameters and optimizer exposure are reported
|
||||||
adjacent frames as independent. Runtime includes P50/P95 decision latency,
|
separately. Every temporal state resets at non-contiguous source boundaries.
|
||||||
peak memory, and trainable parameter count.
|
We report three seeds and mean$\pm$standard deviation. Confidence intervals for
|
||||||
|
event metrics use paired cluster bootstrap over sessions/conversation blocks
|
||||||
|
rather than treating adjacent frames as independent. Runtime includes P50/P95
|
||||||
|
decision latency, throughput, peak memory, and trainable parameter count.
|
||||||
|
|
||||||
\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
|
\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
|
||||||
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
|
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
|
||||||
@@ -264,95 +339,138 @@ count, hardware, and real-time latency after the architecture is frozen.}
|
|||||||
\label{sec:results}
|
\label{sec:results}
|
||||||
|
|
||||||
\begin{table}[t]
|
\begin{table}[t]
|
||||||
\caption{Current aggregate evidence on EasyCom test Sessions 1--3. Values are
|
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
|
||||||
percentages from one seed and are motivation, not the final main result.}
|
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
|
||||||
|
confirmatory result.}
|
||||||
\label{tab:preliminary}
|
\label{tab:preliminary}
|
||||||
\centering
|
\centering
|
||||||
\small
|
\small
|
||||||
|
\resizebox{\columnwidth}{!}{%
|
||||||
\begin{tabular}{lccc}
|
\begin{tabular}{lccc}
|
||||||
\hline
|
\hline
|
||||||
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
|
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
|
||||||
\hline
|
\hline
|
||||||
G+A & 81.72 & 85.07 & 65.11\\
|
Qwen-3B $G$+local A & 81.72 & 85.07 & 65.11\\
|
||||||
G+A+V-pool & 82.03 & 85.09 & 65.40\\
|
Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
|
||||||
\hline
|
\hline
|
||||||
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
|
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
|
||||||
\hline
|
\hline
|
||||||
\end{tabular}
|
\end{tabular}}
|
||||||
\end{table}
|
\end{table}
|
||||||
|
|
||||||
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
|
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
|
||||||
Pooled vision gives a small gain, while immediate wearer AP is nearly
|
Pooled vision gives a small gain, while immediate wearer AP is nearly
|
||||||
unchanged. This is compatible with either weak but useful social grounding or
|
unchanged. This is compatible with either weak but useful social grounding or
|
||||||
generic visual regularization; only matched-recall event evaluation and
|
generic visual regularization; only matched-recall event evaluation and
|
||||||
counterfactual identity controls can distinguish them.
|
track-continuity controls can distinguish them.
|
||||||
|
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
|
||||||
|
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
|
||||||
|
feature protocol, it is not combined numerically with
|
||||||
|
Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
|
||||||
|
study remains necessary.
|
||||||
|
|
||||||
\begin{table}[t]
|
\begin{table}[t]
|
||||||
\caption{Primary EasyCom event evaluation. Lower $\mathrm{FP}_{\mathrm{3p}}$
|
\caption{Planned cross-backbone EasyCom event evaluation. Lower
|
||||||
is better; all rows use the same development-selected operating protocol.}
|
$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
|
||||||
|
development-selected operating protocol. ``Local'' denotes the train-side
|
||||||
|
selected common local-evidence candidate.}
|
||||||
\label{tab:main}
|
\label{tab:main}
|
||||||
\centering
|
\centering
|
||||||
\small
|
\small
|
||||||
\resizebox{\columnwidth}{!}{%
|
\resizebox{\columnwidth}{!}{%
|
||||||
\begin{tabular}{lccccc}
|
\begin{tabular}{lccccc}
|
||||||
\hline
|
\hline
|
||||||
Model & $\mathrm{FP}_{\mathrm{3p}}$/min@$R=.8$ & $R_{\mathrm{addr}}$ &
|
Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R &
|
||||||
Latency & Yield R & Mean mAP\\
|
Onset latency & Yield R & Mean mAP\\
|
||||||
\hline
|
\hline
|
||||||
G & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
Qwen-3B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
G+A & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
Qwen-3B PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
G+A+V-pool & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
Qwen-3B + local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
Temporal control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
Qwen-7B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
\method{} & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
Qwen-7B PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
\method{}+ASD & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
MiniCPM-o frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
\method{}+Addr & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
\hline
|
\hline
|
||||||
\end{tabular}}
|
\end{tabular}}
|
||||||
\end{table}
|
\end{table}
|
||||||
|
|
||||||
\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
|
\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three
|
||||||
seeds, plus 95\% paired cluster-bootstrap confidence intervals for the
|
seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs
|
||||||
proposed-vs-G+A difference. The paper's main claim is supported only if
|
PEFT and frozen vs local within each backbone. A general omni-representation
|
||||||
third-party FP/min decreases at matched recall without worse response latency
|
claim requires a consistent pattern across at least two backbone families; an
|
||||||
or wearer-yield recall.}
|
effect confined to Qwen must be stated as Qwen-specific. The behavior claim is
|
||||||
|
supported only if multi-other FP/min decreases at matched wearer-onset recall
|
||||||
|
without worse onset latency or wearer-yield recall. If the selected local
|
||||||
|
candidate uses tracks, also report its paired intervals against no-ID and
|
||||||
|
temporal-capacity controls.}
|
||||||
|
|
||||||
\begin{table}[t]
|
\begin{table}[t]
|
||||||
\caption{Visual grounding counterfactual on the frozen best model.}
|
\caption{Conditional track-continuity counterfactual on the best track-aware
|
||||||
|
model. This table is retained only if train-side selection chooses such a
|
||||||
|
candidate; all rows reuse one frozen checkpoint.}
|
||||||
\label{tab:counterfactual}
|
\label{tab:counterfactual}
|
||||||
\centering
|
\centering
|
||||||
\small
|
\small
|
||||||
|
\resizebox{\columnwidth}{!}{%
|
||||||
\begin{tabular}{lccc}
|
\begin{tabular}{lccc}
|
||||||
\hline
|
\hline
|
||||||
Visual evidence & $\mathrm{FP}_{\mathrm{3p}}$/min & Addr. recall & 0.2-s mAP\\
|
Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\
|
||||||
\hline
|
\hline
|
||||||
Correct tracks & \todo{--} & \todo{--} & \todo{--}\\
|
Correct tracks & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
Wrong person & \todo{--} & \todo{--} & \todo{--}\\
|
Global ID rename & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
|
Broken continuity & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
Time shifted & \todo{--} & \todo{--} & \todo{--}\\
|
Time shifted & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
Removed & \todo{--} & \todo{--} & \todo{--}\\
|
Removed & \todo{--} & \todo{--} & \todo{--}\\
|
||||||
\hline
|
\hline
|
||||||
\end{tabular}
|
\end{tabular}}
|
||||||
\end{table}
|
\end{table}
|
||||||
|
|
||||||
Table~\ref{tab:counterfactual} is the identity test. A true social-grounding
|
If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the
|
||||||
effect should be largest for correct tracks and should collapse for wrong or
|
continuity test. Global renaming must be
|
||||||
time-shifted people. If every visual variant improves equally, the defensible
|
numerically invariant because IDs carry no semantics. If the model exploits
|
||||||
conclusion is additional capacity/history, not addressee grounding.
|
person history, correct tracks should outperform broken continuity even though
|
||||||
|
both contain exactly the same per-frame people. If broken, time-shifted, and
|
||||||
|
removed variants perform similarly to correct tracks, the defensible
|
||||||
|
conclusion is additional capacity or generic visual context, not useful
|
||||||
|
person-continuous reasoning.
|
||||||
|
|
||||||
\todo{Add one compact error analysis split by target visibility, number of
|
\todo{Add one compact error analysis split by model family, track visibility,
|
||||||
visible people, overlap, and backchannel/interruption proxy. Include event
|
number of visible people, overlap, track age/fragmentation, and
|
||||||
counts with every rate. Move negative LoRA/7B/ROMA diagnostics to supplementary
|
backchannel/interruption proxy. Include event counts with every rate. Move
|
||||||
material unless they directly explain the final method.}
|
rejected PEFT recipes and fusion candidates to supplementary material unless
|
||||||
|
they directly explain the selected method.}
|
||||||
|
|
||||||
|
\textbf{Qualitative evidence.}
|
||||||
|
For a multi-other exchange, visualize per-person speech activity and track IDs,
|
||||||
|
wearer ground truth, the frozen-global and selected-local wearer-entry scores,
|
||||||
|
and the single frozen threshold on one physical time axis. If the selected
|
||||||
|
candidate is track-aware, also show its broken-continuity score. A useful
|
||||||
|
example shows the frozen baseline rising at an other-speaker boundary while
|
||||||
|
the selected adapter remains below threshold; a companion positive example
|
||||||
|
must show that a real wearer onset is not suppressed.
|
||||||
|
|
||||||
|
\todo{Do not hand-pick only successful clips. Before viewing the final test
|
||||||
|
plots, freeze a deterministic selection rule and show: largest paired
|
||||||
|
improvement, near-median improvement, one failure, and one correctly retained
|
||||||
|
wearer onset. Include full local timelines and source frame times, not cropped
|
||||||
|
screenshots without context. State that examples illustrate the aggregate
|
||||||
|
event result and cannot establish it alone.}
|
||||||
|
|
||||||
\section{Conclusion}
|
\section{Conclusion}
|
||||||
\label{sec:conclusion}
|
\label{sec:conclusion}
|
||||||
|
|
||||||
We study when an egocentric agent should enter a multiparty floor, not merely
|
We study when an egocentric agent should enter or yield a multiparty floor, not
|
||||||
whether someone will speak. The proposed formulation retains dense future
|
merely whether someone will speak. The central question is model-agnostic:
|
||||||
activity prediction while grounding wearer entry in a variable set of visible
|
whether frozen or parameter-efficiently adapted omni representations expose
|
||||||
participants. Its event protocol tests the intended behavior directly:
|
the fine-grained floor cues required by a 5-Hz controller, and whether causal
|
||||||
reducing intrusions into third-party exchanges at matched legitimate-response
|
local evidence closes any gap. Dense activity forecasting supplies a common
|
||||||
recall. \todo{Replace with one quantitative conclusion and one limitation;
|
target, while matched-recall event evaluation tests whether an apparent gain
|
||||||
do not claim success from aggregate mAP alone.}
|
actually prevents intrusions during multi-other exchanges. Current Qwen-3B
|
||||||
|
results are exploratory motivation only. \todo{After the controlled study,
|
||||||
|
state which backbone/adapter was selected, whether the representation pattern
|
||||||
|
generalized beyond Qwen, the quantitative false-entry and onset/yield results,
|
||||||
|
and the main deployment limitation. Do not infer success from aggregate mAP or
|
||||||
|
selected examples alone.}
|
||||||
|
|
||||||
\bibliographystyle{IEEEbib}
|
\bibliographystyle{IEEEbib}
|
||||||
\bibliography{references}
|
\bibliography{references}
|
||||||
|
|||||||
@@ -48,6 +48,13 @@
|
|||||||
year = {2025}
|
year = {2025}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@article{cui2026minicpmo45,
|
||||||
|
author = {Junbo Cui and Bokai Xu and Chongyi Wang and Tianyu Yu and Weiyue Sun and Yingjing Xu and Tianran Wang and Zhihui He and Wenshuo Ma and Tianchi Cai and others},
|
||||||
|
title = {{MiniCPM-o 4.5}: Towards Real-Time Full-Duplex Omni-Modal Interaction},
|
||||||
|
journal = {arXiv preprint arXiv:2604.27393},
|
||||||
|
year = {2026}
|
||||||
|
}
|
||||||
|
|
||||||
@inproceedings{lee2019settransformer,
|
@inproceedings{lee2019settransformer,
|
||||||
author = {Juho Lee and Yoonho Lee and Jungtaek Kim and Adam Kosiorek and Seungjin Choi and Yee Whye Teh},
|
author = {Juho Lee and Yoonho Lee and Jungtaek Kim and Adam Kosiorek and Seungjin Choi and Yee Whye Teh},
|
||||||
title = {Set Transformer: A Framework for Attention-Based Permutation-Invariant Neural Networks},
|
title = {Set Transformer: A Framework for Attention-Based Permutation-Invariant Neural Networks},
|
||||||
|
|||||||
Reference in New Issue
Block a user