wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Expand targeted introduction and move method figure to page two

Browse Source
This commit is contained in:
Xin Wang
2026-09-24 16:28:20 +08:00
parent 8ccfc364d0
commit 98b3ad2d69
5 changed files with 105 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
manuscript/targeted/Makefile
+2 -3
View File
@@ -2,11 +2,11 @@ LATEXMK ?= latexmk
.PHONY: all clean figure
all: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
all: figures/system_overview_onecolumn.pdf
mkdir -p build
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
figure: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
figure: figures/system_overview_onecolumn.pdf
figures/%.pdf: figures/%.tex
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
@@ -14,4 +14,3 @@ figures/%.pdf: figures/%.tex
clean:
$(LATEXMK) -C -outdir=build main.tex
$(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex
$(LATEXMK) -c -outdir=figures figures/task_overview.tex
manuscript/targeted/README.md
+6 -4
View File
@@ -63,11 +63,13 @@ make
```
The editable sources are `main.tex`, `tables_compact.tex`, `references.bib`,
`figures/task_overview.tex`, and `figures/system_overview_onecolumn.tex`.
and `figures/system_overview_onecolumn.tex`. The earlier
`figures/task_overview.tex` is retained as unused source.
Generated LaTeX files are written to `build/`.
The preview targets four technical pages plus a reference-only fifth page. Both
the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column.
The preview targets four technical pages plus a reference-only fifth page.
The single-column method diagram (Fig. 1) has SWE on top and PTR below;
the earlier standalone task figure is not included.
The current PDF reserves seven numbered result tables. Table 1 spans two
columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer
rows. Only three EasyCom models have archived complete horizon curves; other
@@ -77,7 +79,7 @@ Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables
Pending cells are explicitly unmeasured. Active table layouts are in
`tables_compact.tex`; the earlier full-precision tabulars
remain disabled in `main.tex` for source-level auditing.
Figure 3 is an H/Q case slot, not a selected or measured example. Its
Figure 2 is an H/Q case slot, not a selected or measured example. Its
cutoff-marked timelines must be inserted only after the slice protocol and
case-selection rule are frozen. The single-column system figure is generated
from figures/system_overview_onecolumn.tex; the earlier full-width source
manuscript/targeted/figures/system_overview_onecolumn.pdf
BIN
View File
Binary file not shown.
manuscript/targeted/figures/system_overview_onecolumn.tex
+58 -37
View File
@@ -1,5 +1,5 @@
\documentclass[tikz,border=2pt]{standalone}
\usetikzlibrary{arrows.meta,positioning}
\usetikzlibrary{arrows.meta}
\definecolor{navy}{HTML}{19324A}
\definecolor{bluefill}{HTML}{EAF1F7}
\definecolor{teal}{HTML}{087F78}
@@ -8,43 +8,64 @@
\definecolor{orangefill}{HTML}{FFF0DF}
\begin{document}
\begin{tikzpicture}[
font=\sffamily\small,
font=\sffamily\footnotesize,
>=Latex,
box/.style={draw=navy,fill=bluefill,rounded corners=1.5pt,
align=center,minimum height=7mm,inner sep=2pt},
local/.style={box,draw=teal,fill=tealfill},
flow/.style={-{Latex[length=1.6mm]},draw=navy,line width=.6pt}]
\node[box,text width=20mm] (global) at (1.12,3.9)
{Observed AV\\8-s maximum};
\node[box,text width=22mm] (thinker) at (4.13,3.9)
{Frozen Thinker\\readout $G$};
\node[local,text width=20mm] (audio) at (1.12,2.9)
{Recent 1-s\\mixed audio};
\node[local,text width=22mm] (aenc) at (4.13,2.9)
{Frozen encoder\\local $A$};
\node[local,dashed,text width=20mm] (video) at (1.12,1.9)
{Past person\\crops (optional)};
\node[local,dashed,text width=22mm] (venc) at (4.13,1.9)
{Frozen encoder\\pooled $V$};
\node[box,text width=18mm] (head) at (7.05,2.9)
{Validities +\\trained head};
\draw[flow] (global) -- (thinker);
\draw[flow,draw=teal] (audio) -- (aenc);
\draw[flow,draw=teal,dashed] (video) -- (venc);
\draw[flow] (thinker.east) -- ++(.18,0) |- (head.north);
\draw[flow,draw=teal] (aenc) -- (head);
\draw[flow,draw=teal,dashed] (venc.east) -- ++(.18,0) |- (head.south);
\node[box,text width=32mm] (out) at (5.7,.88)
{$10\times3$ future activity\\0.2--2.0-s horizons};
\draw[flow] (head.south) -- (out.north);
\draw[draw=navy,line width=.5pt] (0,.12) -- (8.25,.12);
\node[anchor=west,text=navy] at (0,-.28) {Completed blocks};
\foreach \x in {0,1.04,2.08} {
\draw[draw=navy,fill=bluefill] (\x,-.93) rectangle ++(.96,.35);
flow/.style={-{Latex[length=1.7mm]},draw=navy,line width=.7pt}]
\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy]
at (.05,5.45) {(a) Short-window encoder evidence (SWE)};
\node[anchor=east,font=\sffamily\scriptsize,text=navy]
at (8.25,5.45) {inputs $\leq t_n$};
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
(.10,4.48) rectangle (3.80,5.18);
\node[align=center,text=navy] at (1.95,4.83)
{Frozen Omni Thinker\\global AV context $G$};
\draw[draw=teal,fill=tealfill,rounded corners=2pt]
(.10,3.57) rectangle (3.80,4.27);
\node[align=center,text=teal] at (1.95,3.92)
{Frozen audio encoder\\recent 1-s audio $A$};
\draw[draw=teal,fill=tealfill,dashed,rounded corners=2pt]
(.10,2.48) rectangle (3.80,3.43);
\node[align=center,text=teal] at (1.95,2.955)
{Frozen visual encoder\\recent person-crop clues $V$};
\draw[draw=navy,line width=.7pt] (3.80,4.83) -- (4.38,4.83)
-- (4.38,2.955) -- (3.80,2.955);
\draw[draw=teal,line width=.7pt] (3.80,3.92) -- (4.38,3.92);
\draw[flow] (4.38,3.92) -- (4.85,3.92);
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
(4.95,3.57) rectangle (6.40,4.27);
\node[align=center,text=navy] at (5.675,3.92) {Fusion\\head};
\draw[flow] (6.40,3.92) -- (6.78,3.92);
\draw[draw=orange,fill=orangefill,rounded corners=2pt]
(6.88,3.57) rectangle (8.30,4.27);
\node[align=center,text=orange] at (7.59,3.92)
{Future\\activity};
\draw[draw=navy!35,line width=.5pt] (.05,2.34) -- (8.30,2.34);
\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy]
at (.05,2.07) {(b) Partial-tail refresh (PTR)};
\node[font=\sffamily\scriptsize,text=navy] at (1.58,1.70)
{reused completed blocks};
\node[font=\sffamily\scriptsize,text=orange] at (4.08,1.70)
{observed at $t_n$};
\foreach \x in {.15,1.15,2.15} {
\draw[draw=navy,fill=bluefill,rounded corners=1pt]
(\x,1.08) rectangle ++(.88,.46);
\node[text=navy] at (\x+.44,1.31) {2 s};
}
\draw[draw=orange,fill=orangefill] (3.12,-.93) rectangle ++(.96,.35);
\node[anchor=west,text=orange] at (4.25,-.76) {Observed tail};
\node[anchor=west,text=navy] at (0,-1.33)
{Reuse completed payloads; query a temporary tail at $t_n$.};
\draw[draw=orange,fill=orangefill,rounded corners=1pt]
(3.20,1.08) rectangle (4.95,1.54);
\node[text=orange] at (4.075,1.31) {partial tail};
\draw[flow] (4.95,1.31) -- (5.41,1.31);
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
(5.51,.96) rectangle (8.30,1.66);
\node[align=center,text=navy] at (6.905,1.31)
{Temporary Thinker\\readout $G_t$};
\node[anchor=west,font=\sffamily\scriptsize,text=navy]
at (.15,.53) {Only completed blocks enter persistent history; the tail is not committed.};
\end{tikzpicture}
\end{document}
manuscript/targeted/main.tex
+39 -21
View File
@@ -57,7 +57,14 @@ streaming, voice activity projection, conversational agents
An egocentric conversational agent must decide \emph{when} to speak even
though its camera wearer is usually invisible. It hears a mixed conversation
and sees other participants, but must anticipate its own speech from the
observed stream before a turn boundary is known (Fig.~\ref{fig:task}).
observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
This decision differs from recognizing who is speaking now: the relevant
evidence may be a brief pause, overlap, or response by someone in view, while
the target is the wearer's \emph{future} activity. It also differs from
predicting a semantic end of turn or selecting an addressee. We therefore
study a deliberately narrower and measurable question: at each 200-ms
cutoff, how well can a causal observer forecast background, wearer, and
aggregate-other activity over the next 0.2--2.0~s?
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
extensions, backchannel prediction, online end-of-turn detection, and event
benchmarks address increasingly realistic timing decisions
@@ -65,16 +72,10 @@ benchmarks address increasingly realistic timing decisions
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
Their input viewpoints and target decisions, however, are not interchangeable
with forecasting the future activity of an unseen wearer.
\begin{figure}[H]
\centering
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
only the past audio--video prefix is input; ten future activity states are
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
not semantic floor ownership.}
\label{fig:task}
\end{figure}
The distinction matters when comparing systems: a correct current-speaker
label or an utterance-final response can conceal errors in the first few
hundred milliseconds, precisely where a wearable agent must decide whether
to wait or begin speaking.
\textbf{Egocentric audiovisual conversation.}
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
@@ -90,6 +91,13 @@ prediction in noise \cite{oconnorrussell2025noise}. These studies make a
broad ``first audiovisual turn-taking'' claim untenable. What remains
less examined is how a single causal first-person stream should expose
recent audio--visual evidence to forecast the invisible wearer's activity.
EgoSpeak establishes that compact audio and video representations can be
fused for this task, but it does not answer whether a large integrated Omni
representation makes those cutoff-local cues readily available to a small
forecasting head. Conversely, work on active speakers or addressees need
not solve wearer anticipation: the visible interlocutor and the predicted
speaker occupy different roles. This leaves an architectural question,
rather than a claim that audiovisual evidence itself is new.
\textbf{Streaming Omni and temporal access.}
Omni systems integrate modalities and support streaming interaction
@@ -106,21 +114,31 @@ recent modality features outside the Thinker state, while \emph{partial-tail
refresh} (PTR) queries the observed prefix of the incomplete current block.
SWE denotes temporal locality, not shallow encoder layers; the argument
concerns this pipeline, not an intrinsic deficit of every Omni model.
These interventions target different bottlenecks. SWE asks whether recent
encoder evidence improves prediction when the global state is held fixed;
PTR asks whether the same integrated backbone benefits from a fresher readout
without admitting future media. A gain from one does not establish a gain
from the other. In particular, the effect of PTR requires a matched
completed-only comparison at the same context cap, rather than comparison
with a longer or shorter history.
Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity
anticipation; an EasyCom analysis separating global, recent-audio,
pooled-visual, and privileged visible-activity information; and a specified
EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics.
Only EasyCom screening scores are currently available: matched PTR effects,
Ego4D transfer, and event-level behavior remain open tests.
This framing yields three concrete contributions. First, we formulate a
causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
ways to expose local evidence to a frozen Omni model. Second, on EasyCom we
compare global, recent-audio, and pooled-visual features, then use privileged
past visible activity only as an information audit, not a deployable input.
Third, we specify cross-dataset, endpoint, and hard-slice tests. EasyCom
screening is the only measured evidence; matched PTR, Ego4D transfer, and
event behavior remain open.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
audio and optional pooled person-crop vision to a frozen-Omni head. PTR
temporarily queries the observed current-block prefix while reusing completed
2-s payloads. Privileged past-activity labels are not model inputs.}
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
with recent audio and optional pooled person-crop evidence in the forecasting
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
the observed current-block tail at each 200-ms cutoff. The tail is not
committed to persistent history.}
\label{fig:overview}
\end{figure}