Expand targeted introduction and move method figure to page two

This commit is contained in:
Xin Wang
2026-09-24 16:28:20 +08:00
parent 8ccfc364d0
commit 98b3ad2d69
5 changed files with 105 additions and 65 deletions
+2 -3
View File
@@ -2,11 +2,11 @@ LATEXMK ?= latexmk
.PHONY: all clean figure .PHONY: all clean figure
all: figures/system_overview_onecolumn.pdf figures/task_overview.pdf all: figures/system_overview_onecolumn.pdf
mkdir -p build mkdir -p build
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex $(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
figure: figures/system_overview_onecolumn.pdf figures/task_overview.pdf figure: figures/system_overview_onecolumn.pdf
figures/%.pdf: figures/%.tex figures/%.pdf: figures/%.tex
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $< $(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
@@ -14,4 +14,3 @@ figures/%.pdf: figures/%.tex
clean: clean:
$(LATEXMK) -C -outdir=build main.tex $(LATEXMK) -C -outdir=build main.tex
$(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex $(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex
$(LATEXMK) -c -outdir=figures figures/task_overview.tex
+6 -4
View File
@@ -63,11 +63,13 @@ make
``` ```
The editable sources are `main.tex`, `tables_compact.tex`, `references.bib`, The editable sources are `main.tex`, `tables_compact.tex`, `references.bib`,
`figures/task_overview.tex`, and `figures/system_overview_onecolumn.tex`. and `figures/system_overview_onecolumn.tex`. The earlier
`figures/task_overview.tex` is retained as unused source.
Generated LaTeX files are written to `build/`. Generated LaTeX files are written to `build/`.
The preview targets four technical pages plus a reference-only fifth page. Both The preview targets four technical pages plus a reference-only fifth page.
the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column. The single-column method diagram (Fig. 1) has SWE on top and PTR below;
the earlier standalone task figure is not included.
The current PDF reserves seven numbered result tables. Table 1 spans two The current PDF reserves seven numbered result tables. Table 1 spans two
columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer
rows. Only three EasyCom models have archived complete horizon curves; other rows. Only three EasyCom models have archived complete horizon curves; other
@@ -77,7 +79,7 @@ Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables
Pending cells are explicitly unmeasured. Active table layouts are in Pending cells are explicitly unmeasured. Active table layouts are in
`tables_compact.tex`; the earlier full-precision tabulars `tables_compact.tex`; the earlier full-precision tabulars
remain disabled in `main.tex` for source-level auditing. remain disabled in `main.tex` for source-level auditing.
Figure 3 is an H/Q case slot, not a selected or measured example. Its Figure 2 is an H/Q case slot, not a selected or measured example. Its
cutoff-marked timelines must be inserted only after the slice protocol and cutoff-marked timelines must be inserted only after the slice protocol and
case-selection rule are frozen. The single-column system figure is generated case-selection rule are frozen. The single-column system figure is generated
from figures/system_overview_onecolumn.tex; the earlier full-width source from figures/system_overview_onecolumn.tex; the earlier full-width source
@@ -1,5 +1,5 @@
\documentclass[tikz,border=2pt]{standalone} \documentclass[tikz,border=2pt]{standalone}
\usetikzlibrary{arrows.meta,positioning} \usetikzlibrary{arrows.meta}
\definecolor{navy}{HTML}{19324A} \definecolor{navy}{HTML}{19324A}
\definecolor{bluefill}{HTML}{EAF1F7} \definecolor{bluefill}{HTML}{EAF1F7}
\definecolor{teal}{HTML}{087F78} \definecolor{teal}{HTML}{087F78}
@@ -8,43 +8,64 @@
\definecolor{orangefill}{HTML}{FFF0DF} \definecolor{orangefill}{HTML}{FFF0DF}
\begin{document} \begin{document}
\begin{tikzpicture}[ \begin{tikzpicture}[
font=\sffamily\small, font=\sffamily\footnotesize,
>=Latex, >=Latex,
box/.style={draw=navy,fill=bluefill,rounded corners=1.5pt, flow/.style={-{Latex[length=1.7mm]},draw=navy,line width=.7pt}]
align=center,minimum height=7mm,inner sep=2pt},
local/.style={box,draw=teal,fill=tealfill}, \node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy]
flow/.style={-{Latex[length=1.6mm]},draw=navy,line width=.6pt}] at (.05,5.45) {(a) Short-window encoder evidence (SWE)};
\node[box,text width=20mm] (global) at (1.12,3.9) \node[anchor=east,font=\sffamily\scriptsize,text=navy]
{Observed AV\\8-s maximum}; at (8.25,5.45) {inputs $\leq t_n$};
\node[box,text width=22mm] (thinker) at (4.13,3.9)
{Frozen Thinker\\readout $G$}; \draw[draw=navy,fill=bluefill,rounded corners=2pt]
\node[local,text width=20mm] (audio) at (1.12,2.9) (.10,4.48) rectangle (3.80,5.18);
{Recent 1-s\\mixed audio}; \node[align=center,text=navy] at (1.95,4.83)
\node[local,text width=22mm] (aenc) at (4.13,2.9) {Frozen Omni Thinker\\global AV context $G$};
{Frozen encoder\\local $A$}; \draw[draw=teal,fill=tealfill,rounded corners=2pt]
\node[local,dashed,text width=20mm] (video) at (1.12,1.9) (.10,3.57) rectangle (3.80,4.27);
{Past person\\crops (optional)}; \node[align=center,text=teal] at (1.95,3.92)
\node[local,dashed,text width=22mm] (venc) at (4.13,1.9) {Frozen audio encoder\\recent 1-s audio $A$};
{Frozen encoder\\pooled $V$}; \draw[draw=teal,fill=tealfill,dashed,rounded corners=2pt]
\node[box,text width=18mm] (head) at (7.05,2.9) (.10,2.48) rectangle (3.80,3.43);
{Validities +\\trained head}; \node[align=center,text=teal] at (1.95,2.955)
\draw[flow] (global) -- (thinker); {Frozen visual encoder\\recent person-crop clues $V$};
\draw[flow,draw=teal] (audio) -- (aenc);
\draw[flow,draw=teal,dashed] (video) -- (venc); \draw[draw=navy,line width=.7pt] (3.80,4.83) -- (4.38,4.83)
\draw[flow] (thinker.east) -- ++(.18,0) |- (head.north); -- (4.38,2.955) -- (3.80,2.955);
\draw[flow,draw=teal] (aenc) -- (head); \draw[draw=teal,line width=.7pt] (3.80,3.92) -- (4.38,3.92);
\draw[flow,draw=teal,dashed] (venc.east) -- ++(.18,0) |- (head.south); \draw[flow] (4.38,3.92) -- (4.85,3.92);
\node[box,text width=32mm] (out) at (5.7,.88) \draw[draw=navy,fill=bluefill,rounded corners=2pt]
{$10\times3$ future activity\\0.2--2.0-s horizons}; (4.95,3.57) rectangle (6.40,4.27);
\draw[flow] (head.south) -- (out.north); \node[align=center,text=navy] at (5.675,3.92) {Fusion\\head};
\draw[draw=navy,line width=.5pt] (0,.12) -- (8.25,.12); \draw[flow] (6.40,3.92) -- (6.78,3.92);
\node[anchor=west,text=navy] at (0,-.28) {Completed blocks}; \draw[draw=orange,fill=orangefill,rounded corners=2pt]
\foreach \x in {0,1.04,2.08} { (6.88,3.57) rectangle (8.30,4.27);
\draw[draw=navy,fill=bluefill] (\x,-.93) rectangle ++(.96,.35); \node[align=center,text=orange] at (7.59,3.92)
{Future\\activity};
\draw[draw=navy!35,line width=.5pt] (.05,2.34) -- (8.30,2.34);
\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy]
at (.05,2.07) {(b) Partial-tail refresh (PTR)};
\node[font=\sffamily\scriptsize,text=navy] at (1.58,1.70)
{reused completed blocks};
\node[font=\sffamily\scriptsize,text=orange] at (4.08,1.70)
{observed at $t_n$};
\foreach \x in {.15,1.15,2.15} {
\draw[draw=navy,fill=bluefill,rounded corners=1pt]
(\x,1.08) rectangle ++(.88,.46);
\node[text=navy] at (\x+.44,1.31) {2 s};
} }
\draw[draw=orange,fill=orangefill] (3.12,-.93) rectangle ++(.96,.35); \draw[draw=orange,fill=orangefill,rounded corners=1pt]
\node[anchor=west,text=orange] at (4.25,-.76) {Observed tail}; (3.20,1.08) rectangle (4.95,1.54);
\node[anchor=west,text=navy] at (0,-1.33) \node[text=orange] at (4.075,1.31) {partial tail};
{Reuse completed payloads; query a temporary tail at $t_n$.}; \draw[flow] (4.95,1.31) -- (5.41,1.31);
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
(5.51,.96) rectangle (8.30,1.66);
\node[align=center,text=navy] at (6.905,1.31)
{Temporary Thinker\\readout $G_t$};
\node[anchor=west,font=\sffamily\scriptsize,text=navy]
at (.15,.53) {Only completed blocks enter persistent history; the tail is not committed.};
\end{tikzpicture} \end{tikzpicture}
\end{document} \end{document}
+39 -21
View File
@@ -57,7 +57,14 @@ streaming, voice activity projection, conversational agents
An egocentric conversational agent must decide \emph{when} to speak even An egocentric conversational agent must decide \emph{when} to speak even
though its camera wearer is usually invisible. It hears a mixed conversation though its camera wearer is usually invisible. It hears a mixed conversation
and sees other participants, but must anticipate its own speech from the and sees other participants, but must anticipate its own speech from the
observed stream before a turn boundary is known (Fig.~\ref{fig:task}). observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
This decision differs from recognizing who is speaking now: the relevant
evidence may be a brief pause, overlap, or response by someone in view, while
the target is the wearer's \emph{future} activity. It also differs from
predicting a semantic end of turn or selecting an addressee. We therefore
study a deliberately narrower and measurable question: at each 200-ms
cutoff, how well can a causal observer forecast background, wearer, and
aggregate-other activity over the next 0.2--2.0~s?
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
extensions, backchannel prediction, online end-of-turn detection, and event extensions, backchannel prediction, online end-of-turn detection, and event
benchmarks address increasingly realistic timing decisions benchmarks address increasingly realistic timing decisions
@@ -65,16 +72,10 @@ benchmarks address increasingly realistic timing decisions
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}. lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
Their input viewpoints and target decisions, however, are not interchangeable Their input viewpoints and target decisions, however, are not interchangeable
with forecasting the future activity of an unseen wearer. with forecasting the future activity of an unseen wearer.
The distinction matters when comparing systems: a correct current-speaker
\begin{figure}[H] label or an utterance-final response can conceal errors in the first few
\centering hundred milliseconds, precisely where a wearable agent must decide whether
\includegraphics[width=\columnwidth]{figures/task_overview.pdf} to wait or begin speaking.
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
only the past audio--video prefix is input; ten future activity states are
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
not semantic floor ownership.}
\label{fig:task}
\end{figure}
\textbf{Egocentric audiovisual conversation.} \textbf{Egocentric audiovisual conversation.}
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
@@ -90,6 +91,13 @@ prediction in noise \cite{oconnorrussell2025noise}. These studies make a
broad ``first audiovisual turn-taking'' claim untenable. What remains broad ``first audiovisual turn-taking'' claim untenable. What remains
less examined is how a single causal first-person stream should expose less examined is how a single causal first-person stream should expose
recent audio--visual evidence to forecast the invisible wearer's activity. recent audio--visual evidence to forecast the invisible wearer's activity.
EgoSpeak establishes that compact audio and video representations can be
fused for this task, but it does not answer whether a large integrated Omni
representation makes those cutoff-local cues readily available to a small
forecasting head. Conversely, work on active speakers or addressees need
not solve wearer anticipation: the visible interlocutor and the predicted
speaker occupy different roles. This leaves an architectural question,
rather than a claim that audiovisual evidence itself is new.
\textbf{Streaming Omni and temporal access.} \textbf{Streaming Omni and temporal access.}
Omni systems integrate modalities and support streaming interaction Omni systems integrate modalities and support streaming interaction
@@ -106,21 +114,31 @@ recent modality features outside the Thinker state, while \emph{partial-tail
refresh} (PTR) queries the observed prefix of the incomplete current block. refresh} (PTR) queries the observed prefix of the incomplete current block.
SWE denotes temporal locality, not shallow encoder layers; the argument SWE denotes temporal locality, not shallow encoder layers; the argument
concerns this pipeline, not an intrinsic deficit of every Omni model. concerns this pipeline, not an intrinsic deficit of every Omni model.
These interventions target different bottlenecks. SWE asks whether recent
encoder evidence improves prediction when the global state is held fixed;
PTR asks whether the same integrated backbone benefits from a fresher readout
without admitting future media. A gain from one does not establish a gain
from the other. In particular, the effect of PTR requires a matched
completed-only comparison at the same context cap, rather than comparison
with a longer or shorter history.
Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity This framing yields three concrete contributions. First, we formulate a
anticipation; an EasyCom analysis separating global, recent-audio, causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
pooled-visual, and privileged visible-activity information; and a specified ways to expose local evidence to a frozen Omni model. Second, on EasyCom we
EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics. compare global, recent-audio, and pooled-visual features, then use privileged
Only EasyCom screening scores are currently available: matched PTR effects, past visible activity only as an information audit, not a deployable input.
Ego4D transfer, and event-level behavior remain open tests. Third, we specify cross-dataset, endpoint, and hard-slice tests. EasyCom
screening is the only measured evidence; matched PTR, Ego4D transfer, and
event behavior remain open.
\begin{figure}[t] \begin{figure}[t]
\centering \centering
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side \caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
audio and optional pooled person-crop vision to a frozen-Omni head. PTR with recent audio and optional pooled person-crop evidence in the forecasting
temporarily queries the observed current-block prefix while reusing completed head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
2-s payloads. Privileged past-activity labels are not model inputs.} the observed current-block tail at each 200-ms cutoff. The tail is not
committed to persistent history.}
\label{fig:overview} \label{fig:overview}
\end{figure} \end{figure}