Expand targeted introduction and move method figure to page two
This commit is contained in:
@@ -2,11 +2,11 @@ LATEXMK ?= latexmk
|
|||||||
|
|
||||||
.PHONY: all clean figure
|
.PHONY: all clean figure
|
||||||
|
|
||||||
all: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
|
all: figures/system_overview_onecolumn.pdf
|
||||||
mkdir -p build
|
mkdir -p build
|
||||||
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
|
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex
|
||||||
|
|
||||||
figure: figures/system_overview_onecolumn.pdf figures/task_overview.pdf
|
figure: figures/system_overview_onecolumn.pdf
|
||||||
|
|
||||||
figures/%.pdf: figures/%.tex
|
figures/%.pdf: figures/%.tex
|
||||||
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
|
$(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $<
|
||||||
@@ -14,4 +14,3 @@ figures/%.pdf: figures/%.tex
|
|||||||
clean:
|
clean:
|
||||||
$(LATEXMK) -C -outdir=build main.tex
|
$(LATEXMK) -C -outdir=build main.tex
|
||||||
$(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex
|
$(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex
|
||||||
$(LATEXMK) -c -outdir=figures figures/task_overview.tex
|
|
||||||
|
|||||||
@@ -63,11 +63,13 @@ make
|
|||||||
```
|
```
|
||||||
|
|
||||||
The editable sources are `main.tex`, `tables_compact.tex`, `references.bib`,
|
The editable sources are `main.tex`, `tables_compact.tex`, `references.bib`,
|
||||||
`figures/task_overview.tex`, and `figures/system_overview_onecolumn.tex`.
|
and `figures/system_overview_onecolumn.tex`. The earlier
|
||||||
|
`figures/task_overview.tex` is retained as unused source.
|
||||||
Generated LaTeX files are written to `build/`.
|
Generated LaTeX files are written to `build/`.
|
||||||
|
|
||||||
The preview targets four technical pages plus a reference-only fifth page. Both
|
The preview targets four technical pages plus a reference-only fifth page.
|
||||||
the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column.
|
The single-column method diagram (Fig. 1) has SWE on top and PTR below;
|
||||||
|
the earlier standalone task figure is not included.
|
||||||
The current PDF reserves seven numbered result tables. Table 1 spans two
|
The current PDF reserves seven numbered result tables. Table 1 spans two
|
||||||
columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer
|
columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer
|
||||||
rows. Only three EasyCom models have archived complete horizon curves; other
|
rows. Only three EasyCom models have archived complete horizon curves; other
|
||||||
@@ -77,7 +79,7 @@ Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables
|
|||||||
Pending cells are explicitly unmeasured. Active table layouts are in
|
Pending cells are explicitly unmeasured. Active table layouts are in
|
||||||
`tables_compact.tex`; the earlier full-precision tabulars
|
`tables_compact.tex`; the earlier full-precision tabulars
|
||||||
remain disabled in `main.tex` for source-level auditing.
|
remain disabled in `main.tex` for source-level auditing.
|
||||||
Figure 3 is an H/Q case slot, not a selected or measured example. Its
|
Figure 2 is an H/Q case slot, not a selected or measured example. Its
|
||||||
cutoff-marked timelines must be inserted only after the slice protocol and
|
cutoff-marked timelines must be inserted only after the slice protocol and
|
||||||
case-selection rule are frozen. The single-column system figure is generated
|
case-selection rule are frozen. The single-column system figure is generated
|
||||||
from figures/system_overview_onecolumn.tex; the earlier full-width source
|
from figures/system_overview_onecolumn.tex; the earlier full-width source
|
||||||
|
|||||||
Binary file not shown.
@@ -1,5 +1,5 @@
|
|||||||
\documentclass[tikz,border=2pt]{standalone}
|
\documentclass[tikz,border=2pt]{standalone}
|
||||||
\usetikzlibrary{arrows.meta,positioning}
|
\usetikzlibrary{arrows.meta}
|
||||||
\definecolor{navy}{HTML}{19324A}
|
\definecolor{navy}{HTML}{19324A}
|
||||||
\definecolor{bluefill}{HTML}{EAF1F7}
|
\definecolor{bluefill}{HTML}{EAF1F7}
|
||||||
\definecolor{teal}{HTML}{087F78}
|
\definecolor{teal}{HTML}{087F78}
|
||||||
@@ -8,43 +8,64 @@
|
|||||||
\definecolor{orangefill}{HTML}{FFF0DF}
|
\definecolor{orangefill}{HTML}{FFF0DF}
|
||||||
\begin{document}
|
\begin{document}
|
||||||
\begin{tikzpicture}[
|
\begin{tikzpicture}[
|
||||||
font=\sffamily\small,
|
font=\sffamily\footnotesize,
|
||||||
>=Latex,
|
>=Latex,
|
||||||
box/.style={draw=navy,fill=bluefill,rounded corners=1.5pt,
|
flow/.style={-{Latex[length=1.7mm]},draw=navy,line width=.7pt}]
|
||||||
align=center,minimum height=7mm,inner sep=2pt},
|
|
||||||
local/.style={box,draw=teal,fill=tealfill},
|
\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy]
|
||||||
flow/.style={-{Latex[length=1.6mm]},draw=navy,line width=.6pt}]
|
at (.05,5.45) {(a) Short-window encoder evidence (SWE)};
|
||||||
\node[box,text width=20mm] (global) at (1.12,3.9)
|
\node[anchor=east,font=\sffamily\scriptsize,text=navy]
|
||||||
{Observed AV\\8-s maximum};
|
at (8.25,5.45) {inputs $\leq t_n$};
|
||||||
\node[box,text width=22mm] (thinker) at (4.13,3.9)
|
|
||||||
{Frozen Thinker\\readout $G$};
|
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
|
||||||
\node[local,text width=20mm] (audio) at (1.12,2.9)
|
(.10,4.48) rectangle (3.80,5.18);
|
||||||
{Recent 1-s\\mixed audio};
|
\node[align=center,text=navy] at (1.95,4.83)
|
||||||
\node[local,text width=22mm] (aenc) at (4.13,2.9)
|
{Frozen Omni Thinker\\global AV context $G$};
|
||||||
{Frozen encoder\\local $A$};
|
\draw[draw=teal,fill=tealfill,rounded corners=2pt]
|
||||||
\node[local,dashed,text width=20mm] (video) at (1.12,1.9)
|
(.10,3.57) rectangle (3.80,4.27);
|
||||||
{Past person\\crops (optional)};
|
\node[align=center,text=teal] at (1.95,3.92)
|
||||||
\node[local,dashed,text width=22mm] (venc) at (4.13,1.9)
|
{Frozen audio encoder\\recent 1-s audio $A$};
|
||||||
{Frozen encoder\\pooled $V$};
|
\draw[draw=teal,fill=tealfill,dashed,rounded corners=2pt]
|
||||||
\node[box,text width=18mm] (head) at (7.05,2.9)
|
(.10,2.48) rectangle (3.80,3.43);
|
||||||
{Validities +\\trained head};
|
\node[align=center,text=teal] at (1.95,2.955)
|
||||||
\draw[flow] (global) -- (thinker);
|
{Frozen visual encoder\\recent person-crop clues $V$};
|
||||||
\draw[flow,draw=teal] (audio) -- (aenc);
|
|
||||||
\draw[flow,draw=teal,dashed] (video) -- (venc);
|
\draw[draw=navy,line width=.7pt] (3.80,4.83) -- (4.38,4.83)
|
||||||
\draw[flow] (thinker.east) -- ++(.18,0) |- (head.north);
|
-- (4.38,2.955) -- (3.80,2.955);
|
||||||
\draw[flow,draw=teal] (aenc) -- (head);
|
\draw[draw=teal,line width=.7pt] (3.80,3.92) -- (4.38,3.92);
|
||||||
\draw[flow,draw=teal,dashed] (venc.east) -- ++(.18,0) |- (head.south);
|
\draw[flow] (4.38,3.92) -- (4.85,3.92);
|
||||||
\node[box,text width=32mm] (out) at (5.7,.88)
|
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
|
||||||
{$10\times3$ future activity\\0.2--2.0-s horizons};
|
(4.95,3.57) rectangle (6.40,4.27);
|
||||||
\draw[flow] (head.south) -- (out.north);
|
\node[align=center,text=navy] at (5.675,3.92) {Fusion\\head};
|
||||||
\draw[draw=navy,line width=.5pt] (0,.12) -- (8.25,.12);
|
\draw[flow] (6.40,3.92) -- (6.78,3.92);
|
||||||
\node[anchor=west,text=navy] at (0,-.28) {Completed blocks};
|
\draw[draw=orange,fill=orangefill,rounded corners=2pt]
|
||||||
\foreach \x in {0,1.04,2.08} {
|
(6.88,3.57) rectangle (8.30,4.27);
|
||||||
\draw[draw=navy,fill=bluefill] (\x,-.93) rectangle ++(.96,.35);
|
\node[align=center,text=orange] at (7.59,3.92)
|
||||||
|
{Future\\activity};
|
||||||
|
|
||||||
|
\draw[draw=navy!35,line width=.5pt] (.05,2.34) -- (8.30,2.34);
|
||||||
|
\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy]
|
||||||
|
at (.05,2.07) {(b) Partial-tail refresh (PTR)};
|
||||||
|
|
||||||
|
\node[font=\sffamily\scriptsize,text=navy] at (1.58,1.70)
|
||||||
|
{reused completed blocks};
|
||||||
|
\node[font=\sffamily\scriptsize,text=orange] at (4.08,1.70)
|
||||||
|
{observed at $t_n$};
|
||||||
|
\foreach \x in {.15,1.15,2.15} {
|
||||||
|
\draw[draw=navy,fill=bluefill,rounded corners=1pt]
|
||||||
|
(\x,1.08) rectangle ++(.88,.46);
|
||||||
|
\node[text=navy] at (\x+.44,1.31) {2 s};
|
||||||
}
|
}
|
||||||
\draw[draw=orange,fill=orangefill] (3.12,-.93) rectangle ++(.96,.35);
|
\draw[draw=orange,fill=orangefill,rounded corners=1pt]
|
||||||
\node[anchor=west,text=orange] at (4.25,-.76) {Observed tail};
|
(3.20,1.08) rectangle (4.95,1.54);
|
||||||
\node[anchor=west,text=navy] at (0,-1.33)
|
\node[text=orange] at (4.075,1.31) {partial tail};
|
||||||
{Reuse completed payloads; query a temporary tail at $t_n$.};
|
\draw[flow] (4.95,1.31) -- (5.41,1.31);
|
||||||
|
\draw[draw=navy,fill=bluefill,rounded corners=2pt]
|
||||||
|
(5.51,.96) rectangle (8.30,1.66);
|
||||||
|
\node[align=center,text=navy] at (6.905,1.31)
|
||||||
|
{Temporary Thinker\\readout $G_t$};
|
||||||
|
\node[anchor=west,font=\sffamily\scriptsize,text=navy]
|
||||||
|
at (.15,.53) {Only completed blocks enter persistent history; the tail is not committed.};
|
||||||
|
|
||||||
\end{tikzpicture}
|
\end{tikzpicture}
|
||||||
\end{document}
|
\end{document}
|
||||||
|
|||||||
@@ -57,7 +57,14 @@ streaming, voice activity projection, conversational agents
|
|||||||
An egocentric conversational agent must decide \emph{when} to speak even
|
An egocentric conversational agent must decide \emph{when} to speak even
|
||||||
though its camera wearer is usually invisible. It hears a mixed conversation
|
though its camera wearer is usually invisible. It hears a mixed conversation
|
||||||
and sees other participants, but must anticipate its own speech from the
|
and sees other participants, but must anticipate its own speech from the
|
||||||
observed stream before a turn boundary is known (Fig.~\ref{fig:task}).
|
observed stream before a turn boundary is known (Fig.~\ref{fig:overview}).
|
||||||
|
This decision differs from recognizing who is speaking now: the relevant
|
||||||
|
evidence may be a brief pause, overlap, or response by someone in view, while
|
||||||
|
the target is the wearer's \emph{future} activity. It also differs from
|
||||||
|
predicting a semantic end of turn or selecting an addressee. We therefore
|
||||||
|
study a deliberately narrower and measurable question: at each 200-ms
|
||||||
|
cutoff, how well can a causal observer forecast background, wearer, and
|
||||||
|
aggregate-other activity over the next 0.2--2.0~s?
|
||||||
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
|
Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic
|
||||||
extensions, backchannel prediction, online end-of-turn detection, and event
|
extensions, backchannel prediction, online end-of-turn detection, and event
|
||||||
benchmarks address increasingly realistic timing decisions
|
benchmarks address increasingly realistic timing decisions
|
||||||
@@ -65,16 +72,10 @@ benchmarks address increasingly realistic timing decisions
|
|||||||
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
|
lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}.
|
||||||
Their input viewpoints and target decisions, however, are not interchangeable
|
Their input viewpoints and target decisions, however, are not interchangeable
|
||||||
with forecasting the future activity of an unseen wearer.
|
with forecasting the future activity of an unseen wearer.
|
||||||
|
The distinction matters when comparing systems: a correct current-speaker
|
||||||
\begin{figure}[H]
|
label or an utterance-final response can conceal errors in the first few
|
||||||
\centering
|
hundred milliseconds, precisely where a wearable agent must decide whether
|
||||||
\includegraphics[width=\columnwidth]{figures/task_overview.pdf}
|
to wait or begin speaking.
|
||||||
\caption{Task, not an observed event example. At each 5-Hz decision cutoff,
|
|
||||||
only the past audio--video prefix is input; ten future activity states are
|
|
||||||
targets. ENTRY-like and YIELD-like refer to wearer activity transitions,
|
|
||||||
not semantic floor ownership.}
|
|
||||||
\label{fig:task}
|
|
||||||
\end{figure}
|
|
||||||
|
|
||||||
\textbf{Egocentric audiovisual conversation.}
|
\textbf{Egocentric audiovisual conversation.}
|
||||||
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
|
EgoSpeak is the closest wearer-centered predecessor: it extracts audio and
|
||||||
@@ -90,6 +91,13 @@ prediction in noise \cite{oconnorrussell2025noise}. These studies make a
|
|||||||
broad ``first audiovisual turn-taking'' claim untenable. What remains
|
broad ``first audiovisual turn-taking'' claim untenable. What remains
|
||||||
less examined is how a single causal first-person stream should expose
|
less examined is how a single causal first-person stream should expose
|
||||||
recent audio--visual evidence to forecast the invisible wearer's activity.
|
recent audio--visual evidence to forecast the invisible wearer's activity.
|
||||||
|
EgoSpeak establishes that compact audio and video representations can be
|
||||||
|
fused for this task, but it does not answer whether a large integrated Omni
|
||||||
|
representation makes those cutoff-local cues readily available to a small
|
||||||
|
forecasting head. Conversely, work on active speakers or addressees need
|
||||||
|
not solve wearer anticipation: the visible interlocutor and the predicted
|
||||||
|
speaker occupy different roles. This leaves an architectural question,
|
||||||
|
rather than a claim that audiovisual evidence itself is new.
|
||||||
|
|
||||||
\textbf{Streaming Omni and temporal access.}
|
\textbf{Streaming Omni and temporal access.}
|
||||||
Omni systems integrate modalities and support streaming interaction
|
Omni systems integrate modalities and support streaming interaction
|
||||||
@@ -106,21 +114,31 @@ recent modality features outside the Thinker state, while \emph{partial-tail
|
|||||||
refresh} (PTR) queries the observed prefix of the incomplete current block.
|
refresh} (PTR) queries the observed prefix of the incomplete current block.
|
||||||
SWE denotes temporal locality, not shallow encoder layers; the argument
|
SWE denotes temporal locality, not shallow encoder layers; the argument
|
||||||
concerns this pipeline, not an intrinsic deficit of every Omni model.
|
concerns this pipeline, not an intrinsic deficit of every Omni model.
|
||||||
|
These interventions target different bottlenecks. SWE asks whether recent
|
||||||
|
encoder evidence improves prediction when the global state is held fixed;
|
||||||
|
PTR asks whether the same integrated backbone benefits from a fresher readout
|
||||||
|
without admitting future media. A gain from one does not establish a gain
|
||||||
|
from the other. In particular, the effect of PTR requires a matched
|
||||||
|
completed-only comparison at the same context cap, rather than comparison
|
||||||
|
with a longer or shorter history.
|
||||||
|
|
||||||
Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity
|
This framing yields three concrete contributions. First, we formulate a
|
||||||
anticipation; an EasyCom analysis separating global, recent-audio,
|
causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable
|
||||||
pooled-visual, and privileged visible-activity information; and a specified
|
ways to expose local evidence to a frozen Omni model. Second, on EasyCom we
|
||||||
EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics.
|
compare global, recent-audio, and pooled-visual features, then use privileged
|
||||||
Only EasyCom screening scores are currently available: matched PTR effects,
|
past visible activity only as an information audit, not a deployable input.
|
||||||
Ego4D transfer, and event-level behavior remain open tests.
|
Third, we specify cross-dataset, endpoint, and hard-slice tests. EasyCom
|
||||||
|
screening is the only measured evidence; matched PTR, Ego4D transfer, and
|
||||||
|
event behavior remain open.
|
||||||
|
|
||||||
\begin{figure}[t]
|
\begin{figure}[t]
|
||||||
\centering
|
\centering
|
||||||
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
\includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf}
|
||||||
\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side
|
\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context
|
||||||
audio and optional pooled person-crop vision to a frozen-Omni head. PTR
|
with recent audio and optional pooled person-crop evidence in the forecasting
|
||||||
temporarily queries the observed current-block prefix while reusing completed
|
head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only
|
||||||
2-s payloads. Privileged past-activity labels are not model inputs.}
|
the observed current-block tail at each 200-ms cutoff. The tail is not
|
||||||
|
committed to persistent history.}
|
||||||
\label{fig:overview}
|
\label{fig:overview}
|
||||||
\end{figure}
|
\end{figure}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user