diff --git a/manuscript/targeted/Makefile b/manuscript/targeted/Makefile index 38bf904..edfab17 100644 --- a/manuscript/targeted/Makefile +++ b/manuscript/targeted/Makefile @@ -2,11 +2,11 @@ LATEXMK ?= latexmk .PHONY: all clean figure -all: figures/system_overview_onecolumn.pdf figures/task_overview.pdf +all: figures/system_overview_onecolumn.pdf mkdir -p build $(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=build main.tex -figure: figures/system_overview_onecolumn.pdf figures/task_overview.pdf +figure: figures/system_overview_onecolumn.pdf figures/%.pdf: figures/%.tex $(LATEXMK) -pdf -interaction=nonstopmode -halt-on-error -outdir=figures $< @@ -14,4 +14,3 @@ figures/%.pdf: figures/%.tex clean: $(LATEXMK) -C -outdir=build main.tex $(LATEXMK) -c -outdir=figures figures/system_overview_onecolumn.tex - $(LATEXMK) -c -outdir=figures figures/task_overview.tex diff --git a/manuscript/targeted/README.md b/manuscript/targeted/README.md index cd5e515..f7eeaa4 100644 --- a/manuscript/targeted/README.md +++ b/manuscript/targeted/README.md @@ -63,11 +63,13 @@ make ``` The editable sources are `main.tex`, `tables_compact.tex`, `references.bib`, -`figures/task_overview.tex`, and `figures/system_overview_onecolumn.tex`. +and `figures/system_overview_onecolumn.tex`. The earlier +`figures/task_overview.tex` is retained as unused source. Generated LaTeX files are written to `build/`. -The preview targets four technical pages plus a reference-only fifth page. Both -the task diagram (Fig. 1) and the system diagram (Fig. 2) occupy one column. +The preview targets four technical pages plus a reference-only fifth page. +The single-column method diagram (Fig. 1) has SWE on top and PTR below; +the earlier standalone task figure is not included. The current PDF reserves seven numbered result tables. Table 1 spans two columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer rows. Only three EasyCom models have archived complete horizon curves; other @@ -77,7 +79,7 @@ Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables Pending cells are explicitly unmeasured. Active table layouts are in `tables_compact.tex`; the earlier full-precision tabulars remain disabled in `main.tex` for source-level auditing. -Figure 3 is an H/Q case slot, not a selected or measured example. Its +Figure 2 is an H/Q case slot, not a selected or measured example. Its cutoff-marked timelines must be inserted only after the slice protocol and case-selection rule are frozen. The single-column system figure is generated from figures/system_overview_onecolumn.tex; the earlier full-width source diff --git a/manuscript/targeted/figures/system_overview_onecolumn.pdf b/manuscript/targeted/figures/system_overview_onecolumn.pdf index 32e431b..ed178e6 100644 Binary files a/manuscript/targeted/figures/system_overview_onecolumn.pdf and b/manuscript/targeted/figures/system_overview_onecolumn.pdf differ diff --git a/manuscript/targeted/figures/system_overview_onecolumn.tex b/manuscript/targeted/figures/system_overview_onecolumn.tex index ff4896a..42e0ffc 100644 --- a/manuscript/targeted/figures/system_overview_onecolumn.tex +++ b/manuscript/targeted/figures/system_overview_onecolumn.tex @@ -1,5 +1,5 @@ \documentclass[tikz,border=2pt]{standalone} -\usetikzlibrary{arrows.meta,positioning} +\usetikzlibrary{arrows.meta} \definecolor{navy}{HTML}{19324A} \definecolor{bluefill}{HTML}{EAF1F7} \definecolor{teal}{HTML}{087F78} @@ -8,43 +8,64 @@ \definecolor{orangefill}{HTML}{FFF0DF} \begin{document} \begin{tikzpicture}[ - font=\sffamily\small, + font=\sffamily\footnotesize, >=Latex, - box/.style={draw=navy,fill=bluefill,rounded corners=1.5pt, - align=center,minimum height=7mm,inner sep=2pt}, - local/.style={box,draw=teal,fill=tealfill}, - flow/.style={-{Latex[length=1.6mm]},draw=navy,line width=.6pt}] -\node[box,text width=20mm] (global) at (1.12,3.9) - {Observed AV\\8-s maximum}; -\node[box,text width=22mm] (thinker) at (4.13,3.9) - {Frozen Thinker\\readout $G$}; -\node[local,text width=20mm] (audio) at (1.12,2.9) - {Recent 1-s\\mixed audio}; -\node[local,text width=22mm] (aenc) at (4.13,2.9) - {Frozen encoder\\local $A$}; -\node[local,dashed,text width=20mm] (video) at (1.12,1.9) - {Past person\\crops (optional)}; -\node[local,dashed,text width=22mm] (venc) at (4.13,1.9) - {Frozen encoder\\pooled $V$}; -\node[box,text width=18mm] (head) at (7.05,2.9) - {Validities +\\trained head}; -\draw[flow] (global) -- (thinker); -\draw[flow,draw=teal] (audio) -- (aenc); -\draw[flow,draw=teal,dashed] (video) -- (venc); -\draw[flow] (thinker.east) -- ++(.18,0) |- (head.north); -\draw[flow,draw=teal] (aenc) -- (head); -\draw[flow,draw=teal,dashed] (venc.east) -- ++(.18,0) |- (head.south); -\node[box,text width=32mm] (out) at (5.7,.88) - {$10\times3$ future activity\\0.2--2.0-s horizons}; -\draw[flow] (head.south) -- (out.north); -\draw[draw=navy,line width=.5pt] (0,.12) -- (8.25,.12); -\node[anchor=west,text=navy] at (0,-.28) {Completed blocks}; -\foreach \x in {0,1.04,2.08} { - \draw[draw=navy,fill=bluefill] (\x,-.93) rectangle ++(.96,.35); + flow/.style={-{Latex[length=1.7mm]},draw=navy,line width=.7pt}] + +\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy] + at (.05,5.45) {(a) Short-window encoder evidence (SWE)}; +\node[anchor=east,font=\sffamily\scriptsize,text=navy] + at (8.25,5.45) {inputs $\leq t_n$}; + +\draw[draw=navy,fill=bluefill,rounded corners=2pt] + (.10,4.48) rectangle (3.80,5.18); +\node[align=center,text=navy] at (1.95,4.83) + {Frozen Omni Thinker\\global AV context $G$}; +\draw[draw=teal,fill=tealfill,rounded corners=2pt] + (.10,3.57) rectangle (3.80,4.27); +\node[align=center,text=teal] at (1.95,3.92) + {Frozen audio encoder\\recent 1-s audio $A$}; +\draw[draw=teal,fill=tealfill,dashed,rounded corners=2pt] + (.10,2.48) rectangle (3.80,3.43); +\node[align=center,text=teal] at (1.95,2.955) + {Frozen visual encoder\\recent person-crop clues $V$}; + +\draw[draw=navy,line width=.7pt] (3.80,4.83) -- (4.38,4.83) + -- (4.38,2.955) -- (3.80,2.955); +\draw[draw=teal,line width=.7pt] (3.80,3.92) -- (4.38,3.92); +\draw[flow] (4.38,3.92) -- (4.85,3.92); +\draw[draw=navy,fill=bluefill,rounded corners=2pt] + (4.95,3.57) rectangle (6.40,4.27); +\node[align=center,text=navy] at (5.675,3.92) {Fusion\\head}; +\draw[flow] (6.40,3.92) -- (6.78,3.92); +\draw[draw=orange,fill=orangefill,rounded corners=2pt] + (6.88,3.57) rectangle (8.30,4.27); +\node[align=center,text=orange] at (7.59,3.92) + {Future\\activity}; + +\draw[draw=navy!35,line width=.5pt] (.05,2.34) -- (8.30,2.34); +\node[anchor=west,font=\sffamily\footnotesize\bfseries,text=navy] + at (.05,2.07) {(b) Partial-tail refresh (PTR)}; + +\node[font=\sffamily\scriptsize,text=navy] at (1.58,1.70) + {reused completed blocks}; +\node[font=\sffamily\scriptsize,text=orange] at (4.08,1.70) + {observed at $t_n$}; +\foreach \x in {.15,1.15,2.15} { + \draw[draw=navy,fill=bluefill,rounded corners=1pt] + (\x,1.08) rectangle ++(.88,.46); + \node[text=navy] at (\x+.44,1.31) {2 s}; } -\draw[draw=orange,fill=orangefill] (3.12,-.93) rectangle ++(.96,.35); -\node[anchor=west,text=orange] at (4.25,-.76) {Observed tail}; -\node[anchor=west,text=navy] at (0,-1.33) - {Reuse completed payloads; query a temporary tail at $t_n$.}; +\draw[draw=orange,fill=orangefill,rounded corners=1pt] + (3.20,1.08) rectangle (4.95,1.54); +\node[text=orange] at (4.075,1.31) {partial tail}; +\draw[flow] (4.95,1.31) -- (5.41,1.31); +\draw[draw=navy,fill=bluefill,rounded corners=2pt] + (5.51,.96) rectangle (8.30,1.66); +\node[align=center,text=navy] at (6.905,1.31) + {Temporary Thinker\\readout $G_t$}; +\node[anchor=west,font=\sffamily\scriptsize,text=navy] + at (.15,.53) {Only completed blocks enter persistent history; the tail is not committed.}; + \end{tikzpicture} \end{document} diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index aef35a1..c0a7207 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -57,7 +57,14 @@ streaming, voice activity projection, conversational agents An egocentric conversational agent must decide \emph{when} to speak even though its camera wearer is usually invisible. It hears a mixed conversation and sees other participants, but must anticipate its own speech from the -observed stream before a turn boundary is known (Fig.~\ref{fig:task}). +observed stream before a turn boundary is known (Fig.~\ref{fig:overview}). +This decision differs from recognizing who is speaking now: the relevant +evidence may be a brief pause, overlap, or response by someone in view, while +the target is the wearer's \emph{future} activity. It also differs from +predicting a semantic end of turn or selecting an addressee. We therefore +study a deliberately narrower and measurable question: at each 200-ms +cutoff, how well can a causal observer forecast background, wearer, and +aggregate-other activity over the next 0.2--2.0~s? Anticipatory turn-taking has developed rapidly: VAP, audiovisual and triadic extensions, backchannel prediction, online end-of-turn detection, and event benchmarks address increasingly realistic timing decisions @@ -65,16 +72,10 @@ benchmarks address increasingly realistic timing decisions lin2025backchannel,lee2024eot,qi2026muvap,jiang2026turnbench}. Their input viewpoints and target decisions, however, are not interchangeable with forecasting the future activity of an unseen wearer. - -\begin{figure}[H] -\centering -\includegraphics[width=\columnwidth]{figures/task_overview.pdf} -\caption{Task, not an observed event example. At each 5-Hz decision cutoff, -only the past audio--video prefix is input; ten future activity states are -targets. ENTRY-like and YIELD-like refer to wearer activity transitions, -not semantic floor ownership.} -\label{fig:task} -\end{figure} +The distinction matters when comparing systems: a correct current-speaker +label or an utterance-final response can conceal errors in the first few +hundred milliseconds, precisely where a wearable agent must decide whether +to wait or begin speaking. \textbf{Egocentric audiovisual conversation.} EgoSpeak is the closest wearer-centered predecessor: it extracts audio and @@ -90,6 +91,13 @@ prediction in noise \cite{oconnorrussell2025noise}. These studies make a broad ``first audiovisual turn-taking'' claim untenable. What remains less examined is how a single causal first-person stream should expose recent audio--visual evidence to forecast the invisible wearer's activity. +EgoSpeak establishes that compact audio and video representations can be +fused for this task, but it does not answer whether a large integrated Omni +representation makes those cutoff-local cues readily available to a small +forecasting head. Conversely, work on active speakers or addressees need +not solve wearer anticipation: the visible interlocutor and the predicted +speaker occupy different roles. This leaves an architectural question, +rather than a claim that audiovisual evidence itself is new. \textbf{Streaming Omni and temporal access.} Omni systems integrate modalities and support streaming interaction @@ -106,21 +114,31 @@ recent modality features outside the Thinker state, while \emph{partial-tail refresh} (PTR) queries the observed prefix of the incomplete current block. SWE denotes temporal locality, not shallow encoder layers; the argument concerns this pipeline, not an intrinsic deficit of every Omni model. +These interventions target different bottlenecks. SWE asks whether recent +encoder evidence improves prediction when the global state is held fixed; +PTR asks whether the same integrated backbone benefits from a fresher readout +without admitting future media. A gain from one does not establish a gain +from the other. In particular, the effect of PTR requires a matched +completed-only comparison at the same context cap, rather than comparison +with a longer or shorter history. -Our contributions are a causal SWE/PTR interface for 5-Hz wearer activity -anticipation; an EasyCom analysis separating global, recent-audio, -pooled-visual, and privileged visible-activity information; and a specified -EasyCom--Ego4D evaluation with horizon, endpoint, and hard-slice diagnostics. -Only EasyCom screening scores are currently available: matched PTR effects, -Ego4D transfer, and event-level behavior remain open tests. +This framing yields three concrete contributions. First, we formulate a +causal, 5-Hz wearer-activity forecast and specify SWE and PTR as separable +ways to expose local evidence to a frozen Omni model. Second, on EasyCom we +compare global, recent-audio, and pooled-visual features, then use privileged +past visible activity only as an information audit, not a deployable input. +Third, we specify cross-dataset, endpoint, and hard-slice tests. EasyCom +screening is the only measured evidence; matched PTR, Ego4D transfer, and +event behavior remain open. \begin{figure}[t] \centering \includegraphics[width=\columnwidth]{figures/system_overview_onecolumn.pdf} -\caption{Two cutoff-aligned interfaces. SWE supplies recent encoder-side -audio and optional pooled person-crop vision to a frozen-Omni head. PTR -temporarily queries the observed current-block prefix while reusing completed -2-s payloads. Privileged past-activity labels are not model inputs.} +\caption{Method overview at cutoff $t_n$. Top: SWE joins frozen Omni context +with recent audio and optional pooled person-crop evidence in the forecasting +head. Bottom: PTR reuses completed 2-s blocks and temporarily queries only +the observed current-block tail at each 200-ms cutoff. The tail is not +committed to persistent history.} \label{fig:overview} \end{figure}