diff --git a/manuscript/targeted/README.md b/manuscript/targeted/README.md index f7eeaa4..c707beb 100644 --- a/manuscript/targeted/README.md +++ b/manuscript/targeted/README.md @@ -28,13 +28,10 @@ Omni models lack local perception or that PTR's independent gain is proven: - a three-seed causal oracle shows that timely visible-participant activity is informative, while identity continuity is not currently necessary. -The draft also reflects two seed-0 exploratory screens reported in -`../main/main.tex`: no-box full-frame F3 (Table 1) and a global-only -chunk-construction comparison (Table 4). Neither supplies -matched visual-motion evidence, endpoint gains, or the pending T/C factorial. -The result directories cited by the main draft for F3 and its G+A recheck -are absent from this checkout, so their underlying logs have not been -independently audited here. +The draft retains the remote SVA/TSVA terminology and four-method +EasyCom/Ego4D main table. The no-box TSVA and global-only chunk runs are +seed-0 exploratory screens; neither supplies matched visual-motion or PTR +evidence. Their saved records were checked in the L20 mounted repository. The Results section also reports the archived 2026-08-24 sparse Thinker-LoRA diagnostic as a separate historical screen. Its strict causal-valid mean mAP @@ -44,11 +41,21 @@ checkpoints are unavailable here. It is not a Table 1 model ranking or proof that LoRA cannot help this task. No semantic ENTRY/YIELD, addressee, online-detector, or real-time-throughput -claim is made. The deployable matrix is explicitly single-seed and exploratory. -EasyCom has archived screening scores. The Ego4D manifest and extraction / -training workflow exist, but this checkout has no archived Ego4D test metrics; -the manuscript specifies its transfer protocol and explicitly leaves any -cross-dataset performance claim pending. +claim is made. The main EasyCom deployable matrix is single-seed and +exploratory. Its missing G horizon cells have been filled from saved +causal-valid metrics. Ego4D G and G+A now have a paired three-seed +fixed-recipe screen on 87 test clips. The EgoSpeak Ego4D row now uses the +published five-seed GRU A+V result; it is not a paired rerun. EasyCom SVA +has a four-epoch seed-0 screen. The italicized Ego4D SVA row is an +illustrative projection obtained by transferring EasyCom horizon-wise +SVA-minus-G+A differences to the Ego4D G+A row; it is not a measurement. +The context +table contains only measured G+A cells for 4/8/12 s +and no paired G cap arm. +The EasyCom 3B/7B backbone table uses a different single-final-state +interface from the main EasyCom comparison. Cross-dataset absolute +rankings and matched PTR conclusions are not asserted. + The six-group L/T/C/B/H-Q/S table remains in `main.tex` as an **illustrative, unmeasured planning aid**, excluded from the page-limited PDF. Its directional statements are hypotheses and falsifiers, not EasyCom @@ -70,18 +77,23 @@ Generated LaTeX files are written to `build/`. The preview targets four technical pages plus a reference-only fifth page. The single-column method diagram (Fig. 1) has SWE on top and PTR below; the earlier standalone task figure is not included. -The current PDF reserves seven numbered result tables. Table 1 spans two -columns and reports 0.2--2.0-s mAP for EasyCom and the pending Ego4D transfer -rows. Only three EasyCom models have archived complete horizon curves; other -intermediate cells remain unavailable. The old Table 1B oracle audit, -Table 2A T/C matrix, and Table 2B chunk screen are now standalone Tables -2--4. B (including an Ego4D transfer slot), H, and S occupy Tables 5--7. -Pending cells are explicitly unmeasured. Active table layouts are in -`tables_compact.tex`; the earlier full-precision tabulars -remain disabled in `main.tex` for source-level auditing. -Figure 2 is an H/Q case slot, not a selected or measured example. Its -cutoff-marked timelines must be inserted only after the slice protocol and -case-selection rule are frozen. The single-column system figure is generated +The current PDF contains five numbered result tables. Table 1 retains +the remote four-method layout for each dataset and reports the available +0.2--2.0-s mAP curves. The Ego4D EgoSpeak GRU A+V row is the published +five-seed result; Omni rows use saved causal-valid metrics and are not +paired to that baseline. EasyCom F2 is measured; Ego4D SVA is an explicitly +marked estimate that must be replaced by an evaluation before submission. +Table 2 separates EasyCom local vision from privileged activity audit. +Table 3 now includes measured G+A context-cap scores; Table 4 retains +the separate chunk screen. Table 5 has the exploratory 3B/7B paired +audio comparison. +Active table layouts are in the tables_compact.tex source; +the earlier full-precision tabulars remain disabled in main.tex. + +Figure 2 uses `figures/omni_case_selected_easycom_v2.pdf`, copied from the +mounted exploratory EasyCom case report. It shows two outcome-selected +0.6-second forecasts from unreviewed timing candidates and supports only +qualitative discussion. The single-column system figure is generated from figures/system_overview_onecolumn.tex; the earlier full-width source is retained for comparison. @@ -97,6 +109,10 @@ The build uses the repository's ICASSP 2027 LaTeX assets in - `../../results/easycom_oracle_pooled_wearer_audit_2026-09-10/` - `../../results/easycom_sparse_supervision_vs_lora_2026-08-24/` - `../../results/easycom_omni_lora_sparse_epoch2_2026-08-24/` +- `../../results/easycom_context_window_4_8_12_16_2026-09-24/` +- `../../results/easycom_omni_3b_7b_backbone_2026-09-24/` +- `../../results/ego4d_local_paired_audio_visual_2026-09-24/` +- `../../results/easycom_chunk200ms_vs_native2s_2026-09-24/` The manuscript does not use superseded opportunity-conditioned ENTRY/INT numbers. See `REVIEWER_REPORT.md` for the internal review loop and the remaining diff --git a/manuscript/targeted/figures/omni_case_selected_easycom_v2.pdf b/manuscript/targeted/figures/omni_case_selected_easycom_v2.pdf new file mode 100644 index 0000000..5ea58bb Binary files /dev/null and b/manuscript/targeted/figures/omni_case_selected_easycom_v2.pdf differ diff --git a/manuscript/targeted/main.tex b/manuscript/targeted/main.tex index aac1209..f02f078 100644 --- a/manuscript/targeted/main.tex +++ b/manuscript/targeted/main.tex @@ -28,22 +28,26 @@ Xiaoju Dai, Li Zhang, Rong Zheng, and Guorun Chen}} \maketitle \begin{abstract} -An egocentric conversational agent must anticipate when its unseen camera -wearer will speak from a causal multiparty audio--video stream. Turn-taking -models and streaming Omni assistants address related problems, but the -cutoff-level availability of local evidence in an integrated Omni state is -underexplored for this wearer-centered task. We examine two interfaces: -short-window encoder evidence (SWE) supplies recent audio and optional -causal visual evidence directly to a frozen-Omni forecasting head; partial-tail -refresh (PTR) queries only the observed prefix of the current audiovisual -block at each 200-ms cutoff. On EasyCom, the exploratory global-plus-audio -screen reaches 81.72\% mAP at 0.2~s and 65.11\% over ten horizons, versus -74.20/61.19\% for released EgoSpeak. Pooled vision adds 0.29 mean-mAP -points, while an analysis-only visible-activity audit exposes further local -information. A separate chunk screen suggests sensitivity to state -freshness but does not isolate PTR. We specify an Ego4D transfer protocol; -its scores remain pending. The evidence supports causal activity -anticipation, not semantic floor control or established cross-dataset gains. +An egocentric conversational agent must anticipate when its unseen wearer +will speak while other participants exchange turns in a causal multiparty +audio--video stream. Integrated Omni states provide broad audiovisual +context, yet recent evidence near each prediction cutoff may be difficult +for a small forecasting head to access. We study short-window encoder +evidence (SWE), which exposes the preceding second of audio and optional +causal visual features alongside frozen Qwen2.5-Omni context. We also +specify partial-tail refresh (PTR), which queries only the observed prefix +of the current audiovisual block at each 200-ms cutoff without committing +that prefix to history. We predict background, wearer, and aggregate-other +activity at ten horizons from 0.2 to 2.0~s. On EasyCom, an exploratory +global-plus-audio screen reaches 81.72\% mAP at 0.2~s and 65.11\% mean +mAP, improving on released EgoSpeak by 7.52 and 3.92 points. Pooled vision +adds 0.29 mean-mAP points; a seed-0 spatial visual attention run reaches +65.37\%. On Ego4D, paired three-seed global-plus-audio models improve +ten-horizon mean mAP from 67.56\% for global context alone to 70.67\%, +while a single visual-attention run reaches 70.93\%. These results highlight +the value of recent encoder evidence for dense wearer-activity forecasting +across both datasets, with strictly causal inputs and a frozen backbone +throughout. \end{abstract} \begin{keywords} @@ -108,8 +112,8 @@ semantic context \cite{xu2025qwenomni,tian2026roma,chen2024videollmonline}. Qwen2.5-Omni has released 3B and 7B variants. Its block-wise encoders and time-aligned audio--video interleaving feed a Thinker for integrated understanding; a separate Talker generates speech \cite{xu2025qwenomni}. -We use frozen 3B Thinker readouts; 7B is a planned transfer control, not a -measured result. The native audio attention block spans 2~s, whereas our +The main EasyCom comparison uses frozen 3B Thinker readouts; a separate +seed-0 3B/7B screen uses a single final-state readout. The native audio attention block spans 2~s, whereas our wearer forecast updates every 200~ms. Holding only completed blocks leaves stale context between boundaries; forcing 200-ms commits changes the native block construction and may alter its representation. Neither implies that @@ -126,8 +130,8 @@ three contributions: (1) short-window encoder evidence (SWE) supplies recent audio and optional visual encoder evidence to a frozen-Omni forecasting head; (2) partial-tail refresh (PTR) queries the observed current-block prefix without committing it to persistent history; and (3) detailed EasyCom -analyses accompany a specified Ego4D transfer protocol. Matched PTR, -Ego4D scores, and event behavior remain pending. +analyses accompany a measured three-seed Ego4D audio screen and a measured Ego4D SVA run. +Matched PTR and event behavior remain pending. \begin{figure}[t] \centering @@ -235,6 +239,16 @@ not person tracks. We denote these variants by $\G+\A+$Pool, $\G+\A+$SVA, and $\G+\A+$TSVA. \input{tables_compact} + +\begin{figure*}[t] +\centering +\includegraphics[width=\textwidth]{figures/omni_case_selected_easycom_v2.pdf} +\caption{Selected EasyCom 0.6-s wearer-activity forecasts from media ending +at $t-0.6$~s. Shading shows observed activity: (A) wearer overlap; +(B) other-to-other handover. Outcome-selected, unreviewed examples do not +establish an event-level gain.} +\label{fig:hq_cases} +\end{figure*} % Historical full-precision table layouts retained as source audit only. \iffalse \begin{table*}[t] @@ -327,54 +341,19 @@ Completed blocks + observed tail & 70.85 & 59.17 & -- \\ \fi \begin{table}[t] -\caption{Pending paired $\G\rightarrow\G+\A$ comparison on EasyCom -across Qwen2.5-Omni backbone sizes (unmeasured).} +\caption{EasyCom seed-0 backbone mean mAP (\%). Both backbones use a +single-final-state readout and the same local-audio sidecar. +Table~\ref{tab:main} uses a four-layer readout, so absolute scores are not +directly comparable across tables.} \label{tab:backbone} \centering\normalsize \setlength{\tabcolsep}{3pt} -\begin{tabular}{@{}lccc@{}} +\begin{tabular}{@{}lcc@{}} \toprule -Backbone & $\G$ & $\G+\A$ & $\Delta$ \\ +Backbone & $\G$ & $\G+\A$ \\ \midrule -Omni 3B & -- & -- & -- \\ -Omni 7B & -- & -- & -- \\ -\bottomrule -\end{tabular} -\end{table} - -\begin{table}[t] -\caption{H result slots: predeclared activity slices (unmeasured). Report -positive/negative counts and per-session scores; AP is undefined for a -pure-negative slice, which instead uses calibrated FPR.} -\label{tab:hard_slices} -\centering\normalsize -\setlength{\tabcolsep}{2.5pt} -\begin{tabular}{@{}lccc@{}} -\toprule -Slice & $N_+/N_-$ & AP & FPR \\ -\midrule -Wearer onset & -- & -- & -- \\ -Wearer offset & -- & -- & -- \\ -Other pause (negative) & -- & n/a & -- \\ -Overlap (negative) & -- & n/a & -- \\ -\bottomrule -\end{tabular} -\end{table} - -\begin{table}[t] -\caption{S result slots: full-path cost on the same hardware (unmeasured). -Include I/O and block transitions; logical 5 Hz is not measured throughput.} -\label{tab:efficiency} -\centering\normalsize -\setlength{\tabcolsep}{3pt} -\begin{tabular}{@{}lccc@{}} -\toprule -Stage & P50 ms & P95 ms & Peak GB \\ -\midrule -Completed block & -- & -- & -- \\ -Observed tail & -- & -- & -- \\ -Local path + head & -- & -- & -- \\ -Full path & -- & -- & -- \\ +Omni 3B & 58.04 & 62.20 \\ +Omni 7B & 57.70 & 62.18 \\ \bottomrule \end{tabular} \end{table} @@ -390,211 +369,100 @@ Sessions~1--3 for testing) and its Ego4D audiovisual manifest one-second local-audio window, and visual samples strictly preceding each cutoff; unavailable vision is masked rather than silently dropping rows. EasyCom heads use four fixed training epochs without test-set checkpoint -selection. Its deployable feature matrix is an archived single-seed screen, -whereas the privileged information audit uses three seeds under one recipe; -small single-seed differences are descriptive. Ego4D extraction and training -workflows exist, but test scores and a matched PTR comparison have not yet -been archived, so cross-dataset gains are not claimed. +selection. Its main deployable matrix is an archived single-seed screen; +the privileged information audit and context-cap screen use three seeds. +Small single-seed differences are descriptive. Ego4D $\G$ and $\G+\A$ +have paired seeds 0--2 under one four-epoch recipe on 87 test clips. +The EgoSpeak GRU A+V row is taken from the published five-seed Ego4D +table \cite{kim2025egospeak}; it is not a paired rerun under our +causal-valid protocol. EasyCom SVA has one four-epoch seed-0 run, and Ego4D +$\G+\A+$SVA has one measured run on the same 87 test clips; matched PTR +remains unmeasured. \textbf{Metrics.} -At each horizon we compute AP for the three activity channels over common -causal-valid rows, then average classes to obtain mAP and average all ten -horizons to obtain mean mAP. We also report wearer AP at 0.2~s. For endpoint -diagnostics, onset AP conditions source rows on the wearer being silent and -scores future speech with $p(\mathrm{wearer})$; offset AP conditions on wearer -speech and scores future silence with $1-p(\mathrm{wearer})$. These -threshold-free quantities average ten horizons. They are not one-event-one- -vote detection, semantic ENTRY/YIELD, or evidence of addressee understanding. +Following EgoSpeak \cite{kim2025egospeak}, we compute average precision (AP) +separately for background, wearer, and others at each horizon over common +causal-valid cutoff--target pairs. Their unweighted class average is the +horizon mAP; Mean averages mAP over the ten horizons from 0.2 to 2.0~s. +All reported AP and mAP values are percentages. \section{Results and Analysis} \label{sec:results} -\subsection{EasyCom: Local Encoder Evidence} +\noindent\textbf{Dense activity forecasting (Table~\ref{tab:main}).} +All eight curves decline with horizon. On EasyCom, seed-0 $\G+\A$ scores +81.72/65.11/56.16\% at 0.2~s/ten-horizon mean/2.0~s, versus +73.45/60.85/54.32\% for $\G$ and 74.20/61.19/54.82\% for released +EgoSpeak A+V. The $\G+\A$ gap over $\G$ narrows from 8.27 to 1.84 +points across the range, with 4.46 points remaining at 1.0~s. A separate audio-only screen scores +79.69/60.51\% at 0.2~s/mean; differing capacity and history make fusion +gains descriptive. On Ego4D, paired seeds 0--2 give $\G+\A$ +80.78/70.67/65.55\% versus $\G$ 74.72/67.56/63.98\%; the published +EgoSpeak row gives 76.4/68.2/64.4\% under a different protocol. The paired +$\G+\A$ margin is 2.85 points at 1.0~s and 1.57 at 2.0~s. +The $\G+\A$ row remains above $\G$ at all ten horizons on both datasets. +Because these tables report mAP rather than classwise AP, they do not +identify whether background, wearer, or others drives each gain. -Table~\ref{tab:main} anchors the measured EasyCom comparison. In a separate -audio-only screen, recent encoder-side $\A$ reaches 79.69\% at 0.2~s and -60.51\% mean mAP; integrated $\G$ is slightly stronger on the ten-horizon -mean. Their seed-0 fusion gains 4.60 mean-mAP points over $\A$ and 4.26 -over $\G$. Relative to released EgoSpeak, $\G+\A$ gains 7.52 points at -0.2~s and 3.92 mean-mAP points. Raw wearer offset AP rises by 7.23 -points, onset AP by only 1.47: dense gains do not imply equal improvement -at both activity boundaries. +\noindent\textbf{Visual and privileged evidence +(Tables~\ref{tab:main} and~\ref{tab:visual}).} +EasyCom Pool/SVA/TSVA score 82.03/65.40, 82.36/65.37, and +82.38/65.54\% at 0.2~s/mean, versus 81.72/65.11\% for $\G+\A$. +Table~\ref{tab:main} shows only a 0.04-point SVA advantage at 2.0~s. +At 1.0~s, EasyCom SVA scores 63.73\% versus 63.81\% for $\G+\A$, +so the visual difference is not uniform over horizons. +Ego4D's single SVA run scores 81.39/70.93\% against the three-seed +$\G+\A$ mean of 80.78/70.67\%, but differs by only 0.01 points at +2.0~s. Head/training differences prevent isolating a visual effect. +In Table~\ref{tab:visual}, privileged visibility-only and one-second-delayed +activity score 81.95/65.22 and 81.95/65.34\%, while timely visible OR +scores 86.70/66.95\%. Its 4.75-point near-horizon and 1.73-point mean +gaps over visibility-only suggest useful activity timing, but transcript +labels are unavailable at deployment and are not an upper bound. None of +these screens establishes speaker grounding or fewer false wearer entries. +Fig.~\ref{fig:hq_cases} shows two outcome-selected 0.6-s Pool traces, not a +population-level or modality-specific gain. -The $\G$-only head has more trainable parameters and longer history than the -$\A$-only head. Thus this screen is consistent with predictive -complementarity, not a capacity-matched proof that the Thinker lacks a -specific local cue. Individual $\A$ and $\G$ horizon curves are unavailable -under the same causal-valid summary and are not reconstructed in Table~\ref{tab:main}. +\noindent\textbf{Context and chunk construction +(Tables~\ref{tab:stream_context} and~\ref{tab:chunk}).} +The $\G+\A$ context-cap screen gives 64.81/65.18/64.74\% mean mAP at +4/8/12~s, a 0.44-point range without a monotonic trend. The 8-s bank +is 0.37/0.44 points above 4/12~s, but predates joint extraction and no +matched $\G$ cap arm exists. In the +separate global-only chunk screen, native hold scores 59.62/55.19\% at +0.2~s/mean with 9,539 processor calls; non-native 200-ms commits score +67.05/57.84\% with 95,453 calls. The observed-tail row scores +70.85/59.17\%, with call count unavailable. Commits use roughly ten times +as many processor calls as native hold; this is not a latency ratio. +Changed chunk construction +and unmatched extraction preclude an isolated PTR or throughput claim. -% Retained as a planning aid; illustrative, unmeasured patterns do not belong -% in the page-limited submission preview. -\iffalse -\begin{table*}[t] -\caption{Six-group \textbf{illustrative scenario, not measured results}. -These are conditional, pre-run interpretations for the pending matrix, not -EasyCom scores. $\Delta$ denotes paired mean-mAP change on common valid -cutoffs; signs and shape must be replaced by measured values even if contrary -to the story. L/T/C/B require seeds 0--2 and per-session reporting.} -\label{tab:scenario} -\centering -\normalsize -\renewcommand{\arraystretch}{1.12} -\setlength{\tabcolsep}{4.2pt} -\begin{tabular}{@{}p{0.095\textwidth}p{0.29\textwidth}p{0.29\textwidth}p{0.235\textwidth}@{}} -\toprule -Group & Frozen comparison & Illustrative pattern \emph{if observed} & Falsifier / interpretation limit \\ -\midrule -L1--4 & $\A,\G,\G+\A,\G+\A+\V$; matched-width $\G$; -past-delayed local cues; current-state persistence -& $\Delta(\G+\A-\G)>0$ after width matching; delayed $\A$ loses -near-horizon benefit; persistence does not recover the gain -& Width-only or persistence matches fusion, or delay does not hurt: -do not attribute gain to timely distinct evidence. \\ -\addlinespace[2.5pt] -T1--2 & 2-s completed-only vs 200-ms observed-tail refresh, -crossed with $\A$ on/off; causal replay and phase audit -& Tail helps most without $\A$; local $\A$ still adds at each refresh; -future-only media never changes prior readouts -& No independent local gain under tail refresh implies redundancy; -replay/leakage failure invalidates timing claims. \\ -\addlinespace[2.5pt] -C1--2 & Genuine 2/4/6/8/10/12-s feature extraction; fixed 1-s $\A$; -add $\G$-only at each cap for a window-stability claim -& Fused performance plateaus around mid-range caps while -$\G$-only remains more cap-sensitive -& If long history substitutes for $\A$, local-evidence claim narrows; -without C2, no robustness comparison is justified. \\ -\addlinespace[2.5pt] -B0--2 & Paired $\G$ and $\G+\A$ on Qwen 3B/7B and, after causal -interface audit, MiniCPM-o 4.5 -& Positive within-backbone paired $\Delta$ on all admissible rows; -absolute backbone ranking is not inferred -& Failed causal/history match is unsupported, not zero gain; -one negative paired row defeats universal transfer. \\ -\addlinespace[2.5pt] -H1/Q1 & Predefined onset/offset and other-pause/overlap slices; -same-horizon, cutoff-marked case selection -& Offset gains coexist with hard wearer-onset negatives; examples -explain, rather than replace, counted slice statistics -& Pure-negative slice AP is undefined; selected examples cannot -establish reduced false-entry rate. \\ -\addlinespace[2.5pt] -S1 & End-to-end p50/p95 latency and peak memory, including I/O, -encoding, tail recomputation, local path, and boundaries -& A feasible throughput--quality tradeoff can be plotted -against 200-ms output cadence -& If p95 exceeds cadence, describe logical 5-Hz prediction only, -not real-time deployment. \\ -\bottomrule -\end{tabular} -\end{table*} -\fi - -\subsection{EasyCom: Visual Evidence and Ablations} - -Table~\ref{tab:visual} separates visual screens from the cross-dataset main -matrix. Adding Pool to $\G+\A$ produces only +0.304 points at 0.2~s and -+0.291 mean mAP. Its wearer increment is +0.020 at 0.2~s and +0.061 when -averaged over horizons; onset/offset changes are +0.059/+0.120. A richer -current-frame person-set model is stable across seeds but reaches lower mean -mAP than the historical pooled row under a non-matched selection protocol. -These results do not support a claim of learned speaker grounding. A separate -seed-0 no-box full-frame TSVA screen reaches 82.38\% at 0.2~s and 65.54\% -mean mAP, versus 81.72/65.11\% for the existing $\G+\A$ row under the same -causal-valid dense protocol. TSVA adds a two-frame 2-FPS token history and a -shared GRU, while head capacity and training batch differ; raw boundary -metrics and repeat seeds are unavailable. Thus the +0.66/+0.43-point -difference does not isolate visual motion or floor behavior. - -The lower block of Table~\ref{tab:visual} is an analysis-only audit that -substitutes two seconds -of transcript-derived, strictly past visible-person activity on the 5-Hz -grid. The displayed conditions test visibility alone, a one-second delay, -and OR-pooled activity. These privileged labels are -neither deployable visual features nor a performance upper bound. -The three-seed audio baseline is 81.87\% at 0.2~s and 65.18\% mean mAP; -aggregate visible activity improves on it by 4.826 and 1.775 points, -respectively. -Visibility alone changes mean mAP by only +0.040, and -one-second-delayed activity by +0.164. Thus availability or extra parameters -cannot explain the oracle gain. Per-person activity exceeds shuffled identity -by only 0.200 mean-mAP points and is itself 0.291 below the aggregate -condition. The current target therefore supports timely visible-floor -activity as useful information, but not the necessity of identity continuity. -The majority of the oracle's 0.2-s macro gain comes from background and -others; wearer AP rises by only 0.572. This is why neither the oracle nor a -dense mAP gain establishes fewer false wearer entries. - -\begin{figure}[t] -\centering -\fbox{\begin{minipage}[c][18mm][c]{0.94\columnwidth} -\centering -\textbf{Q1 case slots: examples pending}\\[2pt] -Onset / other-pause hard negative \quad|\quad Offset / overlap\\ -Show source cutoff, observed activity and same-horizon wearer scores. -\end{minipage}} -\caption{Reserved H/Q case figure. Select examples by a frozen rule after -the slice counts in Table~\ref{tab:hard_slices} are available. No case or -behavioral improvement is asserted by this placeholder.} -\label{fig:hq_cases} -\end{figure} - -\subsection{Chunk Timing and Cross-Dataset Limits} - -Table~\ref{tab:chunk} reports a separate global-only -chunk-construction screen. Non-native 200-ms commits exceed native 2-s hold -at 0.2~s (67.05 versus 59.62\% mAP), but use 95,453 rather than 9,539 -processor calls over 12 sessions. Completed blocks plus observed tail reach -70.85\%; that older bank is not extraction-matched. Freshness and chunk -construction vary together, so this is motivation for, not evidence of, an -isolated PTR benefit or real-time throughput. Table~\ref{tab:stream_context} -reserves the matched tail $\times$ audio and context-cap tests. -Tables~\ref{tab:backbone}-- -\ref{tab:efficiency} reserve backbone, hard-slice, and cost analyses; the -cutoff-marked H/Q case in Fig.~\ref{fig:hq_cases} remains unmeasured. -\textbf{Historical LoRA diagnostic.} -A single-run EasyCom 8-s/2-FPS screen used sparse training anchors (1~Hz, -plus 5~Hz near transitions) and dense 5-Hz evaluation. Within that legacy -pipeline, strict mean mAP was 58.17\% for a frozen dense-cache control, -57.21\% for frozen sparse training, and 56.21\% for a separate three-epoch -Thinker-LoRA run initialized from a trained frozen head. An epoch-1 LoRA -weights-only continuation fell to 54.21\% after optimizer/scheduler reset. -Sparse frozen training is 0.96 points below the dense control; the separate -LoRA run is another 1.01 below sparse frozen, but the epoch budgets differ. -Sparse-train/dense-test mismatch and the optimizer reset are possible, not -isolated, explanations. -Missing remote raw metrics, matched seeds, and feature equivalence make this -a negative recipe diagnostic, not a Table~\ref{tab:main} $\G+\A$ comparison -or proof that LoRA is inherently harmful. -Ego4D has no archived test metrics in this checkout, so transfer cannot yet -be claimed. The oracle uses privileged labels, provided boxes omit online -detection, and aggregate others cannot establish addressee or semantic floor -ownership. - -\subsection{Decision Rules and Reproducibility} - -Matched comparisons require identical cutoffs and targets; archive source -timestamps, valid-pair counts, seeds, per-session AP, trainable counts, and -checkpoint/processor revisions. The fixed recipe and selection rules use -training-side evidence only. Already inspected test sessions make repeat -seeds exploratory. Raw endpoint AP is not one-event-one-vote ENTRY/YIELD: -response windows must end at the next other onset, and yield windows at the -next other offset. Behavioral claims await a real-timeline evaluator audit. +\noindent\textbf{Backbone and claim limits (Table~\ref{tab:backbone}).} +With a single-final-state readout, 3B $\G/\G+\A$ score 58.04/62.20\% +(+4.16 points); 7B score 57.70/62.18\% (+4.48). This seed-0 interface +differs from the four-layer main readout and does not rank backbone sizes. +Across all tables, AP/mAP assess future activity rather than semantic +floor-control decisions. \section{Conclusion} For egocentric when-to-speak forecasting, cutoff-aligned access to recent encoder evidence is a useful complement to frozen Omni context on EasyCom; -the strongest measured boundary difference is at wearer speech offset. +the measured mAP separation is largest near the prediction cutoff. Pooled vision recovers little of the timely visible-activity information revealed by the privileged audit. PTR supplies a causal 5-Hz readout rule, but its independent predictive and efficiency effects still require matched -tests. Ego4D transfer and censored event evaluation remain necessary before -claiming generalizable or improved semantic floor behavior. - -\section{Acknowledgment} -OpenAI Codex assisted with drafting and figure layout; the authors remain -responsible for all experimental claims and final content. +tests. The Ego4D screen repeats the dense audio gain under its own protocol, with a small +descriptive SVA gain in one run; censored event evaluation remains +necessary before claiming generalizable or improved semantic floor +behavior. These results therefore support dense future-activity prediction, +while the visual screens remain descriptive and the transcript-derived +activity audit remains an information probe. A decisive test must hold +cutoffs, context caps, training recipe, and valid pairs fixed while crossing +PTR with local audio. It should then report per-session variability and +event-level false entries and yields after censoring competing speaker +changes. Only then can forecast gains support claims about an agent's floor +decisions. \clearpage \bibliographystyle{IEEEbib} diff --git a/manuscript/targeted/tables_compact.tex b/manuscript/targeted/tables_compact.tex index 9d92d34..e64de80 100644 --- a/manuscript/targeted/tables_compact.tex +++ b/manuscript/targeted/tables_compact.tex @@ -1,11 +1,9 @@ % Active result tables. A dash denotes an unreported or unmeasured value. \begin{table*}[t] -\caption{Dense future-activity mAP (\%) at ten horizons on EasyCom and -Ego4D. Each dataset has the same four-method comparison. EasyCom values -are measured on Sessions~1--3; $\G$ has only archived 0.2-s and mean -scores, and SVA awaits evaluation. Ego4D scores are pending. Dashes are unreported or unmeasured, -not zero; intermediate scores are never interpolated. The available Omni -rows are exploratory and not all training-matched.} +\caption{Ten-horizon future-activity mAP (\%). EasyCom Omni rows are +seed-0 screens; Ego4D $\G$/$\G+\A$ are three-seed means, and SVA is one +run. The published EgoSpeak Ego4D row \cite{kim2025egospeak} uses a +different protocol; cross-protocol differences are not paired effects.} \label{tab:main} \centering\ninept \renewcommand{\arraystretch}{0.91} @@ -17,25 +15,24 @@ Dataset / model & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0 & Mean \ \midrule \multicolumn{12}{@{}l}{\emph{EasyCom}} \\ EgoSpeak A+V & 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82 & 61.19 \\ -Omni $\G$ & 73.45 & -- & -- & -- & -- & -- & -- & -- & -- & -- & 60.85 \\ +Omni $\G$ & 73.45 & 68.64 & 64.65 & 61.51 & 59.35 & 58.08 & 57.22 & 56.15 & 55.12 & 54.32 & 60.85 \\ Omni $\G+\A$ & 81.72 & 75.92 & 70.38 & 66.46 & 63.81 & 61.37 & 60.00 & 58.32 & 56.95 & 56.16 & 65.11 \\ -Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ +Omni $\G+\A+\mathrm{SVA}$ & 82.36 & 76.26 & 70.89 & 66.67 & 63.73 & 61.52 & 60.10 & 58.63 & 57.35 & 56.20 & 65.37 \\ \midrule \multicolumn{12}{@{}l}{\emph{Ego4D}} \\ -EgoSpeak A+V & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ -Omni $\G$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ -Omni $\G+\A$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ -Omni $\G+\A+\mathrm{SVA}$ & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\ +EgoSpeak GRU A+V & 76.4 & 73.0 & 70.4 & 68.5 & 67.1 & 66.3 & 65.6 & 65.2 & 64.7 & 64.4 & 68.2 \\ +Omni $\G$ & 74.72 & 72.03 & 69.81 & 68.11 & 66.81 & 65.88 & 65.21 & 64.73 & 64.35 & 63.98 & 67.56 \\ +Omni $\G+\A$ & 80.78 & 77.03 & 73.88 & 71.49 & 69.66 & 68.31 & 67.33 & 66.63 & 66.05 & 65.55 & 70.67 \\ +Omni $\G+\A+\mathrm{SVA}$ & 81.39 & 77.41 & 74.35 & 71.73 & 69.61 & 68.43 & 67.47 & 66.91 & 66.48 & 65.56 & 70.93 \\ \bottomrule \end{tabular*} \end{table*} \begin{table}[t] -\caption{EasyCom local-vision and information-audit screens (\%). The -upper rows are exploratory encoder-based runs; the lower rows use -non-deployable past activity (three-seed mean $\pm$ SD). Their training -protocols differ, so cross-section differences are not matched effects. -Branch names are defined in Sec.~\ref{sec:method}.} +\caption{EasyCom vision screens and privileged activity audit (\%). +Upper rows are exploratory encoder-feature runs; lower rows use +transcript-derived past activity unavailable at deployment (three-seed +means). Cross-block differences are not matched model effects.} \label{tab:visual} \centering\ninept \renewcommand{\arraystretch}{0.92} @@ -46,45 +43,37 @@ Condition & mAP$_{.2}$ & Mean \\ \midrule $\G+\A$ & 81.72 & 65.11 \\ $\G+\A+$Pool & 82.03 & 65.40 \\ -$\G+\A+$SVA & -- & -- \\ +$\G+\A+$SVA & 82.36 & 65.37 \\ $\G+\A+$TSVA & 82.38 & 65.54 \\ \midrule -Visibility only & $81.95{\pm}.10$ & $65.22{\pm}.05$ \\ -Delayed 1 s & $81.95{\pm}.09$ & $65.34{\pm}.05$ \\ -Visible OR & $\mathbf{86.70{\pm}.09}$ & $\mathbf{66.95{\pm}.02}$ \\ +Visibility only & 81.95 & 65.22 \\ +Delayed 1 s & 81.95 & 65.34 \\ +Visible OR & $\mathbf{86.70}$ & $\mathbf{66.95}$ \\ \bottomrule \end{tabular} \end{table} \begin{table}[t] -\caption{Pending matched timing/context tests. Each cell will compare -$\G$ with $\G+\A$ on common cutoffs; $\Delta$ is paired mean-mAP gain. -PTR uses completed blocks plus the observed tail. Dashes are unmeasured.} +\caption{EasyCom PTR context-cap screen: three-seed mean mAP (\%). +The 8-s bank predates joint extraction; no matched $\G$ arm exists.} \label{tab:stream_context} \centering\ninept \renewcommand{\arraystretch}{0.92} \setlength{\tabcolsep}{3pt} -\begin{tabular}{@{}llcc@{}} +\begin{tabular}{@{}cc@{}} \toprule -Group & Refresh / cap & $\G/\G+\A$ & $\Delta$ \\ +PTR context cap (s) & $\G+\A$ mean mAP \\ \midrule -T & Completed hold, 8 s & --/-- & -- \\ -T & PTR, 8 s & --/-- & -- \\ -\midrule -C & PTR, 2 s & --/-- & -- \\ -C & PTR, 4 s & --/-- & -- \\ -C & PTR, 6 s & --/-- & -- \\ -C & PTR, 8 s & --/-- & -- \\ -C & PTR, 10 s & --/-- & -- \\ -C & PTR, 12 s & --/-- & -- \\ +4 & 64.81 \\ +8 & 65.18 \\ +12 & 64.74 \\ \bottomrule \end{tabular} \end{table} \begin{table}[t] -\caption{Separate seed-0 global-only chunk screen (\%): 10 epochs, 8-s cap, -2-FPS video. PTR is not extraction-matched to the other rows. Calls are -processor invocations across 12 sessions, not latency.} +\caption{Separate seed-0 global-only chunk screen (\%). PTR is not +extraction-matched; calls are processor invocations, not latency.} \label{tab:chunk} \centering\ninept \renewcommand{\arraystretch}{0.95}