From 9fc4577aea0a4c414e3b4168a3450f4dcec621aa Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Sun, 6 Sep 2026 21:50:15 +0800 Subject: [PATCH] docs: refine person-aware manuscript method --- literature_notes/33_muvap.md | 8 +- manuscript/main/.gitignore | 14 ++++ manuscript/main/main.tex | 131 ++++++++++++++++++++++----------- manuscript/main/references.bib | 16 ++++ 4 files changed, 127 insertions(+), 42 deletions(-) diff --git a/literature_notes/33_muvap.md b/literature_notes/33_muvap.md index ee608b3..93a5ee3 100644 --- a/literature_notes/33_muvap.md +++ b/literature_notes/33_muvap.md @@ -10,6 +10,12 @@ 将单声道混合音频与逐人脸轨迹结合，先学习声学轮次和因果 ASD 表示，再融合为全局及逐人未来活动预测。用相对发言角色控制多人联合状态空间；任务包含 shift/hold 和 next speaker。是我们逐人时序旁路最直接的相关工作。 +## 可复用的结构细节 + +MuVAP 不是对多人脸做普通平均。它先用 CPC 与 causal Transformer 得到全局音频 VAP 表示；每条脸轨再经修改后的 causal TalkNet 得到 ASD 表示。两路分别投影到 256 维后，全局音频作为 query、同一时刻的可见人物作为 key/value 做 frame cross-attention，随后用 temporal Transformer 建模群体动态。全局表示再通过 gated addition 调制每个人物表示，并分别输出 GlobalVAP 与 SpeakerVAP。ASD 预训练还包含当前逐人活动辅助损失和音视频对比损失。 + +对本项目最有价值的迁移是“全局 floor query + 匿名人物集合 + 人内因果历史 + 门控残差”，而不是它的 Role-Relative Projection。我们的输出主体始终是不可见 wearer 的 entry/yield，不需要把动态人数展开为 next-visible-speaker 分类。实际批处理可用 padding mask，零人场景加入 NULL/OFFSCREEN token，临时 track ID 只路由共享时序状态而不作为 embedding。 + 不能把“单声道＋逐人视觉＋因果接话”作为独有贡献。我们的区别需要落在不可见 wearer、讲话对象及受召回约束的误开口上，而不是只换编码器。 -作者承认角色识别存在延迟；验证主要为第三人称双人/三人英语场景。数据预处理的全片身份锚点也需与严格在线 tracking 分开讨论。仓库当前只有 README/LICENSE，写明 code coming soon；不要称已有可直接运行的复现。 +作者承认角色识别存在延迟；验证主要为第三人称双人/三人英语场景。其数据准备使用 RetinaFace/SCRFD 检测、ArcFace 特征、匈牙利匹配，并从每段完整视频随机抽取 50 帧建立全局身份锚点。这适合构建稳定标注，但会使用未来/全片信息，不能作为本项目部署时的严格在线 tracking。论文模型本身宣称 causal，不代表这套数据身份初始化也是可在线运行的。仓库当前只有 README/LICENSE，写明 code coming soon；不要称已有可直接运行的复现。 diff --git a/manuscript/main/.gitignore b/manuscript/main/.gitignore index 84c048a..7a7f09c 100644 --- a/manuscript/main/.gitignore +++ b/manuscript/main/.gitignore @@ -1 +1,15 @@ /build/ + +# LaTeX intermediates accidentally emitted beside main.tex when latexmk/pdflatex +# is invoked without the Makefile's build directory. +/*.aux +/*.bbl +/*.bcf +/*.blg +/*.fdb_latexmk +/*.fls +/*.log +/*.out +/*.run.xml +/*.synctex.gz +/*.toc diff --git a/manuscript/main/main.tex b/manuscript/main/main.tex index c793d9c..2be889e 100644 --- a/manuscript/main/main.tex +++ b/manuscript/main/main.tex @@ -109,14 +109,16 @@ Related work supplies complementary parts of the problem. Egocentric talking-to-me detection predicts whether a completed utterance addresses the camera wearer and benefits from long-term social context \cite{kong2024sicnet}. Multiparty VAP grounds future activity in visible face -tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}. -The former is generally utterance-level rather than continuous anticipation; -the latter predicts visible participants rather than the action of an -egocentric conversational agent represented by a wearer-derived target. Our -gap is therefore narrower than ``audio--visual multiparty turn taking'': we -study whether omni representations support direct, continuous agent -floor-action prediction and whether causal local evidence helps when the -visible participant set changes. +tracks to predict floor shifts and next visible speakers \cite{qi2026muvap}, +while AV-CONV reconstructs current person-level speaking/listening relations +from egocentric audio--video \cite{jia2024avconv}. The former talking-to-me +task is generally utterance-level rather than continuous anticipation; +MuVAP predicts visible participants from a static third-person view; and +AV-CONV estimates a current interaction graph rather than strictly causal +future wearer action. Our gap is therefore narrower than ``audio--visual +multiparty turn taking'': we study whether omni representations support +direct, continuous agent floor-action prediction and whether causal local +evidence helps when the visible participant set changes. We call this task \emph{egocentric floor-action projection}. The study asks four questions: (RQ1) how accurately a frozen global omni state predicts the @@ -142,10 +144,11 @@ final method only if it beats pooled, no-ID, and capacity-matched controls. \centering \fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering \todo{After model selection, draw the final system figure. For now it must show -the shared experiment interface: raw causal AV, backbone-specific global -readout, frozen/PEFT/local-adapter branches, optional recent audio and anonymous -person histories, and the common $10\times3$ head. Mark causal cutoffs, -trainable modules, parameter budgets, and source-boundary resets.}}} +the shared experiment interface: raw causal AV; online face/head detection and +causal tracking; backbone-specific global readout; recent audio and a masked +set of anonymous track states; the gated local adapter; and the common +$10\times3$ head. Mark causal cutoffs, trainable modules, parameter budgets, +NULL/OFFSCREEN handling, and source-boundary resets.}}} \caption{Planned omni-model study for egocentric conversational floor action. The local adapter remains a candidate family until the controlled train-side comparison is complete.} @@ -209,27 +212,56 @@ protocol is locked.} The current hypothesis is that global omni states preserve broad semantic context but may make recent prosody, activity boundaries, and individual visual motion difficult for a shallow floor head to recover. From the most -recent $L_a$ seconds, a local encoder produces audio evidence $\vect A_t$ and, -optionally, an anonymous visible-person set $\mathcal P_t$. A generic candidate -adapter is +recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At +time $t$, an online detector and tracker produce a variable-size anonymous set +$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$, +where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and +$\vect r_{t,i}$ contains only causal reliability variables such as visibility, +track age, and time since observation. A shared causal track encoder updates \begin{equation} - \vect z^{(m)}_t=\vect g^{(m)}_t+ - \gamma_t H_\phi(\vect g^{(m)}_t,\vect A_t,\mathcal P_t), - \qquad \gamma_t\vert_{\mathrm{init}}=0, - \label{eq:adapter} + \vect h_{t,i}=F_{\mathrm{trk}}\!\left(\vect h^-_{t,i}, + W_v\vect v_{t,i}+W_b\phi(\vect b_{t,i})+W_r\vect r_{t,i}\right). + \label{eq:track} \end{equation} -followed by the common prediction head. Local evidence is derived from the -same causal raw media, not close microphones, transcripts, speaker labels, or -future frames. +All tracks share $F_{\mathrm{trk}}$; the temporary ID only retrieves +$\vect h^-_{t,i}$ and is never embedded. The global-audio query then reads the +current person set by masked cross-attention, +\begin{align} + \vect u_t &= \mathrm{MHA}\!\left( + W_q[\vect g^{(m)}_t;\vect a_t], + [\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}] + \right), \\ + \vect z^{(m)}_t &= \vect g^{(m)}_t+\tanh(\alpha)W_o\vect u_t, + \qquad \alpha\vert_{\mathrm{init}}=0, + \label{eq:adapter} +\end{align} +where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is +masked, so the same network handles $N_t=0$ and changing group size. For a +bounded implementation, $N_t>N_{\max}$ is resolved by a deterministic ranking +using detector confidence and track age available at $t$, and truncation is +reported. The operator is permutation equivariant +\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not +identity semantics. + +This decomposition is informed by MuVAP: its global audio representation +queries same-frame causal ASD track representations, followed by temporal +modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the +useful separation between global floor dynamics and person-grounded evidence, +but not its role-relative output: our fixed target is the invisible wearer's +entry/yield action, not the next visible speaker. The zero-initialized gate +also follows the conservative adapter principle used to inject visual evidence +into strong audio models \cite{rouditchenko2024whisperflamingo}. Local evidence +is derived from the same causal raw media, not close microphones, transcripts, +speaker labels, global identity anchors, or future frames. We evaluate candidates in increasing structural complexity: pooled recent audio/vision; a capacity-matched temporal branch; a per-frame person set with no cross-time identity; a shared track-wise temporal encoder; and person--audio -cross-attention. Only a candidate that improves the pre-registered event metric -over simpler and capacity-matched controls becomes the final method. If -person continuity is selected, temporary IDs route hidden state only---there -is no ID embedding, name recognition, or cross-recording identity---and the set -operator is permutation equivariant \cite{lee2019settransformer}. +cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a +candidate that improves the pre-registered event metric over simpler and +capacity-matched controls becomes the final method. A limited latent +bottleneck is tested only if large visible sets make full person attention a +runtime bottleneck; it is not assumed to improve accuracy. \todo{Run the candidate ladder before fixing $H_\phi$. Match trainable parameters and effective local history. Compare a shared local encoder across @@ -263,19 +295,34 @@ fold; do not inspect Sessions 1--3 during selection.} \subsection{Track source and deployment contract} -Person-aware candidates first use dataset-provided boxes and anonymous IDs to -isolate the value of local visual structure. A deployable version instead -obtains causal tracklets from a face/head detector followed by an online -multi-object tracker; no future-frame smoothing or cross-session recognition -is permitted. EasyCom IDs are continued only across genuinely adjacent source -segments and reset across redacted gaps. Ego4D IDs are local to a clip. +The intended inference pipeline runs a face/head detector on each incoming +frame and associates detections with an online multi-object tracker using only +observations at or before $t$. Its temporary IDs route the shared recurrent +state in Eq.~\eqref{eq:track}; they do not denote names, fixed seats, or +cross-session identities. New tracks initialize from a learned state, short +misses retain a causally aged state, and expired tracks are removed. With no +visible face, the global/audio path and NULL/OFFSCREEN token remain valid. No +future-frame smoothing, full-video identity anchor, or cross-session +recognition is permitted. EasyCom state is continued only across genuinely +adjacent source segments and reset across redacted gaps; Ego4D state is local +to a clip. -\todo{If a person-aware candidate is selected, state precisely whether face or -head boxes are used. Report provided-track -coverage, mean/maximum people, occlusion gaps, track fragments, and source-gap -resets for both datasets. Add either one online detector/tracker experiment or -controlled box jitter, dropout, fragmentation, and ID-switch curves; otherwise -limit every deployment claim to ``given causal participant tracklets.''} +During method isolation, dataset-provided boxes and anonymous IDs form a +separate \emph{provided-track} condition. This answers whether person +structure can help under accurate localization, but is not reported as online +detector performance. The deployable \emph{online-track} condition runs the +detector/tracker above and includes its errors and latency. This separation is +important because MuVAP used random full-video identity anchors to build its +dataset tracks; that preprocessing is useful for annotation but is not a +strictly online association algorithm \cite{qi2026muvap}. + +\todo{Select and freeze the face/head detector, causal tracker, thresholds, +maximum age, and $N_{\max}$ on training-side data. Report detector recall, +provided/online-track coverage, mean/maximum people, truncations, ID switches, +fragments, occlusion gaps, source resets, and P50/P95 detector--tracker latency +for both datasets. Compare provided tracks, online tracks, and controlled box +jitter/dropout/fragmentation. Until the online row is run, limit deployment +claims to the intended pipeline and label all numbers ``provided-track.''} \section{Experiments} \label{sec:setup} @@ -339,7 +386,8 @@ This compact matrix tests the representation hypothesis before investing in a complex fusion module. If local vision survives three seeds and the event gate, we then compare a parameter-matched temporal branch without person continuity, a separately trained per-frame/no-ID set branch, and a -track-consistent branch. On a frozen track-aware checkpoint, a clip-consistent +track-consistent masked cross-attention branch. Provided-track and online-track +inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent global renaming must leave outputs unchanged, whereas \emph{broken continuity} periodically remaps IDs while preserving every crop, timestamp, audio sample, and target. Time-shifted and removed vision provide additional @@ -365,7 +413,8 @@ decision latency, throughput, peak memory, and trainable parameter count. \todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions, $L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter -count, hardware, and real-time latency after the architecture is frozen.} +count, tracker timeout/ranking, hardware, and end-to-end real-time latency +after the architecture is frozen.} \section{Results and Analysis} \label{sec:results} diff --git a/manuscript/main/references.bib b/manuscript/main/references.bib index 06483e2..c08b3c5 100644 --- a/manuscript/main/references.bib +++ b/manuscript/main/references.bib @@ -63,6 +63,22 @@ year = {2019} } +@inproceedings{rouditchenko2024whisperflamingo, + author = {Andrew Rouditchenko and Yuan Gong and Samuel Thomas and Leonid Karlinsky and Hilde Kuehne and Rogerio Feris and James Glass}, + title = {{Whisper-Flamingo}: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation}, + booktitle = {Proc. Interspeech}, + pages = {2420--2424}, + year = {2024} +} + +@inproceedings{jia2024avconv, + author = {Wenqi Jia and Miao Liu and Hao Jiang and Ishwarya Ananthabhotla and James M. Rehg and Vamsi Krishna Ithapu and Ruohan Gao}, + title = {The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective}, + booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + pages = {26396--26405}, + year = {2024} +} + @inproceedings{grauman2022ego4d, author = {Kristen Grauman and Andrew Westbury and Eugene Byrne and others}, title = {{Ego4D}: Around the World in 3,000 Hours of Egocentric Video},