Add support for NVDEC backend in video processing and enhance README documentation. Introduce a new script for extracting EasyCom sessions on specific GPUs, detailing session management and error handling. Update requirements.txt to include PyNvVideoCodec as an optional dependency. Refactor streaming_turn_model_directly.py to accommodate multiple decoder backends and improve frame indexing with ffprobe.

This commit is contained in:
Xin Wang
2026-08-18 08:41:54 +08:00
parent e4fb62ba60
commit 24cb92f1f4
5 changed files with 304 additions and 22 deletions
+29
View File
@@ -651,6 +651,35 @@ python streaming_turn_model_directly.py \
--output features/direct/Session_1.pt
```
The direct reader defaults to Decord. On NVIDIA systems, install
`PyNvVideoCodec` and select the NVDEC backend explicitly:
```bash
python -m pip install PyNvVideoCodec
CUDA_VISIBLE_DEVICES=0 python streaming_turn_model_directly.py \
--model-id /path/to/Qwen2.5-Omni-3B \
--easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
--end 24 \
--mode open-stream-replay \
--memory-seconds 20 \
--video-fps 2 \
--frame-cache-size 64 \
--decoder pynvcodec \
--decoder-gpu-id 0 \
--flash-attention-2 \
--device cuda:0 \
--output features/direct/Session_1_24s_pynvcodec.pt
```
The NVDEC backend uses ffprobe for exact display-order PTS indexing, one
reconfigurable `SimpleDecoder` for pixel decode, and the existing ffmpeg PCM
audio path. `--decoder-gpu-id` is relative to `CUDA_VISIBLE_DEVICES`, so it is
normally `0` even when the worker is assigned a nonzero host GPU. Decoder RGB
conversion can differ slightly; validate a short Decord/PyNvVideoCodec pair
before extracting every session, and do not mix decoder backends across
train/test splits.
Matching `states.shape[0]` to the target is necessary but not sufficient.
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
monotonicity, dense duration, crop bounds, audio sample count, and causal