Add support for NVDEC backend in video processing and enhance README documentation. Introduce a new script for extracting EasyCom sessions on specific GPUs, detailing session management and error handling. Update requirements.txt to include PyNvVideoCodec as an optional dependency. Refactor streaming_turn_model_directly.py to accommodate multiple decoder backends and improve frame indexing with ffprobe.
This commit is contained in:
@@ -651,6 +651,35 @@ python streaming_turn_model_directly.py \
|
||||
--output features/direct/Session_1.pt
|
||||
```
|
||||
|
||||
The direct reader defaults to Decord. On NVIDIA systems, install
|
||||
`PyNvVideoCodec` and select the NVDEC backend explicitly:
|
||||
|
||||
```bash
|
||||
python -m pip install PyNvVideoCodec
|
||||
|
||||
CUDA_VISIBLE_DEVICES=0 python streaming_turn_model_directly.py \
|
||||
--model-id /path/to/Qwen2.5-Omni-3B \
|
||||
--easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
|
||||
--end 24 \
|
||||
--mode open-stream-replay \
|
||||
--memory-seconds 20 \
|
||||
--video-fps 2 \
|
||||
--frame-cache-size 64 \
|
||||
--decoder pynvcodec \
|
||||
--decoder-gpu-id 0 \
|
||||
--flash-attention-2 \
|
||||
--device cuda:0 \
|
||||
--output features/direct/Session_1_24s_pynvcodec.pt
|
||||
```
|
||||
|
||||
The NVDEC backend uses ffprobe for exact display-order PTS indexing, one
|
||||
reconfigurable `SimpleDecoder` for pixel decode, and the existing ffmpeg PCM
|
||||
audio path. `--decoder-gpu-id` is relative to `CUDA_VISIBLE_DEVICES`, so it is
|
||||
normally `0` even when the worker is assigned a nonzero host GPU. Decoder RGB
|
||||
conversion can differ slightly; validate a short Decord/PyNvVideoCodec pair
|
||||
before extracting every session, and do not mix decoder backends across
|
||||
train/test splits.
|
||||
|
||||
Matching `states.shape[0]` to the target is necessary but not sufficient.
|
||||
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
|
||||
monotonicity, dense duration, crop bounds, audio sample count, and causal
|
||||
|
||||
Reference in New Issue
Block a user