137 lines
4.7 KiB
Markdown
137 lines
4.7 KiB
Markdown
# Ego4D AV LLM-assisted floor-event annotation
|
|
|
|
This workflow mirrors the EasyCom boundary-review pipeline while using the
|
|
official Ego4D AV annotation schema. It creates semantic review data for
|
|
wearer floor entry and interruption-driven yield; it does not modify Ego4D's
|
|
voice activity annotations and does not turn LLM output into ground truth.
|
|
|
|
## Data and claim boundary
|
|
|
|
Expected dataset layout:
|
|
|
|
```text
|
|
DATA_ROOT/
|
|
v1/
|
|
annotations/av_train.json
|
|
annotations/av_val.json
|
|
clips/<clip_uid>.mp4
|
|
```
|
|
|
|
The generator reads clip-local `voice_segments`, `transcriptions`, and
|
|
`social_segments_talking.is_at_me`. The social label is annotation-only
|
|
evidence. It must never be passed to a forecasting model or used after a
|
|
strictly causal prediction cutoff. Ego4D's voice annotation guideline joins
|
|
sounds separated by less than one second, so a stored segment boundary is an
|
|
activity boundary, not automatically a semantic turn boundary.
|
|
|
|
The output split remains the split declared by Ego4D (`train` or `val`). Do not
|
|
rename validation data to test or tune annotation rules after inspecting the
|
|
final evaluation set.
|
|
|
|
## 1. Generate deterministic candidates
|
|
|
|
Start with one clip and inspect the JSONL before running the full set:
|
|
|
|
```bash
|
|
python3 prepare_ego4d_av_eot_int_review.py \
|
|
--annotations "$DATA_ROOT/v1/annotations/av_train.json" \
|
|
--output reports/ego4d_av_event_candidates/ego4d_av_train_candidates.jsonl \
|
|
--limit 1
|
|
```
|
|
|
|
Then generate the selected splits:
|
|
|
|
```bash
|
|
python3 prepare_ego4d_av_eot_int_review.py \
|
|
--annotations \
|
|
"$DATA_ROOT/v1/annotations/av_train.json" \
|
|
"$DATA_ROOT/v1/annotations/av_val.json" \
|
|
--output reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl
|
|
```
|
|
|
|
The command also writes `<output_stem>_timelines.json` beside the JSONL. Each
|
|
raw wearer onset and offset is retained. Same-other pauses and other-to-other
|
|
handoffs are added as independently reviewable hard-negative proposals.
|
|
Timing rules only select and describe candidates; they never assign semantic
|
|
positive labels.
|
|
|
|
## 2. Inspect what will be uploaded
|
|
|
|
The LLM client refuses to upload transcript excerpts without an explicit
|
|
opt-in. First inspect the exact prompt locally:
|
|
|
|
```bash
|
|
python3 adjudicate_ego4d_av_eot_int_candidates.py \
|
|
--input reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl \
|
|
--output reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
|
|
--max-events 8 \
|
|
--dry-run
|
|
```
|
|
|
|
The prompt uses only timestamps, participant roles, provided transcripts, and
|
|
provided `is_at_me` facts. It does not upload video or audio.
|
|
|
|
## 3. Run LLM pre-labeling
|
|
|
|
After confirming that external transcript upload is authorized:
|
|
|
|
```bash
|
|
export DEEPSEEK_API_KEY=...
|
|
python3 adjudicate_ego4d_av_eot_int_candidates.py \
|
|
--input reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl \
|
|
--output reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
|
|
--passes 2 \
|
|
--batch-size 8 \
|
|
--allow-external-transcript-upload
|
|
```
|
|
|
|
The client checkpoints after every successful batch. Two passes must agree;
|
|
disagreement becomes `EXCLUDE/INSUFFICIENT_EVIDENCE`. Malformed singleton
|
|
responses are written to a pending-human JSONL rather than silently dropped.
|
|
Every successful LLM decision has `needs_human_review=true`.
|
|
|
|
## 4. Review in the browser
|
|
|
|
```bash
|
|
python3 review_ego4d_av_events.py \
|
|
--events reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
|
|
--dataset-root "$DATA_ROOT" \
|
|
--host 127.0.0.1 \
|
|
--port 8765
|
|
```
|
|
|
|
Open `http://127.0.0.1:8765`. The UI shows the local video, clip-level speaker
|
|
timeline, participant-aligned transcripts, LLM rationale, and the closed human
|
|
label taxonomy. Reviews are atomically saved to
|
|
`reports/ego4d_av_event_reviews/reviews.jsonl` with a CSV companion and can be
|
|
resumed. The server rejects media paths outside `DATA_ROOT` and supports byte
|
|
ranges for browser seeking.
|
|
|
|
For an SSH host, keep the service on loopback and forward the port:
|
|
|
|
```bash
|
|
ssh -L 8765:127.0.0.1:8765 USER@HOST
|
|
```
|
|
|
|
## 5. Freeze and audit
|
|
|
|
Before using these annotations in a paper or evaluator:
|
|
|
|
1. Freeze candidate windows and the closed taxonomy on training data.
|
|
2. Double-review a random sample plus every low-confidence/disagreement item.
|
|
3. Report raw candidates, positives, hard negatives, exclusions, and exclusion
|
|
reasons separately for train and validation.
|
|
4. Print and independently inspect example timelines for entry, yield,
|
|
backchannel, same-speaker pause, and other-to-other cases.
|
|
5. Keep the final label named human-adjudicated semantic annotation. LLM output
|
|
alone is never ground truth.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
python3 -m unittest -v \
|
|
test_prepare_ego4d_av_eot_int_review.py \
|
|
test_review_floor_events.py \
|
|
test_adjudicate_easycom_eot_int_candidates.py
|
|
```
|