AVC-1.8M · An open video–language dataset
1.8 million ten-second video windows, each carrying a stack of temporally aligned text layers — speech transcripts, on-screen text, dense visual descriptions, fused captions, and structured entities — plus a modality-attributed retrieval & QA benchmark that tells you which layer holds the evidence for every query.
Anatomy of a window
Two real windows from the test split. Ten frames, one for each second — and beneath them, the text layers that describe the same ten seconds from different sources. Queries are attributed to the layer that generated them.
Howto & Style — a cooking window where speech, screen text, and visuals all carry signal 1mbyCXQIYxU__s140__e150
Science & Technology — a product demo where the evidence lives in the on-screen text lgwKFOTjbCU__s820__e830
Windows are sampled across categories from English-language YouTube videos in Panda-70M. Alongside the text layers, the dataset ships 193 precomputed embedding matrices (six image encoders × three poolings, audio encoders, and text encoders over 13+ text views and entity search texts) and lexical BM25 indexes — so most experiments never touch the raw media.
Composition
Windows span fifteen source categories, so retrieval systems meet cooking tutorials, news, gaming, and music on equal footing. The benchmark keeps its four evidence modalities in near-equal balance by construction.
Of the 2,195 retrieval queries, 695 are graded hard and 1,500 normal.
Get the data
The full distribution lives in a public Azure Blob Storage container with anonymous read access — no SAS tokens, no sign-up. Machine-readable metadata is served as MLCommons Croissant at the container root.
https://publicavcdist.blob.core.windows.net/avc-1-8m/Confirm the container is reachable and skim the layout docs:
# dataset metadata (Croissant 1.0, with Responsible-AI fields)
curl -s https://publicavcdist.blob.core.windows.net/avc-1-8m/croissant.json | head
# layout + usage docs at the container root
curl -s https://publicavcdist.blob.core.windows.net/avc-1-8m/README.mdGrab individual files with wget/curl, or whole trees with
azcopy:
# the benchmark itself is tiny — start here (~4 MB each)
wget https://publicavcdist.blob.core.windows.net/avc-1-8m/test_question_answers.jsonl
wget https://publicavcdist.blob.core.windows.net/avc-1-8m/test_search_queries.jsonl
# per-window caption records for the test split (~184 MB)
wget https://publicavcdist.blob.core.windows.net/avc-1-8m/dist/captions_and_meta-test.jsonl
# bulk-download a subtree (see the size table before recursing!)
azcopy copy 'https://publicavcdist.blob.core.windows.net/avc-1-8m/dist' . --recursiveimport mlcroissant as mlc
ds = mlc.Dataset("https://publicavcdist.blob.core.windows.net/avc-1-8m/croissant.json")
# record sets: "retrieval_queries", "qa_pairs"
for record in ds.records(record_set="retrieval_queries"):
print(record)
breakThe text side of the dataset (captions, benchmark, QA) is light; embeddings and raw media are the heavy part. Check before you recurse:
| Path | Contents | Weight |
|---|---|---|
croissant.json · README.md |
Machine-readable metadata and layout docs at the container root | Tiny |
test_question_answers.jsonl · test_search_queries.jsonl |
The benchmark: 2,042 QA pairs and 2,195 retrieval queries with the window's full caption context inline — self-contained for judging | Tiny (~4 MB each) |
dist/ |
Per-window caption records for train / val / test (1,631,378 / 94,362 / 88,566 windows) plus generated training pools: ~11.6M QA pairs and ~11.6M retrieval queries | Light–moderate (single-file downloads; test captions ~184 MB) |
evals/ |
The frozen retrieval corpus: 88,566-window db strings in 11 modality combinations, plus precomputed query- and corpus-side embeddings per encoder | Heavy (embeddings dominate) |
closedqa/ |
Per-model closed-QA outputs for reference baselines | Light |
test_vids/ |
4,128 raw test-window MP4s (<vid>__s<start>__e<end>.mp4) |
Heavy |
One real record from each of the main files. The dist/ examples all come from
the cooking window shown in the anatomy section above, so you can trace a single window
from captions to queries to benchmark.
dist/captions_and_meta-{train,val,test}.jsonlone record per window{
"id": "1mbyCXQIYxU__s140__e150",
"split": "test",
"source_url": "http://youtube.com/watch?v=1mbyCXQIYxU",
"source_time_range": "02:20-02:30",
"source_video_category": "Howto & Style",
"fused_caption": "A close-up shows a pot of rice and brown liquid bubbling and steaming on a stovetop burner, with a red spatula stirring the mixture. The person cooking explains that they are bringing the rice to a boil on medium heat to reduce the liquid …",
"visual_caption": "The video shows a close-up of a stainless steel pot with a brown liquid inside. The pot is on a black stovetop burner with a flame underneath …",
"narrative_caption": "The video clip shows a person cooking rice on a stovetop, bringing the rice to a boil and stirring it.",
"activities_caption": "The person is cooking rice by bringing it to a boil on a stovetop and stirring it with a red spatula.",
"background_caption": "The background of the video is a kitchen countertop with a black stovetop burner …",
"speech_transcript": "Now, we're going to bring this to a boil on medium heat just to reduce the liquid. Then we're going to give it a stir,",
"vlm_primary_scene_text": "Bring to a boil on medium heat to reduce liquid",
"inferred_video_category": "Food & Drink",
"inferred_video_subcategory": "Cooking",
"inferred_title": "How to Cook Rice: Reduce Liquid & Stir",
"inferred_5_word_summary": "Bring rice to a boil, stir"
}dist/retrieval_queries-{train,val,test}.jsonl~11.6M generated training queries + val/test pools{
"id": "49d28db5-02cc-5c6f-b2b1-10450679d880",
"window_id": "1mbyCXQIYxU__s140__e150",
"source_url": "http://youtube.com/watch?v=1mbyCXQIYxU",
"source_time_range": "02:20-02:30",
"source_video_category": "Howto & Style",
"split": "test",
"generation_modality_source": "asr",
"text": "how to reduce liquid by boiling on medium heat"
}dist/question_answer_pairs-{train,val,test}.jsonl~12.9M generated QA pairs{
"id": "d9dbc72b-d4f6-5241-a98f-09cdf2fed829",
"window_id": "1mbyCXQIYxU__s140__e150",
"source_url": "http://youtube.com/watch?v=1mbyCXQIYxU",
"source_time_range": "02:20-02:30",
"source_video_category": "Howto & Style",
"split": "test",
"generation_modality_source": "visuals",
"question": "What type of pot is being used to cook the rice?",
"answer": "A stainless steel pot."
}dist/benchmark-retrieval_queries.jsonl2,195 validated benchmark queries (QA twin: benchmark-question_answer_pairs.jsonl){
"id": "9eb8c431-944b-5865-909f-fe07817d41e9",
"window_id": "1mbyCXQIYxU__s140__e150",
"source_url": "http://youtube.com/watch?v=1mbyCXQIYxU",
"source_time_range": "02:20-02:30",
"source_video_category": "Howto & Style",
"generation_modality_source": "asr",
"llm_validation_reason": "The query specifically asks about the reason for stirring while reducing liquid on medium heat, which matches the clip's content showing and explaining this cooking process. It is not a verbatim copy of the text or transcript …",
"llm_context_source_difficulty": "hard",
"text": "why do you stir while reducing liquid on medium heat"
}evals/retrieval/db_strings.jsonlretrieval-corpus text in 11 modality combinations{
"id": "-2fmOkN6KH4__s0__e10",
"text": {
"dvd": "The video clip starts with a black screen with the words \"Everything Wrong With DEATH WISH\" in white text. The text is in a bold, sans-serif font …",
"asr": "What? The only 11 seconds",
"ocr": "Everything Wrong With DEATH WISH In 13 minutes or less Spoilers!! (duh.) 0 00:00:00 Movie Sin Counter Movie Sin Timer WTF? Only 11 seconds of one logo …",
"fuse": "The clip opens with a bold title card reading Everything Wrong With DEATH WISH and playful warnings about spoilers, set against a dramatic sunset …",
"asr+ocr": "…", "dvd+asr": "…", "dvd+ocr": "…", "fuse+asr": "…",
"fuse+ocr": "…", "dvd+asr+ocr": "…", "all": "…"
}
}
// dvd = dense video description. Matching .npy embeddings per encoder × text
// combo live in evals/retrieval/db_embeddings/; query-side embeddings alongside.v2/ prefix, and croissant.json is updated additively. Pin paths
with confidence.
Benchmark & evaluation
Most video-retrieval benchmarks can't tell you why a system failed. AVC-1.8M's benchmark items are modality-attributed: each of the 2,195 retrieval queries and 2,042 QA pairs is labeled with the text layer that generated it — asr, ocr, visuals, or dense — together with a difficulty grade. Scores decompose by evidence source, so “our model is weak on on-screen text” becomes a measurable claim rather than a hunch. All items are LLM-validated and human-reviewed.
evals/retrieval/db_strings.jsonl — 11 modality-combination text views per window) and the query strings.evals/.
Twelve of the 2,195 retrieval queries — three per modality. Click a column header to sort.
The full files are dist/benchmark-retrieval_queries.jsonl and
dist/benchmark-question_answer_pairs.jsonl.
| Query | Modality | Difficulty | Category | Window |
|---|---|---|---|---|
| How many planets are in our solar system? | asr | hard | Education | x8ULq9M-aEk__s50__e60 |
| child winning and talking to mommy about eggs | asr | hard | Entertainment | Ky62ZtebT_Y__s290__e300 |
| YouTube yearbook from 2016 | asr | normal | Entertainment | N6G2_5sMhuM__s150__e160 |
| old guard outdated textbooks science revolution | ocr | hard | Education | P6Cr_jRVQ6c__s200__e210 |
| Airwheel removable 15000mAh lithium battery power bank | ocr | hard | Science & Technology | lgwKFOTjbCU__s820__e830 |
| how to select packages in Cygwin setup | ocr | normal | Education | ZVGumut8EJY__s340__e350 |
| child playing on red and blue bouncy castle indoors | visuals | hard | Music | K5UDgP31D74__s40__e50 |
| man in grey t-shirt standing in park with serious expression | visuals | hard | Entertainment | J6Zd2OT6Nds__s70__e80 |
| two women tense conversation in cluttered office | visuals | normal | Gaming | 72eSGJVUsDQ__s240__e250 |
| speaker holding black object with 'Cata' sign on platform | dense | hard | Nonprofits & Activism | cEhk5qCAAVA__s650__e660 |
| video comparing Mazda Miata appearance to a happy fish | dense | hard | Autos & Vehicles | 7epWd-nLPSs__s250__e260 |
| man presenting on teamwork and collaboration on stage | dense | normal | Nonprofits & Activism | WbGi7KdxYhU__s30__e40 |
Training-scale pools of ~11.6M generated queries and ~11.6M QA pairs (attributed the same
way) ship in dist/ for fine-tuning and ablations.
The evaluation harness and review tooling are being prepared for public release; a code
link will appear here.
License & terms
All annotations, captions, entities, embeddings, and benchmark files are released under CC BY-NC-SA 4.0. Source media remain governed by their original platform terms.
If a source video is removed from its platform, or you are a rights holder or subject of a
video and want it excluded, contact
<REDACTED DURING PAPER REVIEW> with the video id. Derived records for
removed or objected-to source videos are removed from the distribution.
FAQ
No. The container allows anonymous public reads — every URL on this page works from a
plain curl with no tokens, SAS signatures, or registration.
Yes. The train split (1,631,378 windows) ships with generated training pools of ~11.6M retrieval queries and ~11.6M QA pairs, under CC BY-NC-SA 4.0 (non-commercial, share-alike, with attribution).
Embed the fixed corpus strings in evals/retrieval/db_strings.jsonl and the
benchmark queries with your encoder, then rank against the fixed 88,566 window ids and
report R@k / MRR / NDCG per modality split. No media processing is required — the corpus
text views and baseline embeddings are precomputed.
Coverage follows content: ~88% of windows have speech (ASR), ~65% carry detected scene text, and 90,060 have music captions — silent or text-free windows simply have nothing to transcribe. Missing layers are empty fields, not dropped rows, so masking is a simple empty-string check.
Email <REDACTED DURING PAPER REVIEW> with the video id and we'll
remove its derived records from the distribution. See Takedown.
Changelog
Full caption records for 1.8M windows, the modality-attributed benchmark, the 88,566-window
evaluation corpus with precomputed embeddings, closed-QA baseline outputs, and 4,128 raw
test-window videos. Paths under v1 will not change; future additions land under
v2/.