Does AI learn from videos?

The Hidden Scale of Video Training Data

Every time you watch an AI-generated video, edit with an AI tool, or ride in a car that reads the road, you're touching a model that learned from video. The evidence is unambiguous: yes, AI learns from videos — and at a scale most professionals never see.

The numbers back this up. According to market research from Grand View Research, image and video data led the entire AI training dataset market in 2025 with a 41.9% revenue share, outpacing text, audio, and every other data type. A competing estimate from Business Research Insights places image/video at roughly 35% of dataset demand — still an enormous slice of a market projected to grow at over 22% annually.

Then there's the investigative record. An investigation by The Atlantic, reported by Nieman Lab, found at least 15 million YouTube videos used as AI training data by Microsoft, Meta, Snap, Tencent, Runway, and ByteDance. The sourcing wasn't random — Runway deliberately curated "high quality" channels, tagged with labels like "newscaster," "tech reviews," and "scrambling eggs," to teach its Gen-3 model specific visual styles. Shared datasets like Microsoft's HD-VILA-100M circulated across multiple companies, meaning the same videos trained competing models.

Here's what that means in practice for the tools you use daily:

• Video generation platforms like Sora 2 and Veo 3.1 are trained on billions of text–video pairings scraped from the internet.
• Video-derived text — subtitles — has trained large language models from Anthropic, Apple, Nvidia, and Salesforce, per Proof News.
• Over 85% of autonomous vehicle projects use image and video datasets as their primary training source.

The problem is that almost nobody using these tools knows where the training data came from. YouTube's own terms prohibit third parties from downloading videos for training, yet investigations show it happened at massive scale anyway. No licensing deals exist between Runway and the news publishers whose footage it ingested. And models cannot unlearn — more than 12,000 videos deleted from YouTube remain embedded in training datasets.

This is a provenance problem, and it doesn't stop at training data. MIT's Data Provenance Initiative audited 1,800+ datasets and found license error rates above 50%, with plans to expand the work to video. The same trust gap exists in the AI agent ecosystem, where published research found roughly 26% of agent skills carry at least one vulnerability — which is exactly why SkillzBench is building a free directory that security-scans, license-checks, and benchmarks every skill before you use it.

For business owners and builders, the takeaway is simple: video-trained AI is already in your workflow. Understanding its data origins — and verifying the AI components you adopt — is now part of due diligence, not a technical afterthought.

How AI Actually Learns from Video: The Mechanics Explained Simply

When you watch a video, your brain tracks motion, sound, and story all at once. Teaching a machine to do the same takes billions of examples and a clever bit of digital dice-work.

According to MIT Technology Review's explainer, most text-to-video models learn from billions of text–video pairings scraped from the internet. Each clip comes with a caption or description, so the model learns to connect words like "waves crashing at sunset" with the actual moving pixels.

Here's where it gets interesting. Before training, videos are diced into chunks spanning both space and time. Tim Brooks, lead researcher on OpenAI's Sora, describes it vividly: "It's like if you were to have a stack of all the video frames and you cut little cubes from it." Because size and orientation no longer matter, models can learn from everything from vertical phone clips to widescreen films.

The learning itself happens through a few cooperating components:

• Diffusion models train on millions of images in various stages of pixelation, gradually learning to reverse noise into clear visuals.
• Transformers keep frames consistent with one another, so a character's face doesn't morph mid-scene.
• A language model steers generation toward whatever the prompt actually asked for.

Google DeepMind pushed this further with Veo 3, compressing audio and video into a single piece of data so both generate in lockstep — a step toward models that "hear" as well as see.

But video teaches AI in more than one way. As Proof News reported, the "YouTube Subtitles" dataset — transcripts from 173,536 videos totaling 489 million words — was used to train LLMs at Anthropic, Apple, Nvidia, and Salesforce. AI policy researcher Jai Vipra calls this speech-to-text data a "gold mine" for teaching models how people actually talk and converse.

So video trains AI twice over: visually, through frames and motion, and verbally, through the words spoken in them. That dual channel is one reason image/video data commanded a 41.9% revenue share of the AI training dataset market in 2025.

The parallel to agent skills is worth noting: just as video models are only as good as the data they ingest, AI agents are only as good as the skill packages they load — which is why SkillzBench is building a free directory that security-scans and benchmarks every skill before you trust it with a workflow.

Where Video-Trained AI Shows Up in Business

Video-trained AI has quietly moved from research labs into products your business touches every week — from the car that brakes for a pedestrian to the training video your new hire watches on day one.

Autonomous vehicles are the flagship example. More than 85% of AV development projects use image and video datasets as their primary training source, and camera data accounts for over 55% of automotive AI inputs — far ahead of LiDAR at roughly 12%, according to industry research. A single test vehicle can generate more than 4 TB of data per day, all of it feeding models that must detect road signs, pedestrians, and obstacles in real time.

Entertainment is next. Netflix debuted an AI visual effect in The Eternaut — the first use of video generation in mass-market TV, using Runway's tools — and Disney has tested the same software, as MIT Technology Review reports. Meanwhile, three video-generation platforms now run in parallel: Gemini Omni Flash (free on YouTube Shorts), Sora 2 (API-only), and Veo 3.1 (enterprise via Vertex AI), with pricing spanning $0.03 to $0.50 per second of 1080p output.

The applications readers ask about most:

• Autonomous driving and advanced driver assistance
• Film and TV visual effects, like Netflix's *The Eternaut*
• Workplace training video, where a UCL study found AI video matched human video in learning outcomes — with ~20% faster completion
• Smart surveillance, retail analytics, and motion tracking

That workplace learning finding deserves attention: 77% of learners prefer video over text, and AI-generated video now delivers equivalent outcomes at lower cost, per the UCL research. For L&D teams, that changes the economics of onboarding content overnight.

Here's the catch: all of this rests on expensive data work. Video annotation can demand up to ~300% more labeling than static images, and over 80% of AI project time goes to data collection, labeling, and preparation. That's why the image/video segment commands a 41.9% revenue share of the AI training dataset market — businesses are paying heavily to teach machines to see.

If you're building AI workflows around video or other agent-driven tasks, the data-prep burden doesn't stop at datasets. Agent skills — the instruction packages powering Claude Code, Cursor, and similar tools — carry their own verification problem, with published research finding roughly 26% contain at least one vulnerability. SkillzBench is building a free answer: a directory and MCP connector that security-scans, benchmarks, and version-pins every skill it lists. Join the waitlist to see verified, benchmarked skills first.

The Trust Problem: Consent, Provenance, and What Can't Be Unlearned

Once a model has watched 15 million YouTube videos, there is no delete key. That single fact sits at the center of the video-training debate: the data that makes AI video models powerful is also the data many creators never agreed to share.

The consent concerns are well documented. An investigation by The Atlantic, reported by Nieman Lab, found that Runway deliberately selected thousands of "high quality" channels — tagged with labels like "lots of newscaster but plenty of b-roll" and "scrambling eggs" — to teach its Gen-3 model specific visual vernaculars. The same reporting found no licensing deals between Runway and the news publishers whose videos appeared in the datasets, including over 88,000 Fox News videos, roughly 70,000 from ABC News, and 11,604 from The New York Times.

The permanence problem compounds the consent problem. According to Proof News, more than 12,000 videos deleted from YouTube remain ingested in the YouTube Subtitles dataset used by Anthropic, Apple, Nvidia, and Salesforce — because AI models cannot "unlearn" training data once it has shaped their weights.

Yet the quality argument cuts the other way. Curated, deliberately selected data is precisely what makes these models useful, and Vox Media's spokesperson captured the trade-off bluntly: "Without our quality content, the reality for these platforms will be: garbage in, garbage out."

The deeper issue is that nobody can reliably verify where training data comes from. MIT's Data Provenance Initiative audited more than 1,800 text datasets and found license miscategorization error rates above 50%, with license omission rates above 70%. The team plans to expand this provenance work to video — a gap that matters for anyone building with AI:

• Licenses attached to datasets are wrong or missing more often than not
• Deleted or withdrawn content persists inside trained models
• No independent benchmark exists for comparing video-generation models

This provenance gap is not unique to video. Agent skills — the instruction packages powering Claude Code, Cursor, and other coding agents — face the same trust problem: published research found roughly 26% carry at least one vulnerability, and most arrive with no license verification or independent testing. SkillzBench is being built to close that gap for skills, with commit-pinned versions, license checks, security scans, and sandboxed benchmarking, all free.

For creators, the lesson is that consent and quality are not opposites — they are both prerequisites the industry has yet to reconcile.

What This Means for You: Applying Video-Trained AI with Verified Tools

Knowing that AI learns from videos is only useful if you act on it. Whether you run a business, write code, or build agents, the same practical playbook applies: pick video tools deliberately, ask hard questions about where their training data came from, and bring verification discipline to every AI building block you rely on.

Start with tool selection. As of May 2026, three video-generation platforms operate in parallel — Gemini Omni Flash (free for consumers on YouTube Shorts), Sora 2 (API-only), and Veo 3.1 (enterprise via Vertex AI) — with per-second pricing from third-party aggregators ranging from Veo 3.1 Lite at $0.03/sec to Sora 2 Pro at $0.50/sec in 1080p. Match the tool to the job rather than the hype: a social clip and a broadcast-quality effect have very different budgets. Netflix's AI visual effect in The Eternaut shows production-grade output is already here, but note that no independent third-party benchmark comparing these models exists yet — vendor claims like "a GPT-3.5 moment for video" are exactly that, vendor claims.

Second, demand transparency about training data. The MIT-led Data Provenance Initiative audited 1,800+ datasets and found license miscategorization error rates above 50% and license omission above 70% — and provenance work is only now expanding to video. Investigative reporting found at least 15 million YouTube videos used in AI training, with no licensing deals between Runway and affected publishers. Before committing to a video tool, ask:

• What sources were used to train the model, and were they licensed or scraped?
• Can the vendor document its data pipeline the way Runway's curated "high quality" channels were tagged?
• What happens if a rights holder objects — remember, models cannot unlearn data once ingested?
• Is there independent benchmark evidence, or only the vendor's own marketing?

Third, apply this same verification discipline to your daily AI stack. With 80%+ of AI project time spent on data preparation and 95%+ annotation consistency typically required before training, thorough professionals already know that unverified inputs produce unreliable outputs. Your agent skills deserve the same scrutiny as a training dataset.

That's the tie-in: agent skills face the same trust problem as training data. Published research found roughly 26% of skills carry at least one vulnerability, yet nothing tells you which ones actually help. That's why SkillzBench is building a free directory of AI agent skills with security scanning, sandbox benchmarking that publishes each skill's Lift separately, and version pinning to a commit SHA — plus a free MCP tool to connect your chosen skills to any agent. Everything will be free, and scores will never be for sale.

Join the waitlist to be first in when it launches. Skills, proven.