Train an agent with the Video API

What the agent gets from one call

"Training" here does not mean fine-tuning a model. It means giving an agent a tool it did not have, a short rule about when and how to use it, and a test that tells you whether its answers hold up. The tool is read_video. One call with a YouTube URL returns one Markdown timeline: the description, every chapter as a heading, the transcript in the spoken language merged into 20-second blocks stamped [m:ss], and a [frame N · m:ss · why] marker wherever a small picture belongs. Over MCP the pictures arrive inline as images; over REST they are base64 JPEG strings in frames[].

This is what the agent read for the tutorial video in the terminal above, trimmed to the moment that matters for question three:

# Understanding MCP From Scratch
Channel: LangChain · Duration: 12:22
Transcript: en (automatic captions), original language, timestamps [m:ss]
Frames: 12 storyboard frames of 160×90 px, marked [frame N · m:ss · why]

## Timeline
…
[frame 5 · 3:50 · the speaker points at the screen]
[3:50] here's the bridge to mCP what if I want …

## [4:00] Bridge to MCP: Connecting Tools to AI Applications
…

Two details to notice, because the rule below depends on them. The automatic captions wrote "mCP": the response says automatic captions in its header and adds a note that proper names may be spelled as heard, so an agent that reads the header knows to trust the description and the frames for names. And the frame sits before the words said over it, so the model looks, then reads.

The rule that trains it

A tool the model does not know when to use is a tool it will misuse. These six lines go in the place your client keeps standing instructions: CLAUDE.md for Claude Code, the project rules in Cursor, the system prompt in your own loop. They encode what we learned from the three measured reads: use windows on long videos, quote with the stamp, trust the description for names, say when there is no transcript.

## Videos
- When a question is about a YouTube video, call read_video with the URL before answering.
- Videos over 20 minutes: read them in windows with start/end of about 10 minutes,
  or pass max_tokens; the response says when it was trimmed and where to resume.
- Answer with the [m:ss] stamp and the exact words from the transcript, then what the
  frame at that moment shows. Never invent a timestamp.
- The header says "automatic captions" when names may be spelled as heard: take
  names from the description or the chapters, not from the captions.
- If transcript.source is none, say the video has no captions and answer from the
  chapters, the description and the frames only.
- Frames are 160×90: describe scenes and objects, do not read small text from them.

The same text, saved as a SKILL.md, is what the Agent Skills QuanticData publishes already do for the whole MCP server; npx skills add https://quanticdata.io installs them in Claude Code, Cursor, Codex and any agent that reads the format.

Check it on three videos

An agent is trained when you can measure it. On 20 October 2026 we read three public videos with the defaults and picked one question per video whose answer is in the timeline with a timestamp: the stamp of the transcript block, the quote the frame was taken for, or the chapter start. Ask your agent the same questions and compare with the column on the right; the words are the transcript's own.

The check itself is a few lines: every expected answer is a timestamp the agent must mention, within a minute, and a phrase it must quote. Run it after every change to the rule or the model, the way you would run a unit test.

import re

CHECKS = [
    ("https://www.youtube.com/watch?v=qg7uHUHq9zk",
     "What does the robot cost, and when is the price said?", "0:0", "499"),
    ("https://www.youtube.com/watch?v=aircAruvnKk",
     "When does the edge detection example start, and what is on screen?", "8:3", "edge detection"),
    ("https://www.youtube.com/watch?v=CDjjaTALI68",
     "Where is the bridge to MCP introduced?", "3:5", "bridge"),
]

def passes(answer, minute_prefix, phrase):
    stamps = re.findall(r"\b\d{1,2}:\d{2}\b", answer)
    return any(s.startswith(minute_prefix) for s in stamps) and phrase.lower() in answer.lower()

for url, question, minute_prefix, phrase in CHECKS:
    answer = ask(read_video(url), question)      # parts two and three, as a function
    print("PASS" if passes(answer, minute_prefix, phrase) else "FAIL", "|", question)

A failure usually means one of three things, in this order: the agent answered without calling the tool (the rule's first line), it read a window that did not contain the moment (the rule's second line), or it paraphrased instead of quoting (the third). Fix the rule, not the test.

Context and cost

Each read is one price, whatever the length; what you control is how much of it reaches the model. These are the three videos of this tutorial, read with the defaults from outside our network.

• Windows, not whole lectures: start: "20:00", end: "30:00" reads ten minutes, frames included, for the same price as the whole video.
• A budget the response respects: max_tokens: 6000 trims the timeline at a block boundary and sets tokens.truncated; the note says to continue with start and end.
• Frames only when the question needs eyes: max_frames: 0 for summaries and quote hunting, frames: false to keep the list of moments without the image data.
• Agents probe for free: a removed, private or gated video answers a stable code and costs nothing, and $2 of usage is free every month.