Most AI video workflows start well before the model call. Before a multimodal model ever sees a frame, something has to decode the container, pick the right moments, and hand over clean assets. That something is usually FFmpeg.

Extracting frames

Pull one frame per second as JPEGs, a common first step for feeding a vision model:

ffmpeg -i input.mp4 -vf fps=1 frames/frame_%04d.jpg

Change fps=1 to sample more or less often. fps=1/5 gives one frame every five seconds. For most tagging and description tasks, one frame every one to two seconds is enough. Going denser mostly adds token cost without adding information.

To cut the frame count without picking a rate by hand, sample the encoder’s keyframes, the self-contained I-frames it writes at regular intervals and at hard cuts:

ffmpeg -i input.mp4 -vf "select='eq(pict_type,I)'" -vsync vfr keyframes/frame_%04d.jpg

You get far fewer images than a dense fps sample, which is cheaper to send. It keys off how the file was encoded rather than the content, so a long static shot still yields periodic near-identical frames.

Sampling on scene changes

That is where scene detection helps. A sharper signal than the encoding is the picture itself. FFmpeg can score how much each frame differs from the one before it and keep only the frames where that score jumps:

ffmpeg -i input.mp4 -vf "select='gt(scene,0.4)'" -vsync vfr scenes/frame_%04d.jpg

The scene score runs from 0 to 1. A threshold near 0.3 to 0.4 catches real cuts and hard transitions and ignores small motion. Lower it when you miss shots; raise it when you get near-duplicates. On a screen recording or a talking-head clip, this turns a thousand frames into a few dozen that actually differ.

Grabbing one frame at a timestamp

Sometimes you want a single still: a poster frame, a thumbnail, one moment to inspect. Seek to the time and take one frame:

ffmpeg -ss 00:01:30 -i input.mp4 -frames:v 1 -q:v 2 thumb.jpg

-frames:v 1 writes exactly one image. -q:v 2 holds the JPEG near its best quality, on a scale that runs from 2 (high) to 31 (low). The -ss sits before -i so the seek is fast, which matters when the timestamp is deep into a long file.

Cutting clips

Cut a segment without re-encoding. This is fast and lossless because it copies the existing streams:

ffmpeg -ss 00:00:30 -to 00:00:45 -i input.mp4 -c copy clip.mp4

The catch is keyframe alignment. Stream copy can only cut on a keyframe, so the clip may start a little before the timestamp you asked for. When you need a frame-exact cut, drop -c copy and let FFmpeg re-encode:

ffmpeg -ss 00:00:30 -to 00:00:45 -i input.mp4 -c:v libx264 -crf 18 clip.mp4

Re-encoding is slower and slightly lossy, but the cut lands exactly where you set it.

Pulling the audio for transcription

Description and search often need the words, not the pixels. Strip the audio to the shape a speech model expects. Whisper and most other speech-to-text models want 16 kHz mono:

ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav

-vn drops the video. -ac 1 folds the audio down to one channel, -ar 16000 resamples to 16 kHz, and pcm_s16le writes plain 16-bit WAV. That is the smallest clean input a transcription model reads without resampling it again on the way in.

Preparing assets for multimodal models

Models see a downscaled image anyway, so send them a downscaled image. Resizing before the API call cuts upload size and cost with no real loss in what the model can read:

ffmpeg -i input.mp4 -vf "fps=1,scale=768:-2" frames/frame_%04d.jpg

scale=768:-2 fixes the width at 768 and keeps the aspect ratio; the -2 rounds the height to an even number. Match the width to your model’s expected input. Larger is not better once you pass what the model actually uses.

A contact sheet for a quick look

To see a whole clip at once, tile sampled frames into a grid:

ffmpeg -i input.mp4 -vf "fps=1/10,scale=320:-1,tile=4x4" sheet_%03d.jpg

One frame every ten seconds, scaled down, laid out sixteen to a sheet. It is a quick way to judge what is in a video before you commit to a sampling rate.

Batch-processing a folder

Most real work runs over a directory, not a single file. A short shell loop handles it:

for f in clips/*.mp4; do
  name=$(basename "$f" .mp4)
  mkdir -p "frames/$name"
  ffmpeg -i "$f" -vf "fps=1,scale=768:-2" "frames/$name/%04d.jpg"
done

For anything larger, wrap the same command in Python with subprocess so you can log failures and retry.

Common mistakes

  • Sampling too densely. One frame per second is usually plenty. Ten frames per second gives the model ten near-identical images and ten times the cost.
  • Forgetting keyframe alignment on -c copy cuts, then wondering why clips start early.
  • Putting -ss after -i. Before the input it seeks quickly. After the input it decodes from the start, which is much slower on long files.
  • Leaving the scene threshold at whatever you first typed. Too low floods you with near-duplicates; too high drops real cuts. Tune it on one representative clip before you run the batch.

Where FFmpeg fits

The model call is the small part. FFmpeg is the layer that turns a raw upload into the frames, clips, and formats the rest of the pipeline expects. Get the extraction and framing right and the model has a clean, cheap input. Get it wrong and no prompt will save you.