Edit Videos with Claude Code and Remotion: Captions, Graphics, and Sound

How It Works

Remotion is a code-based video tool that renders video from React components. Claude Code acts as the editor — you describe what you want in plain English, Claude writes the Remotion code, renders the output, and iterates on your feedback.

Drop in a raw clip, tell Claude what you want, get a finished vertical short in minutes. No timeline dragging, no keyframing.

Requirements: Claude desktop app, Node.js LTS, ffmpeg.

Step 1 — Bootstrap the Framework

Open Claude Code and paste this setup prompt:

promptCopy
Set up a Remotion video project for vertical short-form reels (1080×1920 @ 30fps) with the following:
1. Initialize a Remotion project (npm create video@latest, blank template).
2. Install ffmpeg if it's not already on this machine (brew install ffmpeg on Mac).
3. Clone and build whisper.cpp into a local bin/ folder, and download the base.en model.
4. Create scripts/transcribe.mjs — a Node script that takes an mp4 path, runs whisper.cpp on the audio, and outputs a JSON file of word-level timestamps in the shape { text, startMs, endMs }.
5. Create a public/recordings/ folder where I'll drop raw MP4s.
6. Create src/Captions.tsx — a Remotion component that reads the transcript JSON and displays the active word at one of three Y positions.
7. Create src/Root.tsx with a single Composition that mounts the raw video, overlays gap scenes, and renders captions.

Don't write any actual reels yet — just the framework. Use TypeScript. Print a final summary of every command I need to run, and confirm everything builds.

Step 2 — Install the Remotion Skill

In Claude Code: Customize → Skills → Add Skill (+) → Create Skill → Upload Skill. Download the Remotion skill file (available from the source guide), drag the .md file into the Upload Skill box. The skill activates immediately and gives Claude project-specific knowledge.

Step 3 — Set Up Sound and Music Packs

promptCopy
public/assets/sound pack/   ← short effects (.mp3)
public/assets/music pack/   ← background tracks (.mp3)

Name files clearly (woosh.mp3, ding.mp3, riser.mp3) so Claude can pick the right sound. Free sources: Pixabay Sound Effects, MyInstants.

Three Use Cases

Use Case 1 — Captions

Drop your raw clip into public/recordings/. Tell Claude:

promptCopy
Transcribe this and add TikTok-style captions, 2 to 3 words at a time.

Claude transcribes the audio via Whisper, lays word-by-word captions across the video, and fixes any misheard names if you tell it the correct spelling.

Use Case 2 — Motion Graphics

While filming, crop your footage to create a black rectangle where you want a graphic. Drop the clip in and tell Claude:

promptCopy
In the black box while I talk about [topic], animate a [graphic description].

Claude finds the black box automatically and builds motion graphics inside it matched to what you're saying.

Use Case 3 — Sound Design

With your packs in place:

promptCopy
Add [track name] as background music, low under my voice, and place sound effects from the sound pack wherever they fit.

Claude sets music volume low enough that your voice stays prominent and drops effects on animation beats.

Iterating

Everything is adjustable through conversation:

• 'The caption timing is off at 0:12 — fix it'
• 'The music is too loud at the end — fade to -20dB after 0:45'
• 'Move captions to the bottom third'
• 'The graphic needs to be a bar chart, not a list'

Frequently Asked Questions

• Render time? A 60-second clip with captions and one motion graphic: 2–4 minutes on a modern Mac. Renders locally — no cloud queue, no cost.
• What video formats? Any format ffmpeg supports. Convert with ffmpeg -i input.mov output.mp4 if needed.
• Horizontal video? Ask Claude to add a second Composition to src/Root.tsx with a 1920×1080 landscape layout.
• Whisper misheard a word? Tell Claude: 'Whisper heard [wrong word] at 0:23 — it should say [correct word].' It updates the JSON and re-renders.