video-to-audio-foley
Use for video-conditioned audio and automated Foley production: adding synchronized sound effects, ambience, impacts, footsteps, cloth, prop, and environmental audio to silent or under-sounded video using video-to-audio models, text-to-sound tools, manual spot lists, editing, rights checks, and QA.
calesthio · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add calesthio/generative-media-skills --skill video-to-audio-foleyAbout this skill
name: video-to-audio-foley description: "Use for video-conditioned audio and automated Foley production: adding synchronized sound effects, ambience, impacts, footsteps, cloth, prop, and environmental audio to silent or under-sounded video using video-to-audio models, text-to-sound tools, manual spot lists, editing, rights checks, and QA."
Video-to-audio Foley
Use this skill to turn a silent, generated, animated, game, ad, social, or edit-ready video clip into a credible sound-effects bed. Treat video-to-audio (V2A) as a production accelerator, not as a finished mix. The job is to decide what should be automated, what must be hand-spotted, how to prompt and segment the model, how to preserve sync, and how to verify the result against the picture.
Boundaries
Use automated V2A when:
- the clip is short enough for the available model or can be divided into coherent beats;
- the target is a believable first-pass Foley bed, social clip, animatic, ad temp track, game prototype, or atmospheric pass;
- the visible action has obvious sound causes such as water, footsteps, vehicles, impacts, machinery, crowd, fabric, animals, weather, UI gestures, or prop handling;
- the user can accept iteration and editorial repair.
Do not rely on automated V2A alone when:
- frame-exact sync is mission-critical, such as rhythm games, trailers with hard cuts, slapstick hits, weapon impacts, or product demos where every click matters;
- the output must be delivered as clean reusable game assets or separated stems;
- the source video contains private people, unreleased product footage, client-confidential material, or copyrighted audio you are not allowed to upload to a hosted provider;
- the desired sound is an identifiable real person's voice, a copyrighted sound logo, a famous film/game sound, or a protected musical cue;
- the user needs final dialogue, lip-sync, narration, mix mastering, or music composition rather than Foley/sound effects.
If the user asks for "add sound to this video," first clarify whether they want:
- a single mixed audio track attached to the video;
- separate ambience / Foley / impacts / props / music stems;
- isolated reusable sound assets for a game or library;
- only a prompt/workflow, not generated media.
What is documented versus heuristic
Facts below are source-verified as of 2026-07-10 unless a current project/tool registry says otherwise. Provider endpoints, model IDs, costs, licenses, duration limits, output formats, and commercial-use terms are volatile; re-check live docs before making paid calls or promising a deliverable.
Documented facts:
- Google DeepMind described V2A as research that combines video pixels with natural-language prompts to produce synchronized soundtracks, including sound effects, dialogue, and score-like audio. Do not assume public API access unless current tooling confirms it.
- fal
fal-ai/controlfoleyis documented as a hosted video-to-audio model for synchronized sound effects shaped by text prompts. Its API acceptsvideo_url, optionalprompt,negative_prompt, optional 2-4 secondreference_audio_url,duration, inference controls, and seed; it returns a video with audio and a 44.1 kHz mono WAV audio file. - fal
fal-ai/thinksound/audiois documented as a hosted V2A endpoint that generates realistic audio from a video with an optional text prompt; if no prompt is provided, the prompt may be extracted from the video. Its schema includesvideo_url,prompt,seed,num_inference_steps, andcfg_scale, and returns an audio file plus the prompt used. - MMAudio is a CVPR 2025 open-source model/repo that generates synchronized audio from video and/or text; its README says the default output/training duration is 8 seconds, larger deviations may reduce quality, the default CLI model is
large_44k_v2, and the authors tested on Ubuntu. - FoleyCrafter is an open-source video-to-audio framework that uses a text-to-audio base model with a semantic adapter and temporal controller; it supports text prompt and negative prompt control in its repo examples.
- HunyuanVideo-Foley is a Tencent open-source text-video-to-audio framework; its repo documents single-video and batch inference with
infer.py, an audio description prompt, and XL/XXL model sizes. Check current license and model terms before commercial use. - ElevenLabs Sound Effects is text-to-sound, not video-conditioned V2A. It is useful for manual Foley layers and ambience loops; docs describe duration control from 0.1 to 30 seconds, looping for seamless atmospheres, and prompt-influence control.
- FoleyBench frames V2A Foley quality as both semantic alignment with visible events and temporal alignment with event timing. Use both dimensions in QA.
Production heuristics:
- Prefer one broad V2A pass for continuous ambience and low-risk motion, then layer manual text-to-SFX or library effects for important transients.
- Split clips by acoustic scene, not by arbitrary equal lengths. A cut from subway platform to kitchen should be two generations even if the model accepts the full duration.
- If the model cannot accept explicit timecodes, put the timing discipline in the edit: generate candidates, align transients in the DAW/NLE, trim, crossfade, and layer.
- A good prompt describes the acoustic world, material, perspective, density, and exclusions. It is not just a list of visible objects.
- For social video, "believable and not distracting" usually beats maximum realism. For game assets and hero ads, separation, repeatability, and legal custody matter more than one-click convenience.
Source-video preparation
Before generation:
- Duplicate the source video and preserve the original.
- Confirm the user has rights to upload/process the footage and to publish synthetic audio with it.
- Remove or mute existing audio unless it is intentionally used as a reference and the provider permits it.
- Trim to the natural acoustic unit: one action, one location, one c