跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
视频与音频

video-to-audio-foley

Use for video-conditioned audio and automated Foley production: adding synchronized sound effects, ambience, impacts, footsteps, cloth, prop, and environmental audio to silent or under-sounded video using video-to-audio models, text-to-sound tools, manual spot lists, editing, rights checks, and QA.

calesthio · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add calesthio/generative-media-skills --skill video-to-audio-foley

技能说明


name: video-to-audio-foley description: "Use for video-conditioned audio and automated Foley production: adding synchronized sound effects, ambience, impacts, footsteps, cloth, prop, and environmental audio to silent or under-sounded video using video-to-audio models, text-to-sound tools, manual spot lists, editing, rights checks, and QA."

Video-to-audio Foley

Use this skill to turn a silent, generated, animated, game, ad, social, or edit-ready video clip into a credible sound-effects bed. Treat video-to-audio (V2A) as a production accelerator, not as a finished mix. The job is to decide what should be automated, what must be hand-spotted, how to prompt and segment the model, how to preserve sync, and how to verify the result against the picture.

Boundaries

Use automated V2A when:

  • the clip is short enough for the available model or can be divided into coherent beats;
  • the target is a believable first-pass Foley bed, social clip, animatic, ad temp track, game prototype, or atmospheric pass;
  • the visible action has obvious sound causes such as water, footsteps, vehicles, impacts, machinery, crowd, fabric, animals, weather, UI gestures, or prop handling;
  • the user can accept iteration and editorial repair.

Do not rely on automated V2A alone when:

  • frame-exact sync is mission-critical, such as rhythm games, trailers with hard cuts, slapstick hits, weapon impacts, or product demos where every click matters;
  • the output must be delivered as clean reusable game assets or separated stems;
  • the source video contains private people, unreleased product footage, client-confidential material, or copyrighted audio you are not allowed to upload to a hosted provider;
  • the desired sound is an identifiable real person's voice, a copyrighted sound logo, a famous film/game sound, or a protected musical cue;
  • the user needs final dialogue, lip-sync, narration, mix mastering, or music composition rather than Foley/sound effects.

If the user asks for "add sound to this video," first clarify whether they want:

  1. a single mixed audio track attached to the video;
  2. separate ambience / Foley / impacts / props / music stems;
  3. isolated reusable sound assets for a game or library;
  4. only a prompt/workflow, not generated media.

What is documented versus heuristic

Facts below are source-verified as of 2026-07-10 unless a current project/tool registry says otherwise. Provider endpoints, model IDs, costs, licenses, duration limits, output formats, and commercial-use terms are volatile; re-check live docs before making paid calls or promising a deliverable.

Documented facts:

  • Google DeepMind described V2A as research that combines video pixels with natural-language prompts to produce synchronized soundtracks, including sound effects, dialogue, and score-like audio. Do not assume public API access unless current tooling confirms it.
  • fal fal-ai/controlfoley is documented as a hosted video-to-audio model for synchronized sound effects shaped by text prompts. Its API accepts video_url, optional prompt, negative_prompt, optional 2-4 second reference_audio_url, duration, inference controls, and seed; it returns a video with audio and a 44.1 kHz mono WAV audio file.
  • fal fal-ai/thinksound/audio is documented as a hosted V2A endpoint that generates realistic audio from a video with an optional text prompt; if no prompt is provided, the prompt may be extracted from the video. Its schema includes video_url, prompt, seed, num_inference_steps, and cfg_scale, and returns an audio file plus the prompt used.
  • MMAudio is a CVPR 2025 open-source model/repo that generates synchronized audio from video and/or text; its README says the default output/training duration is 8 seconds, larger deviations may reduce quality, the default CLI model is large_44k_v2, and the authors tested on Ubuntu.
  • FoleyCrafter is an open-source video-to-audio framework that uses a text-to-audio base model with a semantic adapter and temporal controller; it supports text prompt and negative prompt control in its repo examples.
  • HunyuanVideo-Foley is a Tencent open-source text-video-to-audio framework; its repo documents single-video and batch inference with infer.py, an audio description prompt, and XL/XXL model sizes. Check current license and model terms before commercial use.
  • ElevenLabs Sound Effects is text-to-sound, not video-conditioned V2A. It is useful for manual Foley layers and ambience loops; docs describe duration control from 0.1 to 30 seconds, looping for seamless atmospheres, and prompt-influence control.
  • FoleyBench frames V2A Foley quality as both semantic alignment with visible events and temporal alignment with event timing. Use both dimensions in QA.

Production heuristics:

  • Prefer one broad V2A pass for continuous ambience and low-risk motion, then layer manual text-to-SFX or library effects for important transients.
  • Split clips by acoustic scene, not by arbitrary equal lengths. A cut from subway platform to kitchen should be two generations even if the model accepts the full duration.
  • If the model cannot accept explicit timecodes, put the timing discipline in the edit: generate candidates, align transients in the DAW/NLE, trim, crossfade, and layer.
  • A good prompt describes the acoustic world, material, perspective, density, and exclusions. It is not just a list of visible objects.
  • For social video, "believable and not distracting" usually beats maximum realism. For game assets and hero ads, separation, repeatability, and legal custody matter more than one-click convenience.

Source-video preparation

Before generation:

  1. Duplicate the source video and preserve the original.
  2. Confirm the user has rights to upload/process the footage and to publish synthetic audio with it.
  3. Remove or mute existing audio unless it is intentionally used as a reference and the provider permits it.
  4. Trim to the natural acoustic unit: one action, one location, one c
阅读完整来源 ↗