跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
视频与音频

make-npcs-talk

Make NPCs in a game talk back. The player speaks or types, the NPC answers in character, in a cloned voice, about 1-2 seconds later, and can act on what was said. Covers the whole loop and the order to build it in (who is being talked to, what the scene is, speech-to-text, an LLM in character, Fish Audio voices on a GPU, streamed playback from the NPC, subtitles, mouth movement, actions), the four hooks a game has to provide, latency budgets, costs, testing in the real game, and the rules (single-player, voices you have the rights to, nothing of the game shipped). Use when the user wants NPCs, characters, villagers, companions or enemies in any game to hold conversations, react to the player's voice, or be voiced by Fish Audio (S2 Pro or Drama 3) and an LLM.

christos-antono · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add christos-antono/NPCTalk --skill make-npcs-talk

技能说明


name: make-npcs-talk description: Make NPCs in a game talk back. The player speaks or types, the NPC answers in character, in a cloned voice, about 1-2 seconds later, and can act on what was said. Covers the whole loop and the order to build it in (who is being talked to, what the scene is, speech-to-text, an LLM in character, Fish Audio voices on a GPU, streamed playback from the NPC, subtitles, mouth movement, actions), the four hooks a game has to provide, latency budgets, costs, testing in the real game, and the rules (single-player, voices you have the rights to, nothing of the game shipped). Use when the user wants NPCs, characters, villagers, companions or enemies in any game to hold conversations, react to the player's voice, or be voiced by Fish Audio (S2 Pro or Drama 3) and an LLM.

Make NPCs talk

The finished loop, as built for GTA V (examples/gta-v), with times measured in the game:

player's mic ──► voice activity ──► speech-to-text ──► who? + scene ──► LLM in character ──► voice server ──► NPC speaks
   (open mic,       (end of the       (fal Nemotron,     (the game)      (Cerebras, with      (Fish Audio on     (streamed PCM,
    push-to-talk,    sentence)         0.4-0.8 s)                         persona + lines,    an H100, first     panned, subtitles,
    or typing)                                                            0.3-1.0 s)          audio ~0.4 s)      mouth, action)
                                                                │
                                                                └──► action choice (Jev, ~0.1 s, in parallel) ──► animation

From the end of the player's sentence to the NPC's voice: 1.2-2.2 s warm with push-to-talk (logged exchanges: 1.15 s, 1.20 s, 1.57 s, 2.19 s); the open mic adds the 0.7 s of silence it waits for before deciding the sentence ended. The LLM sometimes spikes to 5-8 s, so log every step's time per exchange.

The loop: build it in this order

  1. Recon the game. Version, how code runs inside it (script hook, mod loader, plugin API, or your own engine), single-player or not, how other mods find nearby characters and play sounds. Read references/game-hooks.md for the engine, and knowledge/ for anyone who already did this game.
  2. Pick the engines (table below). Default: Fish Audio Drama 3 on Modal for voices, Cerebras for the LLM, fal for speech-to-text.
  3. Stand the voice server up first, outside the game (skill fish-voice-server): npctalk health, then npctalk say "..." --voice Name/clone. A game is the worst place to debug a GPU server.
  4. Make reference voices for two or three characters (skill voice-references).
  5. Vertical slice in the game: one NPC, typed text → LLM → voice → playback. Implement the four hooks below, nothing else. Log stt / llm / tts / total for every exchange.
  6. Voice input (skill voice-input): push-to-talk first, then the open mic.
  7. Characters (skill npc-dialogue): persona files, the character's own lines as style samples, scene facts, per-NPC history, output cleaning.
  8. Actions (skill npc-actions): what they physically do, and doing what they're told.
  9. Verify in the real game, with a dev mode that shows the target, the timings and the mod's state (below).
  10. Show it (skill npc-showcase-clips), and if voice likeness matters, measure it (skill voice-likeness-eval) before claiming a setting is better.
  11. Write it down: a field note in knowledge/games/<game>/ (see knowledge/README.md).

The four hooks a game has to provide

Everything else is game-free code. In the GTA V example about 2,450 of the mod's ~5,500 lines (dialogue, LLM, voice clients, streaming playback, speech-to-text, open mic, action picking) build and run outside the game: examples/gta-v/tools/npcvoice-check compiles exactly those files into a console app.

HookWhat it returnsGTA V (ScriptHookVDotNet)
Who is being talked toThe NPC, or nobodyLiving humans within 12 m; within 5 m always counts, 5-12 m only if the camera points at them (dot > 0.85). Named characters win ties; the last partner sticks while within 6 m
Scene factsA few short stringsZone and street name, time, weather, the player's vehicle and weapon, wanted level, health, whether this is the NPC's home
Voice at the NPCPlay streamed PCM from that NPCNAudio playback, panned by screen position, faded 4-30 m, mouth driven by the audio level, the game's subtitle style
ActionsName → animation or behaviourAnim dictionaries checked at load (a missing one falls back), clothing swaps, tasks (flee, fight, kneel)

Keep the hooks thin. If a game exposes them, the rest of this repo's code carries over.

Choosing engines

NeedDefaultAlternatives
Voice, realtimeFish Audio Drama 3 self-hosted on Modal (server/drama3.py): first audio ~0.4 s warm, acts [tags] wellS2 Pro (server/s2pro.py, smaller, fine-tunable); MiniMax via fal (no GPU to run, $1.50 per voice clone, 2.5-4.5 s per line)
Voice, pre-rendered (cutscenes, demos)Drama 3 with best_of: 8 (best likeness measured)ElevenLabs (scored lowest here, 0.612)
LLMCerebras qwen-3.8-27b, reasoning off (~0.3 s)Any OpenAI-compatible endpoint; reasoning models add seconds
Speech-to-textfal nvidia/nemotron-asr-multilingual/asr (~0.4 s, audio inline)Whisper anywhere
ActionsTypeSafe Jev (typed choice + probability, ~0.1 s)A second LLM call with a JSON schema (slower, less calibrated)

Costs: an H100 runs about $4/hour while it's up; it scales to zero 5 minutes after the last request, and the next request waits 1-3 minutes for a cold start. The game pings the server while it runs, so it stays warm exactly while someone plays (server/keepwarm.py holds it up for a demo without the game).

Verify like you mean it

  • Log every exchange with each st
阅读完整来源 ↗