跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
视频与音频

ai-media-generation-expert

Expert guide for AI image generation (Flux, DALL-E, Stable Diffusion), video generation (Sora, Runway), voice synthesis (ElevenLabs TTS), and speech recognition (Whisper STT) integration / Panduan ahli integrasi AI generasi gambar, video, suara (TTS), dan pengenalan suara (STT).

roedyrustam · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add roedyrustam/vibes-plug --skill ai-media-generation-expert

技能说明


name: ai-media-generation-expert description: "Expert guide for AI image generation (Flux, DALL-E, Stable Diffusion), video generation (Sora, Runway), voice synthesis (ElevenLabs TTS), and speech recognition (Whisper STT) integration / Panduan ahli integrasi AI generasi gambar, video, suara (TTS), dan pengenalan suara (STT)." author: "Roedy Rustam"

version: "3.0.0"

AI Media Generation Expert (2026 Edition)

English | Bahasa Indonesia


English

Orchestration & Integration

  • ai-llm-integration-expert: Core LLM patterns and model selection.
  • file-upload-media-expert: Storage, CDN, and media pipeline for generated assets.
  • sse-websocket-streaming-expert: Real-time streaming for progressive image/audio generation.
  • async-queue-temporal-expert: Background job processing for long-running generation tasks.
  • zero-trust-secret-vault: Secure API key management for Replicate, OpenAI, ElevenLabs.

Description

Production-grade guide for integrating AI-powered media generation into web and mobile applications. Covers image generation (Flux 1.1 Pro, SDXL, DALL-E 3), video generation (Sora, Runway Gen-3), text-to-speech (ElevenLabs v3, OpenAI TTS), speech-to-text (Whisper large-v3, Deepgram Nova-3), and voice cloning. Includes async processing patterns, cost optimization, and content safety filtering.

Trigger Conditions

  • Integrating AI image generation (Flux, DALL-E, Stable Diffusion, Midjourney API).
  • Building text-to-speech or speech-to-text features.
  • Implementing voice cloning or AI avatar generation.
  • Adding AI video generation or editing capabilities.
  • Building media processing pipelines with AI models.

Core Architecture

1. Image Generation

Provider Selection Matrix:

ProviderModelSpeedQualityCostBest For
ReplicateFlux 1.1 Pro~3s★★★★★$0.04/imgPhotorealism, text rendering
ReplicateFlux Schnell~1s★★★★$0.003/imgHigh-volume, previews
OpenAIDALL-E 3~5s★★★★$0.04/imgCreative, prompt following
StabilitySDXL Turbo~2s★★★★$0.002/imgSelf-hosted, customization

Recommendation: Use Flux 1.1 Pro via Replicate for production quality. Use Flux Schnell for previews/drafts.

// Image generation with Replicate (Flux)
import Replicate from 'replicate';

const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN });

async function generateImage(prompt: string, options?: {
  width?: number;
  height?: number;
  model?: 'flux-1.1-pro' | 'flux-schnell';
}) {
  const model = options?.model ?? 'flux-1.1-pro';
  const output = await replicate.run(
    `black-forest-labs/${model}`,
    {
      input: {
        prompt,
        width: options?.width ?? 1024,
        height: options?.height ?? 1024,
        num_inference_steps: model === 'flux-schnell' ? 4 : 28,
      },
    }
  );
  return output; // URL to generated image
}

2. Text-to-Speech (TTS)

Provider Selection:

ProviderModelLatencyExpressivenessCost
ElevenLabseleven_v3~500ms★★★★★$0.30/1K chars
ElevenLabseleven_flash_v2_5~200ms★★★★$0.15/1K chars
OpenAItts-1-hd~300ms★★★$0.030/1K chars

Recommendation: Use ElevenLabs eleven_v3 for expressive content. Use eleven_flash_v2_5 for real-time chatbots.

// ElevenLabs TTS streaming
async function textToSpeech(text: string, voiceId: string): Promise<ReadableStream> {
  const response = await fetch(
    `https://api.elevenlabs.io/v1/text-to-speech/${voiceId}/stream`,
    {
      method: 'POST',
      headers: {
        'xi-api-key': process.env.ELEVENLABS_API_KEY!,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        text,
        model_id: 'eleven_v3',
        voice_settings: { stability: 0.5, similarity_boost: 0.75 },
      }),
    }
  );
  return response.body!; // Stream audio chunks
}

3. Speech-to-Text (STT)

// OpenAI Whisper transcription
import OpenAI from 'openai';

const openai = new OpenAI();

async function transcribeAudio(audioFile: File) {
  const transcription = await openai.audio.transcriptions.create({
    file: audioFile,
    model: 'whisper-1',
    response_format: 'verbose_json',
    timestamp_granularities: ['word', 'segment'],
  });
  return transcription;
}

4. Video Generation

// Replicate video generation (async with webhook)
async function generateVideo(prompt: string) {
  const prediction = await replicate.predictions.create({
    model: 'minimax/video-01',
    input: { prompt, duration: 5 },
    webhook: `${process.env.APP_URL}/api/webhooks/replicate`,
    webhook_events_filter: ['completed'],
  });
  return prediction.id; // Poll or wait for webhook
}

5. Production Patterns

  • Async Processing: Always use background jobs (BullMQ, Inngest) for generation tasks >2s.
  • Webhook Architecture: Use webhooks for Replicate predictions instead of polling.
  • Content Safety: Implement NSFW filtering (OpenAI Moderation API, Replicate safety checker).
  • Cost Control: Set per-user daily limits. Cache identical prompts. Use cheaper models for drafts.
  • Storage Pipeline: Generate → upload to S3/R2 → serve via CDN → store URL in DB.

Bahasa Indonesia

Integrasi Orkestrasi

  • ai-llm-integration-expert: Pola LLM inti dan pemilihan model.
  • file-upload-media-expert: Penyimpanan, CDN, dan pipeline media untuk aset yang dihasilkan.
  • sse-websocket-streaming-expert: Streaming real-time untuk generasi gambar/
阅读完整来源 ↗