跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

llm-cost-and-latency-optimization

Guides reducing token cost and response latency of LLM-based agents without degrading quality. Use when a user asks to "reduce our LLM API bill," "make the agent respond faster," "our token usage is too high," "should we use a smaller/cheaper model here," decide where to apply prompt caching, streaming, or batching, or needs to size a cost/latency budget before scaling an agent to more users.

selvarajmurugesan90 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add selvarajmurugesan90/ops-engineering-skills --skill llm-cost-and-latency-optimization

技能说明


name: llm-cost-and-latency-optimization description: > Guides reducing token cost and response latency of LLM-based agents without degrading quality. Use when a user asks to "reduce our LLM API bill," "make the agent respond faster," "our token usage is too high," "should we use a smaller/cheaper model here," decide where to apply prompt caching, streaming, or batching, or needs to size a cost/latency budget before scaling an agent to more users. license: Apache-2.0 compatibility: "Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI" metadata: domain: ai-agent maturity: stable

LLM Cost and Latency Optimization

Purpose

LLM API cost and response latency scale with tokens processed and number of model calls — both of which are almost always higher than necessary in a first working version of an agent, because it's easier to build without budgeting either. Left unaddressed, this shows up as a surprising bill at scale, or an agent that feels sluggish enough that users stop trusting it to be interactive. Unlike raw model-quality tuning, most of the levers here are structural and don't require changing which model you use at all: reducing redundant context, caching stable prompt prefixes, choosing the right model per step rather than the strongest model for everything, and parallelizing or streaming where the task allows it. This skill treats cost and latency together because most fixes affect both, though not always in the same direction.

When to use

  • Token/API costs for an agent are higher than expected or growing faster than usage.
  • An agent's end-to-end response time is too slow for its use case (interactive chat vs. background batch job have very different tolerances).
  • Deciding whether a task step needs the strongest available model or can use a smaller/cheaper one.
  • Evaluating whether prompt caching, batching, or streaming applies to a given workload.
  • Sizing a cost/latency budget before scaling an agent from a prototype to production traffic.
  • Reviewing an agent design for redundant or unnecessary model calls before it ships.

Prerequisites & environment

  • Access to per-call token usage and latency metrics from your model provider's API responses (most APIs return input/output token counts per call; capture and log these, don't estimate).
  • Current pricing and context-window/caching capabilities for the specific model(s) in use — these vary by vendor and change over time, so verify against current provider documentation rather than assuming figures from memory or from a different model generation.
  • A representative load profile (typical conversation length, typical tool call count per task) to reason about cost/latency at realistic scale, not just a single test call.

Step-by-step guidance

  1. Measure before optimizing. Instrument every model call with input tokens, output tokens, latency, and (if using tools) tool-call count. Aggregate by agent, by task type, and by pipeline stage — you cannot prioritize fixes without knowing which stage actually dominates cost or latency.

    def call_llm(messages, tools=None):
        start = time.monotonic()
        response = client.messages.create(model=MODEL, messages=messages, tools=tools)
        metrics.record(
            stage="agent_loop",
            input_tokens=response.usage.input_tokens,
            output_tokens=response.usage.output_tokens,
            latency_ms=(time.monotonic() - start) * 1000,
        )
        return response
    
  2. Cut redundant context first — it's usually the largest and cheapest fix. Audit what's actually being sent on each call: full conversation history with no windowing, full raw tool outputs instead of trimmed results, duplicated retrieved chunks across turns. See prompt-and-context-engineering for concrete history-management and budgeting techniques — this is usually higher-leverage than model choice.

  3. Use prompt caching for stable prefixes. If your provider supports prompt/context caching, structure calls so the stable part (system prompt, tool definitions, static reference material) forms a consistent prefix, and only the per-turn variable content (user message, retrieved chunks) changes after it. This reduces both cost and latency on cache hits, often substantially, but the exact discount and minimum cacheable prefix length are provider- and model-specific — check current documentation for the model you're using.

  4. Right-size the model per step, not per agent. A multi-step pipeline rarely needs the strongest available model at every step:

    plan step (ambiguous, needs strong reasoning)  -> strongest available model
    extraction/formatting step (well-specified)     -> smaller/faster model
    final safety/quality check                      -> smaller model or rule-based check
    

    Validate this split against your eval suite (see agent-evaluation-and-guardrails) before committing — a cheaper model may be entirely adequate for a well-specified step, or may not be, and that's an empirical question, not an assumption.

  5. Parallelize independent calls instead of serializing them. If a task requires several independent tool calls or sub-agent calls with no data dependency between them (see multi-agent-orchestration), issue them concurrently rather than one after another — this reduces wall-clock latency without changing total token cost.

  6. Stream output for interactive use cases. For anything a human waits on synchronously, stream tokens as they're generated rather than waiting for the full response — this improves perceived latency significantly even when total generation time is unchanged, and costs nothing extra.

  7. **Batch non-interactive

阅读完整来源 ↗