Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
AI & knowledge

agent-cost-and-latency-spike-investigation

Guides rapidly triaging a sudden cost or latency spike affecting one specific agent workflow — scoping it, correlating it against recent changes, and applying a fast, safe stopgap — before launching a full optimization pass. Use when a user asks "why did our LLM bill jump overnight," "this one workflow got slow/expensive all of a sudden," "investigate a cost/latency spike," or reports an alert/invoice surprise for a single agent or task type, as distinct from a deliberate, scheduled effort to reduce baseline cost or latency.

selvarajmurugesan90 · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add selvarajmurugesan90/ops-engineering-skills --skill agent-cost-and-latency-spike-investigation

About this skill


name: agent-cost-and-latency-spike-investigation description: > Guides rapidly triaging a sudden cost or latency spike affecting one specific agent workflow — scoping it, correlating it against recent changes, and applying a fast, safe stopgap — before launching a full optimization pass. Use when a user asks "why did our LLM bill jump overnight," "this one workflow got slow/expensive all of a sudden," "investigate a cost/latency spike," or reports an alert/invoice surprise for a single agent or task type, as distinct from a deliberate, scheduled effort to reduce baseline cost or latency. license: Apache-2.0 compatibility: "Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI" metadata: domain: ai-agent maturity: stable

Agent Cost and Latency Spike Investigation

Purpose

A sudden cost or latency spike in one agent workflow is an incident, not an optimization project — the goal in the first hours is to scope it, identify what changed, and stop the bleeding, not to redesign the pipeline. llm-cost-and-latency-optimization covers the deliberate, scheduled work of reducing baseline cost/latency across an agent (right-sizing models, caching, batching); this skill covers the narrower, time-pressured question that comes first: why did this one workflow suddenly get more expensive or slower than it was yesterday, and what's the fastest safe action to take. Confusing the two wastes the window where a quick rollback would have worked and instead launches a multi-day optimization effort under incident pressure.

When to use

  • A cost dashboard or billing alert shows a step-change increase for one agent/workflow/task type, not a gradual trend.
  • p50/p95 latency for one specific workflow doubles (or worse) compared to its recent baseline, while other workflows are unaffected.
  • An unexpected line item appears on an LLM provider invoice tied to a specific agent.
  • Before scheduling a full cost/latency optimization pass — this investigation determines whether there's an active regression to fix first, so the optimization pass starts from a correct baseline.
  • Deciding whether a spike is a regression (something broke) or legitimate growth (more users, more traffic) — these require entirely different responses.

Prerequisites & environment

  • Per-call token usage and latency logging, segmented by workflow/task type (not just a single aggregate metric) — if the spike can't be isolated to one workflow because everything rolls into one dashboard number, segmenting the metrics is itself the first prerequisite to fix.
  • A deploy/change log with timestamps: prompt edits, tool schema changes, model version or provider changes, retrieval index re-indexing runs, and infrastructure/routing changes — the single most useful artifact for this investigation is a timeline that can be laid next to the metrics timeline.
  • Request volume metrics alongside cost/latency, so a spike in absolute cost can be distinguished from a spike in cost-per-request.
  • Access to a recent-history transcript sample for the affected workflow (see agent-bad-response-triage-and-root-cause-classification for full-transcript capture practices) so the investigation isn't limited to aggregate numbers alone.

Step-by-step guidance

  1. Scope the blast radius first: one workflow, or everything? If cost or latency moved across all workflows and all providers simultaneously, this is more likely a provider-wide event (outage, pricing change, regional latency issue) than a workflow-specific regression — that's the domain of llm-gateway-and-multi-provider-routing (check provider status, confirm fallback routing triggered correctly) rather than this skill. Confirm the spike is actually isolated to one workflow before proceeding with a workflow-specific investigation.

  2. Pull time series for four signals side by side, not cost alone: request volume, tokens per request (input and output separately), tool-call count per request, and latency p50/p95. Overlay all four against the deploy/change timeline from day one.

    metric               yesterday   today      delta
    requests/hour        1,180       1,205      flat (+2%)
    avg input tokens/req    2,400      2,410      flat
    avg output tokens/req     310      1,850      +497%  <-- signal
    avg tool calls/req        2.1        2.1      flat
    p95 latency (ms)        1,850      6,200      +235%
    

    A table like this immediately narrows the search: flat volume and flat input tokens with a jump in output tokens and latency points at a generation-side change (prompt, output-format drift, or a model change), not a traffic or context-bloat problem.

  3. Apply the decision tree once the shape of the change is visible:

    • Volume up, per-request metrics flat → legitimate traffic growth, not a regression; this is a capacity/budget conversation, not an incident.
    • Volume flat, tokens-per-request up → likely context bloat (uncontrolled history growth, duplicated retrieved chunks) or a prompt/tool-description change — see prompt-and-context-engineering.
    • Volume flat, tool-call count per request up → likely a stalled or looping agent — see agent-tool-call-loop-diagnosis-and-circuit-breaking.
    • Latency up, tokens flat → likely a provider-side latency change, a model/region switch, or a downstream tool/dependency slowdown — see llm-gateway-and-multi-provider-routing.
    • Cost up, tokens flat → likely a pricing change, a model routing change (sil
Read the complete source ↗