agent-cost-and-latency-spike-investigation
Guides rapidly triaging a sudden cost or latency spike affecting one specific agent workflow — scoping it, correlating it against recent changes, and applying a fast, safe stopgap — before launching a full optimization pass. Use when a user asks "why did our LLM bill jump overnight," "this one workflow got slow/expensive all of a sudden," "investigate a cost/latency spike," or reports an alert/invoice surprise for a single agent or task type, as distinct from a deliberate, scheduled effort to reduce baseline cost or latency.
selvarajmurugesan90 · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add selvarajmurugesan90/ops-engineering-skills --skill agent-cost-and-latency-spike-investigationAbout this skill
name: agent-cost-and-latency-spike-investigation description: > Guides rapidly triaging a sudden cost or latency spike affecting one specific agent workflow — scoping it, correlating it against recent changes, and applying a fast, safe stopgap — before launching a full optimization pass. Use when a user asks "why did our LLM bill jump overnight," "this one workflow got slow/expensive all of a sudden," "investigate a cost/latency spike," or reports an alert/invoice surprise for a single agent or task type, as distinct from a deliberate, scheduled effort to reduce baseline cost or latency. license: Apache-2.0 compatibility: "Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI" metadata: domain: ai-agent maturity: stable
Agent Cost and Latency Spike Investigation
Purpose
A sudden cost or latency spike in one agent workflow is an incident, not an optimization project — the goal in the first hours is to scope it, identify what changed, and stop the bleeding, not to redesign the pipeline. llm-cost-and-latency-optimization covers the deliberate, scheduled work of reducing baseline cost/latency across an agent (right-sizing models, caching, batching); this skill covers the narrower, time-pressured question that comes first: why did this one workflow suddenly get more expensive or slower than it was yesterday, and what's the fastest safe action to take. Confusing the two wastes the window where a quick rollback would have worked and instead launches a multi-day optimization effort under incident pressure.
When to use
- A cost dashboard or billing alert shows a step-change increase for one agent/workflow/task type, not a gradual trend.
- p50/p95 latency for one specific workflow doubles (or worse) compared to its recent baseline, while other workflows are unaffected.
- An unexpected line item appears on an LLM provider invoice tied to a specific agent.
- Before scheduling a full cost/latency optimization pass — this investigation determines whether there's an active regression to fix first, so the optimization pass starts from a correct baseline.
- Deciding whether a spike is a regression (something broke) or legitimate growth (more users, more traffic) — these require entirely different responses.
Prerequisites & environment
- Per-call token usage and latency logging, segmented by workflow/task type (not just a single aggregate metric) — if the spike can't be isolated to one workflow because everything rolls into one dashboard number, segmenting the metrics is itself the first prerequisite to fix.
- A deploy/change log with timestamps: prompt edits, tool schema changes, model version or provider changes, retrieval index re-indexing runs, and infrastructure/routing changes — the single most useful artifact for this investigation is a timeline that can be laid next to the metrics timeline.
- Request volume metrics alongside cost/latency, so a spike in absolute cost can be distinguished from a spike in cost-per-request.
- Access to a recent-history transcript sample for the affected workflow (see agent-bad-response-triage-and-root-cause-classification for full-transcript capture practices) so the investigation isn't limited to aggregate numbers alone.
Step-by-step guidance
-
Scope the blast radius first: one workflow, or everything? If cost or latency moved across all workflows and all providers simultaneously, this is more likely a provider-wide event (outage, pricing change, regional latency issue) than a workflow-specific regression — that's the domain of llm-gateway-and-multi-provider-routing (check provider status, confirm fallback routing triggered correctly) rather than this skill. Confirm the spike is actually isolated to one workflow before proceeding with a workflow-specific investigation.
-
Pull time series for four signals side by side, not cost alone: request volume, tokens per request (input and output separately), tool-call count per request, and latency p50/p95. Overlay all four against the deploy/change timeline from day one.
metric yesterday today delta requests/hour 1,180 1,205 flat (+2%) avg input tokens/req 2,400 2,410 flat avg output tokens/req 310 1,850 +497% <-- signal avg tool calls/req 2.1 2.1 flat p95 latency (ms) 1,850 6,200 +235%A table like this immediately narrows the search: flat volume and flat input tokens with a jump in output tokens and latency points at a generation-side change (prompt, output-format drift, or a model change), not a traffic or context-bloat problem.
-
Apply the decision tree once the shape of the change is visible:
- Volume up, per-request metrics flat → legitimate traffic growth, not a regression; this is a capacity/budget conversation, not an incident.
- Volume flat, tokens-per-request up → likely context bloat (uncontrolled history growth, duplicated retrieved chunks) or a prompt/tool-description change — see prompt-and-context-engineering.
- Volume flat, tool-call count per request up → likely a stalled or looping agent — see agent-tool-call-loop-diagnosis-and-circuit-breaking.
- Latency up, tokens flat → likely a provider-side latency change, a model/region switch, or a downstream tool/dependency slowdown — see llm-gateway-and-multi-provider-routing.
- Cost up, tokens flat → likely a pricing change, a model routing change (sil