signoz-reducing-telemetry-cost
Investigate and reduce SigNoz telemetry ingestion cost and metric cardinality across metrics, logs, and traces. Find what drives SigNoz spend (via the Cost Meter), which metrics have runaway or unbounded label cardinality, and safe, dashboard-, alert-, and Infra-page-aware ways to cut volume. Make sure to use this skill whenever the user asks "why is my SigNoz bill so high", "what's driving my ingestion cost", "reduce telemetry volume", "which metrics cost the most", "cardinality health check", or "what can I safely drop" — or otherwise asks about telemetry spend, ingestion volume, or metric cardinality, even if they don't say "cost" or "optimize" explicitly.
SigNoz · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add SigNoz/agent-skills --skill signoz-reducing-telemetry-cost技能说明
name: signoz-reducing-telemetry-cost description: > Investigate and reduce SigNoz telemetry ingestion cost and metric cardinality across metrics, logs, and traces. Find what drives SigNoz spend (via the Cost Meter), which metrics have runaway or unbounded label cardinality, and safe, dashboard-, alert-, and Infra-page-aware ways to cut volume. Make sure to use this skill whenever the user asks "why is my SigNoz bill so high", "what's driving my ingestion cost", "reduce telemetry volume", "which metrics cost the most", "cardinality health check", or "what can I safely drop" — or otherwise asks about telemetry spend, ingestion volume, or metric cardinality, even if they don't say "cost" or "optimize" explicitly. argument-hint: <investigation focus, such as metrics cost or cardinality health>
Prerequisites
This skill calls SigNoz MCP server tools heavily (signoz_list_metrics,
signoz_get_field_keys, signoz_execute_builder_query,
signoz_get_top_metrics, signoz_check_metric_usage, signoz_check_metric_cardinality,
signoz_aggregate_logs, signoz_aggregate_traces, signoz_search_logs,
signoz_list_alert_rules, signoz_get_alert, signoz_get_service_top_operations). Before
running the workflow, confirm the signoz_* tools are available. If they are not, the SigNoz
MCP server is not installed or configured — run signoz-mcp-setup first. The whole
investigation is grounded in these queries; without the server there is nothing to analyze.
Read both reference files before drawing conclusions:
references/otel-attribute-cardinality.md— classify any metric label you encounter.references/infra-do-not-drop.md— the metrics that power the built-in Infrastructure page and the APM/Services page (span-derivedsignoz_*RED metrics); never present these as "safe to drop" even when usage shows them unused.
Workflow
Always start with the Cost Meter snapshot (Step 1). For a full cost investigation, run the metrics, logs, and traces steps for every signal with data, ordered by current cost contribution (primary, secondary, tertiary). Finish with the report (Step 5).
Step 1: Cost Meter snapshot
Establish the cross-signal cost picture first. Always call signoz_list_metrics with
source: "meter"; treat its returned metric names, types, temporalities, and units as the live
source of truth because the meter set evolves. Then query each relevant discovered metric with
signoz_execute_builder_query (source: "meter", requestType: "time_series",
stepInterval: 3600, the discovered temporality, and timeAggregation: "sum") — see
references/cost-meter-queries.md for the full tool-argument template. Sum complete hourly
buckets and exclude every datapoint marked partial: true. Do not use
signoz_query_metrics for Cost Meter totals or grouped total attribution.
Report only values returned by successful queries. If a query fails or returns no usable values
after the MCP tools are available, show the intended query and say that the total could not be
computed; never invent a total. If the tools are unavailable, follow the prerequisite instead.
For a rolling 7-day window (end = now, start = end − 7 days), get the per-signal totals
(span size, log size, metric datapoints), then compute and report:
- Primary cost driver. The signal with the highest dollar weight — traces/logs at $0.30/GB, metrics at $0.10/M samples (orientation only; never quote dollar savings). This picks by cost, not by raw volume.
- Bytes per record. span.size ÷ span.count and log.size ÷ log.count — tells you whether a signal is a payload-size problem or a volume problem.
Then break the primary signal down by environment and service. First call
signoz_get_field_keys with signal: "metrics" and source: "meter"; use only keys it returns
and copy each key's name, fieldDataType, fieldContext, and signal into the raw
builder groupBy without translating or dropping fields. Run the same meter query with that
complete groupBy and report the top
~10 per group with their share. If a non-prod environment (staging, dev, test, qa,
sandbox, preview, uat, …) is > 40% of volume, recommend Ingestion Limits on that key
before any signal-level change: https://signoz.io/docs/ingestion/signoz-cloud/keys/
Step 2: Metrics
Run when the Cost Meter shows metric data, ordered by its cost contribution, or when the user explicitly asks about metric cost or cardinality.
2a. Rank by volume — signoz_get_top_metrics. Returns the top 100 metrics by ingested
samples with percentages pre-computed and totalValue sample counts (pass start/end). This
is the volume-ranked worklist. Histogram metrics (.bucket suffix) are usually the top
contributors — each bucket boundary is a separate sample per scrape.
2b. Check usage — signoz_check_metric_usage. Pass the top metric names (batch of ≤ 50 per
call). Returns { dashboards, alerts, error } per metric. A metric is a drop candidate only when
its error is empty and both dashboards and alerts are empty. If error is non-empty the
lookup is incomplete (a timeout, or an older SigNoz that lacks the endpoint) and the returned
lists are unreliable — never treat that metric as unused; mark it Needs one check first (verify
its usage manually). A clean lookup with both lists empty is a drop candidate — except the guard
below.
阅读完整来源 ↗Do-not-drop guard (mandatory). Before calling any empty-usage metric a "safe drop", check it against
references/infra-do-not-drop.md. The Infrastructure page (Hosts / Kubernetes) queriessystem.*and manyk8s.*/container.*metrics through built-in queries — not dashboards — so usage-check reports them empty even though dropping them breaks that page. If a candidate matches the do-not-drop set, present it as "Infra-page dependency — breaks the Hosts/Kubernetes view; confirm you don't use that view before dropping,"