跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

signoz-investigating-alerts

Diagnose why a SigNoz alert fired by correlating the alert's own signal with neighbor signals (error rate, latency, throughput, CPU/memory), traces, and logs around the fire window — and rank likely causes. Make sure to use this skill whenever the user asks "why did this alert fire", "what caused alert X", "investigate this alert", "RCA for the alert that paged me", "what's wrong with [service]" in the context of a recent fire, or otherwise asks for a root-cause analysis of a firing or recently-fired alert. Read-only — does not modify any alert or notification.

SigNoz · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add SigNoz/agent-skills --skill signoz-investigating-alerts

技能说明


name: signoz-investigating-alerts description: > Diagnose why a SigNoz alert fired by correlating the alert's own signal with neighbor signals (error rate, latency, throughput, CPU/memory), traces, and logs around the fire window, and rank likely causes. Make sure to use this skill whenever the user asks "why did this alert fire", "what caused alert X", "investigate this alert", "RCA for the alert that paged me", "what's wrong with [service]" in the context of a recent fire, or otherwise asks for a root-cause analysis of a firing or recently-fired alert. Read-only; does not modify any alert or notification. argument-hint: [time window]

Alert Investigate

Diagnose why a SigNoz alert fired. The skill correlates the alert's own signal with neighbor signals around the fire window, and surfaces a ranked list of likely causes with supporting evidence. It is the companion to signoz-explaining-alerts: explain decodes the rule statically; investigate diagnoses a specific incident.

Prerequisites

This skill calls SigNoz MCP server tools heavily (signoz_get_alert, signoz_get_alert_history, signoz_execute_builder_query, signoz_query_metrics, signoz_search_traces, signoz_search_logs, signoz_get_trace_details, etc.). Before running the workflow, confirm the signoz_* tools are available. If they are not, the SigNoz MCP server is not installed or configured; run signoz-mcp-setup first to initialize or repair the MCP connection. The investigation depends on correlating multiple MCP queries; without the server there is no way to ground the analysis.

When to use

Use this skill when the user wants to:

  • Understand why a specific alert fired.
  • Find the root cause of a recent incident triggered by an alert.
  • Correlate the alert's signal with related metrics, traces, and logs.
  • Distinguish "real signal" fires from flapping or threshold-mistuning.

Do NOT use when the user wants to:

  • Understand what an alert is configured to monitor → signoz-explaining-alerts.
  • Create a new alert → signoz-creating-alerts.
  • Modify an alert (raise threshold, add hysteresis) → call signoz_update_alert directly.
  • Run a free-form ad-hoc investigation without an alert as the anchor → signoz-generating-queries.

Required inputs

InputRequiredSource if missing
Alert identifier (rule ID or name)yes$ARGUMENTS[0] or recent context
Time windownodefault to most recent fire from signoz_get_alert_history

If the alert name is fuzzy, this skill is best-effort (read-only):

  1. Call signoz_list_alert_rules, paginate, fuzzy-match the name.
  2. State the interpretation: "Investigating fire of 'High Error Rate - Checkout' (id 42) at 14:32 UTC. If you meant a different alert or fire, tell me."
  3. Proceed.

If no firing transition exists in the queried lookback window, stop: there is nothing to investigate. Respond with:

"Alert '[name]' has not fired in the last 7d, so there is no fire window to investigate. Use signoz-explaining-alerts to walk through the rule, or check whether the alert is enabled."

Workflow

The investigation runs in three tiers with strict early-stop gates. Tier 1 always runs. Tier 2 runs only if tier 1 confirms a real fire. Tier 3 runs only if tier 2 surfaces correlated anomalies. Skipping the gates produces hundreds of unnecessary trace/log queries on quiet alerts.

Step 1: Resolve alert + fire window (Tier 0)

  1. Resolve the alert id via signoz_list_alert_rules (paginated) if not given.
  2. Call signoz_get_alert for the full rule config, needed to know what query, threshold, and resource scope the alert evaluated.
  3. First call signoz_get_alert_history with timeRange: "7d" and order: "desc"; omit state so the timeline includes firing and inactive transitions. Continue only when data.nextCursor exists (the completeness note also reports hasMore: true). Pass it as cursor, replace timeRange with the note's resolved absolute start and end, and preserve the same state/filter (including omission) and order. Stop when nextCursor is absent / the note reports hasMore: false; never use offset or page fullness. If a later intentional state filter means "resolved" / "recovered", use inactive. The enum is inactive|pending|recovering|firing|nodata|disabled; recovering is a transient keep-firing state, not resolution. Pattern analysis needs the complete transition set. Rows are emitted per label-group fingerprint; do not interleave them. From the response:
    • Build rule-wide incident windows from distinct rows where overallStateChanged: true: an overallState: "firing" transition opens an incident; the next overallState: "inactive" closes it. Deduplicate matching timestamps and sort by unixMilli ascending before pairing. Default to the most recent incident unless $ARGUMENTS[1] selects another.
    • Partition affected series by fingerprint and retain each row's labels. Use only stateChanged: true rows to decide when that group fired and resolved, and which group should scope Tier 1–3 queries.
    • Note the fire pattern from rule-wide transitions or one named fingerprint:
      • one-off → single fire with a long quiet period before/after.
      • sustained → fires that stayed firing for ≥ 1 evaluation cycle.
      • flapping → ≥ 3 fires within a 1h window, alternating fire/resolve.
      • recurring → fires at regular intervals (cron-like, e.g., every hour).
    • Never infer flapping from different fingerprints. The pattern guides tiers 2/3.

Step 2: Tier 1 (what fired and how hard)

This tier always runs. It establishes the fire is real (vs. transient threshold tickle or flap) and quantifies the magnitude.

  1. Re-run the alert's primary query over a window centered on the fire start: [fire_start - 30m, fire_start + 30m]
阅读完整来源 ↗