signoz-investigating-alerts
Diagnose why a SigNoz alert fired by correlating the alert's own signal with neighbor signals (error rate, latency, throughput, CPU/memory), traces, and logs around the fire window — and rank likely causes. Make sure to use this skill whenever the user asks "why did this alert fire", "what caused alert X", "investigate this alert", "RCA for the alert that paged me", "what's wrong with [service]" in the context of a recent fire, or otherwise asks for a root-cause analysis of a firing or recently-fired alert. Read-only — does not modify any alert or notification.
SigNoz · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add SigNoz/agent-skills --skill signoz-investigating-alerts技能说明
name: signoz-investigating-alerts description: > Diagnose why a SigNoz alert fired by correlating the alert's own signal with neighbor signals (error rate, latency, throughput, CPU/memory), traces, and logs around the fire window, and rank likely causes. Make sure to use this skill whenever the user asks "why did this alert fire", "what caused alert X", "investigate this alert", "RCA for the alert that paged me", "what's wrong with [service]" in the context of a recent fire, or otherwise asks for a root-cause analysis of a firing or recently-fired alert. Read-only; does not modify any alert or notification. argument-hint: [time window]
Alert Investigate
Diagnose why a SigNoz alert fired. The skill correlates the alert's own
signal with neighbor signals around the fire window, and surfaces a
ranked list of likely causes with supporting evidence. It is the
companion to signoz-explaining-alerts: explain decodes the rule
statically; investigate diagnoses a specific incident.
Prerequisites
This skill calls SigNoz MCP server tools heavily (signoz_get_alert,
signoz_get_alert_history, signoz_execute_builder_query,
signoz_query_metrics, signoz_search_traces, signoz_search_logs,
signoz_get_trace_details, etc.). Before running the workflow,
confirm the signoz_* tools are available. If they are not, the
SigNoz MCP server is not installed or configured; run signoz-mcp-setup first
to initialize or repair the MCP connection. The investigation depends on
correlating multiple MCP queries; without the server there is no way to ground
the analysis.
When to use
Use this skill when the user wants to:
- Understand why a specific alert fired.
- Find the root cause of a recent incident triggered by an alert.
- Correlate the alert's signal with related metrics, traces, and logs.
- Distinguish "real signal" fires from flapping or threshold-mistuning.
Do NOT use when the user wants to:
- Understand what an alert is configured to monitor →
signoz-explaining-alerts. - Create a new alert →
signoz-creating-alerts. - Modify an alert (raise threshold, add hysteresis) → call
signoz_update_alertdirectly. - Run a free-form ad-hoc investigation without an alert as the anchor →
signoz-generating-queries.
Required inputs
| Input | Required | Source if missing |
|---|---|---|
| Alert identifier (rule ID or name) | yes | $ARGUMENTS[0] or recent context |
| Time window | no | default to most recent fire from signoz_get_alert_history |
If the alert name is fuzzy, this skill is best-effort (read-only):
- Call
signoz_list_alert_rules, paginate, fuzzy-match the name. - State the interpretation: "Investigating fire of 'High Error Rate - Checkout' (id 42) at 14:32 UTC. If you meant a different alert or fire, tell me."
- Proceed.
If no firing transition exists in the queried lookback window, stop: there is nothing to investigate. Respond with:
"Alert '[name]' has not fired in the last 7d, so there is no fire window to investigate. Use
signoz-explaining-alertsto walk through the rule, or check whether the alert is enabled."
Workflow
The investigation runs in three tiers with strict early-stop gates. Tier 1 always runs. Tier 2 runs only if tier 1 confirms a real fire. Tier 3 runs only if tier 2 surfaces correlated anomalies. Skipping the gates produces hundreds of unnecessary trace/log queries on quiet alerts.
Step 1: Resolve alert + fire window (Tier 0)
- Resolve the alert id via
signoz_list_alert_rules(paginated) if not given. - Call
signoz_get_alertfor the full rule config, needed to know what query, threshold, and resource scope the alert evaluated. - First call
signoz_get_alert_historywithtimeRange: "7d"andorder: "desc"; omitstateso the timeline includes firing and inactive transitions. Continue only whendata.nextCursorexists (the completeness note also reportshasMore: true). Pass it ascursor, replacetimeRangewith the note's resolved absolutestartandend, and preserve the same state/filter (including omission) and order. Stop whennextCursoris absent / the note reportshasMore: false; never useoffsetor page fullness. If a later intentional state filter means "resolved" / "recovered", useinactive. The enum isinactive|pending|recovering|firing|nodata|disabled;recoveringis a transient keep-firing state, not resolution. Pattern analysis needs the complete transition set. Rows are emitted per label-groupfingerprint; do not interleave them. From the response:- Build rule-wide incident windows from distinct rows where
overallStateChanged: true: anoverallState: "firing"transition opens an incident; the nextoverallState: "inactive"closes it. Deduplicate matching timestamps and sort byunixMilliascending before pairing. Default to the most recent incident unless$ARGUMENTS[1]selects another. - Partition affected series by
fingerprintand retain each row's labels. Use onlystateChanged: truerows to decide when that group fired and resolved, and which group should scope Tier 1–3 queries. - Note the fire pattern from rule-wide transitions or one named fingerprint:
one-off→ single fire with a long quiet period before/after.sustained→ fires that stayed firing for ≥ 1 evaluation cycle.flapping→ ≥ 3 fires within a 1h window, alternating fire/resolve.recurring→ fires at regular intervals (cron-like, e.g., every hour).
- Never infer flapping from different fingerprints. The pattern guides tiers 2/3.
- Build rule-wide incident windows from distinct rows where
Step 2: Tier 1 (what fired and how hard)
This tier always runs. It establishes the fire is real (vs. transient threshold tickle or flap) and quantifies the magnitude.
- Re-run the alert's primary query over a window centered on the fire
start:
[fire_start - 30m, fire_start + 30m]