troubleshoot-ssi
Diagnose and fix Single Step Instrumentation (SSI) issues on Kubernetes — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are already configured but traces are missing or instrumentation is not working.
datadog-labs · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add datadog-labs/agent-skills --skill troubleshoot-ssi技能说明
name: troubleshoot-ssi description: Diagnose and fix Single Step Instrumentation (SSI) issues on Linux hosts — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are configured but traces are missing or instrumentation is not working. metadata: version: "1.0.0" author: datadog-labs repository: https://github.com/datadog-labs/agent-skills tags: datadog,apm,linux,ssi,troubleshooting,instrumentation,ld-preload alwaysApply: "false"
Troubleshoot APM SSI on Linux
Triggers
Invoke this skill when the user expresses intent to:
- Debug why a Linux process is not being instrumented
- Investigate why traces are not appearing in Datadog from a Linux host
- Diagnose SSI injection failures on Linux
- Follow up on failed checks from
verify-ssi - Report that a specific service or host has no traces
Do NOT invoke this skill if:
- SSI has not been enabled yet — run
enable-ssifirst
Critical: pup First, SSH Second
You do NOT need SSH access to start troubleshooting. The pup CLI queries Datadog's backend directly. Start with pup commands immediately using information the user already gave you (hostname, service name, env). Only go to SSH if pup doesn't reveal the cause.
pup-cli: check, install, and authenticate
Claude runs
pup --version
If not found, install it (OS-aware):
Claude runs
if [[ "$(uname)" == "Darwin" ]]; then
brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
else
PUP_VERSION=$(curl -s https://api.github.com/repos/datadog-labs/pup/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
curl -L "https://github.com/datadog-labs/pup/releases/download/${PUP_VERSION}/pup_linux_amd64.tar.gz" | tar xz -C /usr/local/bin pup
chmod +x /usr/local/bin/pup
fi
pup --version
Auth — check in this order:
- Check OAuth status:
pup auth status --site <DD_SITE>
If authenticated — proceed directly to Step 1.
ERROR: Not authenticated:
Claude runs
pup auth login --site <DD_SITE>
This opens a browser tab for OAuth. Complete the login there — Claude will continue once the command exits.
- If OAuth login is not possible (e.g., no browser access), fall back to API keys:
echo "DD_API_KEY set: $([ -n "${DD_API_KEY:-}" ] && echo yes || echo no)"
echo "DD_APP_KEY set: $([ -n "${DD_APP_KEY:-}" ] && echo yes || echo no)"
If DD_API_KEY and DD_APP_KEY are both set — proceed to Step 1. pup will use them automatically even if pup auth status shows unauthenticated.
Context
Use what the user already provided. Do not ask for missing context upfront — resolve variables lazily, only when a specific step needs them.
| Variable | How to resolve | When needed |
|---|---|---|
DD_HOSTNAME | From the user's message, or datadog-agent status via SSH | Step 1 — start here |
SERVICE_NAME | From the user's message | Step 1 — start here |
ENV | Ask the user only when a command requires it | Step 1 (service-library-config get), Step 3 |
DD_SITE | Ask the user, or grep "^site:" /etc/datadog-agent/datadog.yaml via SSH | Only if pup auth check fails |
SSH_KEY | From user or /workspace/.ssh/id_ed25519 | Step 4 (SSH investigation) only |
SSH_USER | From user or default root | Step 4 (SSH investigation) only |
SSH_HOST | From user's message | Step 4 (SSH investigation) only |
If the user has already provided DD_HOSTNAME and SERVICE_NAME, go directly to Step 1. Do not ask for ENV or SSH details first.
How SSI Works on Linux — Domain Knowledge
Read this before investigating. It gives you the mental model to reason about novel failures.
Injection chain:
- Install script (with
DD_APM_INSTRUMENTATION_ENABLED=host) installsdatadog-apm-injectand language library packages under/opt/datadog-packages/ - The inject package writes its launcher path into
/etc/ld.so.preload - The Linux dynamic linker pre-loads the launcher into every new process at startup
- The launcher detects the process language and loads the appropriate tracer
.sofrom/opt/datadog-packages/datadog-apm-library-<lang>/ - The tracer sends spans to the Agent at
localhost:8126 - The Agent forwards traces to Datadog at
intake.<DD_SITE>
Diagnostic layers:
pup— sees what Datadog's backend received + injection errors reported by the launcher. Start here./proc/<pid>/maps— sees the actual shared libraries loaded into a running process. The authoritative check for whether injection succeeded.datadog-agent status— sees whether the local Agent is receiving traces.
Known silent failures:
- musl libc (Alpine) — launcher is glibc-compiled; musl is ABI-incompatible. Linker loads it but injection silently aborts
- Existing ddtrace/OTel — launcher detects user-installed tracer and silently disables itself (
already_instrumentedresult class) - Unsupported runtime version — silently skipped
- Process started before SSI was enabled —
/etc/ld.so.preloadonly affects new processes - Static binary / Go — Go programs link statically and ignore
LD_PRELOADentirely - SELinux/AppArmor — can block
/etc/ld.so.preloadreads for confined processes - Package directory empty/corrupt —
datadog-installer statusreflects DB registration, not actual files. A package can show as installed while its directory is empty. Always verify files exist under/opt/datadog-packages/<package>/
Service name identity — important:
With SSI, DD_SERVICE is often not set in the process environment. The tracer auto-detects a service name. The telemetry-reported name (what pup fleet tracers list and service-library-config get show) may not match what you expect in the APM UI:
- JVM: telemetry reports jar artifact name with version (e.g.
inventory-service-1.0.0), spans use the base name (inventory-service) - Python: telemetry may report `