跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

troubleshoot-ssi

Diagnose and fix Single Step Instrumentation (SSI) issues on Kubernetes — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are already configured but traces are missing or instrumentation is not working.

datadog-labs · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add datadog-labs/agent-skills --skill troubleshoot-ssi

技能说明


name: troubleshoot-ssi description: Diagnose and fix Single Step Instrumentation (SSI) issues on Linux hosts — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are configured but traces are missing or instrumentation is not working. metadata: version: "1.0.0" author: datadog-labs repository: https://github.com/datadog-labs/agent-skills tags: datadog,apm,linux,ssi,troubleshooting,instrumentation,ld-preload alwaysApply: "false"

Troubleshoot APM SSI on Linux

Triggers

Invoke this skill when the user expresses intent to:

  • Debug why a Linux process is not being instrumented
  • Investigate why traces are not appearing in Datadog from a Linux host
  • Diagnose SSI injection failures on Linux
  • Follow up on failed checks from verify-ssi
  • Report that a specific service or host has no traces

Do NOT invoke this skill if:

  • SSI has not been enabled yet — run enable-ssi first

Critical: pup First, SSH Second

You do NOT need SSH access to start troubleshooting. The pup CLI queries Datadog's backend directly. Start with pup commands immediately using information the user already gave you (hostname, service name, env). Only go to SSH if pup doesn't reveal the cause.

pup-cli: check, install, and authenticate

Claude runs

pup --version

If not found, install it (OS-aware):

Claude runs

if [[ "$(uname)" == "Darwin" ]]; then
  brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
else
  PUP_VERSION=$(curl -s https://api.github.com/repos/datadog-labs/pup/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
  curl -L "https://github.com/datadog-labs/pup/releases/download/${PUP_VERSION}/pup_linux_amd64.tar.gz" | tar xz -C /usr/local/bin pup
  chmod +x /usr/local/bin/pup
fi
pup --version

Auth — check in this order:

  1. Check OAuth status:
pup auth status --site <DD_SITE>

If authenticated — proceed directly to Step 1.

ERROR: Not authenticated:

Claude runs

pup auth login --site <DD_SITE>

This opens a browser tab for OAuth. Complete the login there — Claude will continue once the command exits.

  1. If OAuth login is not possible (e.g., no browser access), fall back to API keys:
echo "DD_API_KEY set: $([ -n "${DD_API_KEY:-}" ] && echo yes || echo no)"
echo "DD_APP_KEY set: $([ -n "${DD_APP_KEY:-}" ] && echo yes || echo no)"

If DD_API_KEY and DD_APP_KEY are both set — proceed to Step 1. pup will use them automatically even if pup auth status shows unauthenticated.


Context

Use what the user already provided. Do not ask for missing context upfront — resolve variables lazily, only when a specific step needs them.

VariableHow to resolveWhen needed
DD_HOSTNAMEFrom the user's message, or datadog-agent status via SSHStep 1 — start here
SERVICE_NAMEFrom the user's messageStep 1 — start here
ENVAsk the user only when a command requires itStep 1 (service-library-config get), Step 3
DD_SITEAsk the user, or grep "^site:" /etc/datadog-agent/datadog.yaml via SSHOnly if pup auth check fails
SSH_KEYFrom user or /workspace/.ssh/id_ed25519Step 4 (SSH investigation) only
SSH_USERFrom user or default rootStep 4 (SSH investigation) only
SSH_HOSTFrom user's messageStep 4 (SSH investigation) only

If the user has already provided DD_HOSTNAME and SERVICE_NAME, go directly to Step 1. Do not ask for ENV or SSH details first.


How SSI Works on Linux — Domain Knowledge

Read this before investigating. It gives you the mental model to reason about novel failures.

Injection chain:

  1. Install script (with DD_APM_INSTRUMENTATION_ENABLED=host) installs datadog-apm-inject and language library packages under /opt/datadog-packages/
  2. The inject package writes its launcher path into /etc/ld.so.preload
  3. The Linux dynamic linker pre-loads the launcher into every new process at startup
  4. The launcher detects the process language and loads the appropriate tracer .so from /opt/datadog-packages/datadog-apm-library-<lang>/
  5. The tracer sends spans to the Agent at localhost:8126
  6. The Agent forwards traces to Datadog at intake.<DD_SITE>

Diagnostic layers:

  • pup — sees what Datadog's backend received + injection errors reported by the launcher. Start here.
  • /proc/<pid>/maps — sees the actual shared libraries loaded into a running process. The authoritative check for whether injection succeeded.
  • datadog-agent status — sees whether the local Agent is receiving traces.

Known silent failures:

  • musl libc (Alpine) — launcher is glibc-compiled; musl is ABI-incompatible. Linker loads it but injection silently aborts
  • Existing ddtrace/OTel — launcher detects user-installed tracer and silently disables itself (already_instrumented result class)
  • Unsupported runtime version — silently skipped
  • Process started before SSI was enabled — /etc/ld.so.preload only affects new processes
  • Static binary / Go — Go programs link statically and ignore LD_PRELOAD entirely
  • SELinux/AppArmor — can block /etc/ld.so.preload reads for confined processes
  • Package directory empty/corrupt — datadog-installer status reflects DB registration, not actual files. A package can show as installed while its directory is empty. Always verify files exist under /opt/datadog-packages/<package>/

Service name identity — important: With SSI, DD_SERVICE is often not set in the process environment. The tracer auto-detects a service name. The telemetry-reported name (what pup fleet tracers list and service-library-config get show) may not match what you expect in the APM UI:

  • JVM: telemetry reports jar artifact name with version (e.g. inventory-service-1.0.0), spans use the base name (inventory-service)
  • Python: telemetry may report `
阅读完整来源 ↗