troubleshoot-ssi

Source documentation, not instructions for this website. Review permissions before running any commands.

pup --version

if [[ "$(uname)" == "Darwin" ]]; then
  brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
else
  PUP_VERSION=$(curl -s https://api.github.com/repos/datadog-labs/pup/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
  curl -L "https://github.com/datadog-labs/pup/releases/download/${PUP_VERSION}/pup_linux_amd64.tar.gz" | tar xz -C /usr/local/bin pup
  chmod +x /usr/local/bin/pup
fi
pup --version

pup auth status --site <DD_SITE>

pup auth login --site <DD_SITE>

echo "DD_API_KEY set: $([ -n "${DD_API_KEY:-}" ] && echo yes || echo no)"
echo "DD_APP_KEY set: $([ -n "${DD_APP_KEY:-}" ] && echo yes || echo no)"

# Check for injection errors (failures only — successful injections don't appear here)
pup apm troubleshooting list --hostname <DD_HOSTNAME>

# Check full tracer config — look at apm_enabled, trace_agent_url, site
pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>

# Check what services have sent traces (reveals actual service names visible to backend)
pup apm services list --from 1h

# Check if traces exist at all
pup traces search --query "service:<SERVICE_NAME>" --from 15m --limit 5

# Fastest trace confirmation — metrics appear before indexed traces
pup metrics query --query "sum:trace.*.request.hits{host:<DD_HOSTNAME>,service:<SERVICE_NAME>}.as_count()" --from 15m

# Show all config values with their source (env_var, remote_config, code, default)
pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>

# Show only configs where instances disagree (config drift)
pup apm service-library-config get --service-name <SERVICE_NAME> --mixed

pup traces sea

name: troubleshoot-ssi
description: Diagnose and fix Single Step Instrumentation (SSI) issues on Linux hosts — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are configured but traces are missing or instrumentation is not working.
metadata:
  version: "1.0.0"
  author: datadog-labs
  repository: https://github.com/datadog-labs/agent-skills
  tags: datadog,apm,linux,ssi,troubleshooting,instrumentation,ld-preload
  alwaysApply: "false"

---
name: troubleshoot-ssi
description: Diagnose and fix Single Step Instrumentation (SSI) issues on Linux hosts — SSI automatically instruments applications for APM without code changes. Only use if the agent and SSI are configured but traces are missing or instrumentation is not working.
metadata:
  version: "1.0.0"
  author: datadog-labs
  repository: https://github.com/datadog-labs/agent-skills
  tags: datadog,apm,linux,ssi,troubleshooting,instrumentation,ld-preload
  alwaysApply: "false"
---

# Troubleshoot APM SSI on Linux

## Triggers

Invoke this skill when the user expresses intent to:
- Debug why a Linux process is not being instrumented
- Investigate why traces are not appearing in Datadog from a Linux host
- Diagnose SSI injection failures on Linux
- Follow up on failed checks from `verify-ssi`
- Report that a specific service or host has no traces

Do NOT invoke this skill if:
- SSI has not been enabled yet — run `enable-ssi` first

---

## Critical: pup First, SSH Second

**You do NOT need SSH access to start troubleshooting.** The `pup` CLI queries Datadog's backend directly. Start with pup commands immediately using information the user already gave you (hostname, service name, env). Only go to SSH if pup doesn't reveal the cause.

### pup-cli: check, install, and authenticate

### Claude runs

```bash
pup --version
```

If not found, install it (OS-aware):

### Claude runs

```bash
if [[ "$(uname)" == "Darwin" ]]; then
  brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
else
  PUP_VERSION=$(curl -s https://api.github.com/repos/datadog-labs/pup/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
  curl -L "https://github.com/datadog-labs/pup/releases/download/${PUP_VERSION}/pup_linux_amd64.tar.gz" | tar xz -C /usr/local/bin pup
  chmod +x /usr/local/bin/pup
fi
pup --version
```

**Auth — check in this order:**

1. Check OAuth status:
```bash
pup auth status --site <DD_SITE>
```

If authenticated — proceed directly to Step 1.

ERROR: Not authenticated:

### Claude runs

```bash
pup auth login --site <DD_SITE>
```

> This opens a browser tab for OAuth. Complete the login there — Claude will continue once the command exits.

2. If OAuth login is not possible (e.g., no browser access), fall back to API keys:
```bash
echo "DD_API_KEY set: $([ -n "${DD_API_KEY:-}" ] && echo yes || echo no)"
echo "DD_APP_KEY set: $([ -n "${DD_APP_KEY:-}" ] && echo yes || echo no)"
```

If `DD_API_KEY` and `DD_APP_KEY` are both set — **proceed to Step 1**. pup will use them automatically even if `pup auth status` shows unauthenticated.

---

## Context

Use what the user already provided. Do not ask for missing context upfront — resolve variables lazily, only when a specific step needs them.

| Variable | How to resolve | When needed |
|---|---|---|
| `DD_HOSTNAME` | From the user's message, or `datadog-agent status` via SSH | Step 1 — start here |
| `SERVICE_NAME` | From the user's message | Step 1 — start here |
| `ENV` | Ask the user only when a command requires it | Step 1 (`service-library-config get`), Step 3 |
| `DD_SITE` | Ask the user, or `grep "^site:" /etc/datadog-agent/datadog.yaml` via SSH | Only if pup auth check fails |
| `SSH_KEY` | From user or `/workspace/.ssh/id_ed25519` | Step 4 (SSH investigation) only |
| `SSH_USER` | From user or default `root` | Step 4 (SSH investigation) only |
| `SSH_HOST` | From user's message | Step 4 (SSH investigation) only |

**If the user has already provided `DD_HOSTNAME` and `SERVICE_NAME`, go directly to Step 1. Do not ask for ENV or SSH details first.**

---

## How SSI Works on Linux — Domain Knowledge

Read this before investigating. It gives you the mental model to reason about novel failures.

**Injection chain:**
1. Install script (with `DD_APM_INSTRUMENTATION_ENABLED=host`) installs `datadog-apm-inject` and language library packages under `/opt/datadog-packages/`
2. The inject package writes its launcher path into `/etc/ld.so.preload`
3. The Linux dynamic linker pre-loads the launcher into every new process at startup
4. The launcher detects the process language and loads the appropriate tracer `.so` from `/opt/datadog-packages/datadog-apm-library-<lang>/`
5. The tracer sends spans to the Agent at `localhost:8126`
6. The Agent forwards traces to Datadog at `intake.<DD_SITE>`

**Diagnostic layers:**
- **`pup`** — sees what Datadog's backend received + injection errors reported by the launcher. Start here.
- **`/proc/<pid>/maps`** — sees the actual shared libraries loaded into a running process. The authoritative check for whether injection succeeded.
- **`datadog-agent status`** — sees whether the local Agent is receiving traces.

**Known silent failures:**
- **musl libc (Alpine)** — launcher is glibc-compiled; musl is ABI-incompatible. Linker loads it but injection silently aborts
- **Existing ddtrace/OTel** — launcher detects user-installed tracer and silently disables itself (`already_instrumented` result class)
- **Unsupported runtime version** — silently skipped
- **Process started before SSI was enabled** — `/etc/ld.so.preload` only affects new processes
- **Static binary / Go** — Go programs link statically and ignore `LD_PRELOAD` entirely
- **SELinux/AppArmor** — can block `/etc/ld.so.preload` reads for confined processes
- **Package directory empty/corrupt** — `datadog-installer status` reflects DB registration, not actual files. A package can show as installed while its directory is empty. Always verify files exist under `/opt/datadog-packages/<package>/`

**Service name identity — important:**
With SSI, `DD_SERVICE` is often not set in the process environment. The tracer auto-detects a service name. The telemetry-reported name (what `pup fleet tracers list` and `service-library-config get` show) may not match what you expect in the APM UI:
- **JVM**: telemetry reports jar artifact name with version (e.g. `inventory-service-1.0.0`), spans use the base name (`inventory-service`)
- **Python**: telemetry may report `fastapi` or `django` rather than the app name
- **Node.js**: names typically match

If `service-library-config get` returns empty, use `pup traces search --query "host:<DD_HOSTNAME>" --from 1h --limit 5` to discover what service names have been sending traces, then retry.

---

## Step 1: Triage with pup (no SSH required)

Run these first. The answers determine everything that follows.

### Claude runs

```bash
# Check for injection errors (failures only — successful injections don't appear here)
pup apm troubleshooting list --hostname <DD_HOSTNAME>

# Check full tracer config — look at apm_enabled, trace_agent_url, site
pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>

# Check what services have sent traces (reveals actual service names visible to backend)
pup apm services list --from 1h

# Check if traces exist at all
pup traces search --query "service:<SERVICE_NAME>" --from 15m --limit 5

# Fastest trace confirmation — metrics appear before indexed traces
pup metrics query --query "sum:trace.*.request.hits{host:<DD_HOSTNAME>,service:<SERVICE_NAME>}.as_count()" --from 15m
```

`ENV` is required for `service-library-config get`. If the user didn't provide it, state your assumed value (e.g. `prod`) and run it anyway — don't stop to ask.

Key values to check in `service-library-config get` output:
- `apm_enabled` — must be `true`. If `false`, the tracer won't send traces regardless of injection.
- `trace_agent_url` — must point to `http://localhost:8126` or the correct agent socket. Wrong value = tracer can't reach the Agent.
- `site` — must match your Datadog org's site.

---

## Presenting your findings (required)

Your final response is the deliverable — not your investigation transcript. It must include **every diagnostic from this skill that you ran or that applies**, each with its purpose and what you found. Avoid these failure modes:

- **Omitting a required diagnostic.** Your response must explicitly include, by name:
  - `pup apm troubleshooting list --hostname <DD_HOSTNAME>` — injection errors
  - `pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>` — runtime SDK config
  - verification that **`apm_enabled` is `true`** and that **`trace_agent_url` points to the correct agent endpoint** (e.g. `http://localhost:8126`)
  - `datadog-agent status` for the APM receiver, and a `/proc/<pid>/maps` (or equivalent) check that the tracer `.so` is loaded into the process

  Run them if you have access; recommend them for the user to run if you don't.
- **Substituting a proxy check.** Confirming port 8126 is listening shows the Agent's receiver is up — it is **not** a substitute for verifying the tracer-side `apm_enabled` and `trace_agent_url` config keys. They answer different questions; report both.
- **Concluding before the checks.** Outline and run the diagnostics before settling on a root cause, then give specific per-finding remediation and note that the service must be restarted after any config or package fix.

---

## Step 2: State Your Hypotheses

Before investigating, explicitly state your ranked hypotheses based on triage output. Do not skip this step.

| Triage signal | Strong hypothesis |
|---|---|
| `pup troubleshooting list` shows `result: error`, `result_class: incorrect_installation` | Package directory empty or corrupt — verify files exist under `/opt/datadog-packages/datadog-apm-library-<lang>/`, then use remediation flow |
| `pup troubleshooting list` shows `result: error`, import/load error | Tracer library couldn't be loaded — check runtime version, libc compatibility |
| `pup troubleshooting list` shows `result: abort`, reason `already_instrumented` | Manual ddtrace/OTel already in the app — launcher silently disabled itself |
| `pup troubleshooting list` shows `result: abort`, reason `language not detected` | Expected for non-app processes (e.g., bash, cron). Not a failure. |
| `pup troubleshooting list` empty | Either no injection attempts yet (process not restarted), or injection succeeded silently |
| `service-library-config get` shows `apm_enabled: false` | Tracer is loaded but explicitly disabled — check `source` field to see who set it |
| `service-library-config get` shows `trace_agent_url` pointing to wrong host/port | Tracer can't reach the Agent — fix the URL |
| `service-library-config get` shows wrong `site` | Traces going to wrong Datadog org |
| No traces in `pup traces search`, no troubleshooting errors | Process was never injected — check: process not restarted after SSI enabled, `/etc/ld.so.preload` missing, static binary |
| Unexpected service name in `pup apm services list` results | Service name mismatch — use the actual name from trace data for subsequent config lookups |
| Traces arriving in pup | Not a real problem — likely a UI filter or time window. Tell the user and stop. |

State your top 1-3 hypotheses explicitly: *"Based on triage, I think the most likely cause is X because Y."*

---

## Step 3: Investigate with pup (deeper)

Use only the tools relevant to your hypotheses.

**Check SDK config in detail:**
```bash
# Show all config values with their source (env_var, remote_config, code, default)
pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>

# Show only configs where instances disagree (config drift)
pup apm service-library-config get --service-name <SERVICE_NAME> --mixed
```

Key values to check:
- `apm_enabled` — if `false`, tracer won't send traces. Check `source` to see who disabled it (`code` > `env_var` > `remote_config` > `default`)
- `trace_agent_url` — should be `http://localhost:8126` or a Unix socket. Wrong value = tracer can't reach Agent
- `site` — must match your Datadog org's site. Mismatch = traces going to wrong org
- `service` — with SSI and no `DD_SERVICE` set, `source: default` is expected

**If `service-library-config get` returns empty** — the service name you're using may not match the actual name in trace data:
```bash
pup traces sea