--- name: clickstack-otel-collector description: Use when a user wants to wire an OpenTelemetry collector into a Managed ClickStack service on ClickHouse Cloud, either by deploying a new local collector (Docker run or Docker Compose) or by configuring their own existing collector, then send rich synthetic telemetry and verify it is visible in ClickStack. license: Apache-2.0 metadata: author: ClickHouse Inc version: "0.6.0" ---

# Set up an OpenTelemetry collector for Managed ClickStack

This skill wires an OpenTelemetry collector into a Managed ClickStack service running on ClickHouse Cloud, sends rich synthetic telemetry through it, and confirms the data is actually visible in ClickStack. It uses [`clickhousectl`](https://clickhouse.com/docs/interfaces/cli) for all cloud and SQL operations.

**Scope.** This skill supports two paths, chosen in Step 0:

1. **Deploy a new collector** locally. You can do this **two ways: individual `docker` commands, or a `docker compose` file** (recommended, fewer commands and one file to start/stop). Make the user aware of both up front and let them pick in Step 0; do not assume plain `docker`. Either way runs the ClickStack distribution of the collector, preconfigured for Managed ClickStack. 2. **Configure your own existing collector** by adding the ClickHouse exporter configuration. We give you the exact config to drop in; you reload your collector. Use this if you already run a collector in a **gateway** role.

A full Kubernetes deployment (Helm, secrets in K8s Secrets) is out of scope here; the config we generate in path 2 can be applied to a collector running anywhere.

The end state is:

- A dedicated `hyperdx_ingest` SQL user on the target service, with exactly the grants the collector needs (it creates the `otel.*` schema on first write). - A collector forwarding logs, traces, and metrics into the `otel` database on the service, either the new local ClickStack collector or your existing one. - Rich synthetic telemetry across several services, severities, span statuses, and metric types, so ClickStack's Search, Service Map, and dashboards have something real to show. - The service confirmed **awake**, and the user walked through the ClickStack onboarding in the Cloud console so they can actually *see* their data.

Secrets (the OTLP auth token and the SQL password) are generated locally, written **once** to a `0600` env file, and passed to Docker via `--env-file`. They are never pasted into the chat, never passed with `docker run -e`, and never echoed back after creation.

Follow these steps in order. Each step depends on state established by the previous one.

---

## Step 0: Choose your path

Ask the user two short questions before doing anything else, because they determine which later steps run.

**Question 1: Do you already have an OpenTelemetry collector running in a gateway role?**

- **No, set one up for me.** -> the **new-collector** path. Continue to Question 2. - **Yes, I have one.** -> the **existing-collector** path. Skip Question 2 (it does not apply), and in Step 6 you will configure their collector rather than deploy a new one.

**Question 2 (new-collector path only): Run the collector with individual Docker commands, or a Docker Compose file?**

- **Docker Compose (recommended).** Fewer commands, one file to start and stop, easiest to re-run. Best if `docker compose` is available. - **Individual Docker commands.** Use if Compose is not installed or you prefer explicit commands.

Record the answers as `COLLECTOR_PATH` (`new` or `existing`) and, for the new path, `DEPLOY_MODE` (`compose` or `run`). Refer back to them in Step 6 and Step 7.

---

## Step 1: Batch the permissions up front

Coding agents prompt for approval the first time they see each shell command. To avoid interrupting the user every few steps, ask them once, up front, to allowlist the command prefixes below (the "always allow for this project / session" option in their agent). There are no destructive operations and nothing targets anything outside this project or their ClickHouse Cloud service.

| Command prefix | Used for | Needed when | | --- | --- | --- | | `openssl rand …` | generate the OTLP token and SQL password | always | | `clickhousectl cloud …` | auth, resolve the service, run SQL via the Query API | always | | `jq …` | parse JSON from `clickhousectl` | always | | `docker …` / `docker compose …` | run/inspect the collector and the telemetry generator | new-collector path, and the optional telemetry check | | `curl …` | local health check against `localhost:13133` (and installing `clickhousectl` if missing) | new-collector path |

Tell the user, in your own words: *"If your agent supports it, choose 'always allow' for each of these the first time it asks. The whole run is read-only against your machine except for the collector container, and write operations against ClickHouse are limited to creating the ingest user and the `otel` schema."*

If the user is on the existing-collector path and does not want to run the optional telemetry check, you can drop `docker` and `curl` from the list.

**Two approvals are semantic, not prefix-based, so allowlisting won't pre-clear them.** Warn the user to expect these and approve them explicitly when they appear:

- The `clickhousectl` install in Step 3 uses `curl … | sh`, which many agent sandboxes flag as "downloading and running untrusted code" regardless of any `curl` allowlist rule. - The `CREATE USER` / `GRANT` in Step 5 may be flagged as "modifying shared production infrastructure," again independent of the `clickhousectl` prefix rule.

Neither is solved by the table above; they are one-time, intentional, and safe to approve.

Then continue.

---

## Step 2: Confirm the target service and lay down the secrets file

The user's prompt contains a service identifier, either a service ID (UUID) or a service name. Treat that value as `SERVICE_REF`.

Create a working directory and a **`0600` env file** that will hold all configuration and secrets for this run. The key names match exactly what the collector image reads, so this same file is passed straight to `docker run --env-file` (or referenced by Compose) in Step 6. Write it under a tight `umask` so the secret is never briefly world-readable:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}" mkdir -p "$WORKDIR" && chmod 700 "$WORKDIR" ENV_FILE="$WORKDIR/collector.env"

# Generate secrets WITHOUT printing them; write straight into a private file. ( umask 177 { echo "SERVICE_REF=$SERVICE_REF" echo "OTLP_AUTH_TOKEN=$(openssl rand -hex 32)" echo "CLICKHOUSE_USER=hyperdx_ingest" echo "CLICKHOUSE_PASSWORD=$(openssl rand -hex 24)Aa1-" echo "HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel" } > "$ENV_FILE" ) chmod 600 "$ENV_FILE" ls -l "$ENV_FILE" ```

Two things about these values matter and are easy to get wrong:

- **Key names are exact.** The collector reads `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_ENDPOINT`, and `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE`. Store the SQL password under `CLICKHOUSE_PASSWORD` (not a custom name); if it is missing, the collector starts with an **empty** password and dies with `code: 516, Authentication failed`. - **The password charset is constrained from three directions at once.** ClickHouse Cloud rejects passwords without at least one uppercase character and one special character, so a plain hex string fails at `CREATE USER`. At the same time, the collector's migration tool embeds the password in a connection URL, so `@`, `:`, `/`, `?`, `#`, and `%` corrupt it (symptom: `code: 516` at startup even though the password is "correct"). The recipe above is random hex (lowercase + digits) plus the suffix `Aa1-`, which adds the required uppercase, a digit, and a **URL-unreserved** special character (`-`). The OTLP token has no such rules (it is just a bearer token), so plain hex is fine for it.

The env file uses **bare `KEY=VALUE` lines with no quotes**: Docker's `--env-file` does not do shell parsing, so any quotes you add become part of the value.

On the **existing-collector path** the `OTLP_AUTH_TOKEN` is not used by your collector (auth on your receiver is your own setup); it is generated only so the same file works if you later switch to the local collector. The `CLICKHOUSE_*` values are still used: they go into the exporter config you add to your collector in Step 6.

**Every later step runs in a fresh shell, so `WORKDIR`, `ENV_FILE`, and any exported credentials do not persist, and `WORKDIR`/`ENV_FILE` are not stored inside the env file, so sourcing it can't recover them.** Begin each subsequent step's shell with this **standard preamble**, which re-derives the paths from the deterministic default, loads the saved credentials (Step 3), and loads the config:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env" [ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a ```

If you chose a non-default `WORKDIR`, set it explicitly at the top of every step (the `${WORKDIR:-…}` default only covers the standard location). Later steps refer to this as "the standard preamble".

**Confirm with the user** that `SERVICE_REF` is correct. Tell them the working directory and that `collector.env` (mode `0600`) now holds the OTLP token and the SQL password. Do **not** print either secret. If they want to see a value, point them at the file (`grep OTLP_AUTH_TOKEN "$ENV_FILE"`).

If the user supplied their own token or password, write those into the file instead of the generated ones, but keep the same `0600` discipline and make sure any custom password still meets the charset rules above.

---

## Step 3: Authenticate `clickhousectl` (separate terminal by default)

Check `clickhousectl` is on `PATH`. Run this presence check **on its own**, not chained to the installer: the `|| curl … | sh` form drags a harmless check into a compound command that sandboxes deny wholesale as an untrusted-code download.

```bash which clickhousectl ```

Only if that prints nothing, install it (the user may need to approve this explicitly, see Step 1):

```bash curl -fsSL https://clickhouse.com/cli | sh ```

Check authentication:

```bash clickhousectl cloud auth status ```

This skill needs **API key authentication**: OAuth is read-only and cannot create users or run write queries. If the `API key` row is not `Active`, the user must authenticate.

**Do not ask the user to paste their API key and secret into the chat.** Anything pasted into the conversation lives in the transcript and has to be rotated afterward. Instead, ask them to authenticate in a **separate terminal**, then tell you when they are done:

> I need a ClickHouse Cloud **Admin** API key to create the ingest user and verify the data. > Please don't paste it here. Instead: > > 1. In the [Cloud console](https://console.clickhouse.cloud), open **Organization → API keys > → New API key**, and give it the **Admin** role. (Developer-scoped keys can't provision the > per-service Query API endpoint that `cloud service query` uses.) > 2. In a **separate terminal**, run: > > ```bash > clickhousectl cloud auth login --api-key <key-id> --api-secret <key-secret> > ``` > > 3. Tell me when that's done and I'll re-check the auth status.

Poll until the API key row reports `Active`, then confirm with a real privileged call rather than trusting the status table alone. Use a **ref-agnostic** call here: `SERVICE_REF` may be a name, and `cloud service get` only accepts a UUID, so confirming with `get` would fail on a name for reasons unrelated to auth. `cloud service list` needs no ref and proves the API key works:

```bash clickhousectl cloud auth status clickhousectl cloud service list --json | jq -r '.[].name' ```

If the list returns your services, you are authenticated; continue. The actual name-or-UUID resolution of `S

--- name: clickstack-otel-collector description: Use when a user wants to wire an OpenTelemetry collector into a Managed ClickStack service on ClickHouse Cloud, either by deploying a new local collector (Docker run or Docker Compose) or by configuring their own existing collector, then send rich synthetic telemetry and verify it is visible in ClickStack. license: Apache-2.0 metadata: author: ClickHouse Inc version: "0.6.0" ---

# Set up an OpenTelemetry collector for Managed ClickStack

This skill wires an OpenTelemetry collector into a Managed ClickStack service running on ClickHouse Cloud, sends rich synthetic telemetry through it, and confirms the data is actually visible in ClickStack. It uses [`clickhousectl`](https://clickhouse.com/docs/interfaces/cli) for all cloud and SQL operations.

**Scope.** This skill supports two paths, chosen in Step 0:

1. **Deploy a new collector** locally. You can do this **two ways: individual `docker` commands, or a `docker compose` file** (recommended, fewer commands and one file to start/stop). Make the user aware of both up front and let them pick in Step 0; do not assume plain `docker`. Either way runs the ClickStack distribution of the collector, preconfigured for Managed ClickStack. 2. **Configure your own existing collector** by adding the ClickHouse exporter configuration. We give you the exact config to drop in; you reload your collector. Use this if you already run a collector in a **gateway** role.

A full Kubernetes deployment (Helm, secrets in K8s Secrets) is out of scope here; the config we generate in path 2 can be applied to a collector running anywhere.

The end state is:

- A dedicated `hyperdx_ingest` SQL user on the target service, with exactly the grants the collector needs (it creates the `otel.*` schema on first write). - A collector forwarding logs, traces, and metrics into the `otel` database on the service, either the new local ClickStack collector or your existing one. - Rich synthetic telemetry across several services, severities, span statuses, and metric types, so ClickStack's Search, Service Map, and dashboards have something real to show. - The service confirmed **awake**, and the user walked through the ClickStack onboarding in the Cloud console so they can actually *see* their data.

Secrets (the OTLP auth token and the SQL password) are generated locally, written **once** to a `0600` env file, and passed to Docker via `--env-file`. They are never pasted into the chat, never passed with `docker run -e`, and never echoed back after creation.

Follow these steps in order. Each step depends on state established by the previous one.

---

## Step 0: Choose your path

Ask the user two short questions before doing anything else, because they determine which later steps run.

**Question 1: Do you already have an OpenTelemetry collector running in a gateway role?**

- **No, set one up for me.** -> the **new-collector** path. Continue to Question 2. - **Yes, I have one.** -> the **existing-collector** path. Skip Question 2 (it does not apply), and in Step 6 you will configure their collector rather than deploy a new one.

**Question 2 (new-collector path only): Run the collector with individual Docker commands, or a Docker Compose file?**

- **Docker Compose (recommended).** Fewer commands, one file to start and stop, easiest to re-run. Best if `docker compose` is available. - **Individual Docker commands.** Use if Compose is not installed or you prefer explicit commands.

Record the answers as `COLLECTOR_PATH` (`new` or `existing`) and, for the new path, `DEPLOY_MODE` (`compose` or `run`). Refer back to them in Step 6 and Step 7.

---

## Step 1: Batch the permissions up front

Coding agents prompt for approval the first time they see each shell command. To avoid interrupting the user every few steps, ask them once, up front, to allowlist the command prefixes below (the "always allow for this project / session" option in their agent). There are no destructive operations and nothing targets anything outside this project or their ClickHouse Cloud service.

| Command prefix | Used for | Needed when | | --- | --- | --- | | `openssl rand …` | generate the OTLP token and SQL password | always | | `clickhousectl cloud …` | auth, resolve the service, run SQL via the Query API | always | | `jq …` | parse JSON from `clickhousectl` | always | | `docker …` / `docker compose …` | run/inspect the collector and the telemetry generator | new-collector path, and the optional telemetry check | | `curl …` | local health check against `localhost:13133` (and installing `clickhousectl` if missing) | new-collector path |

Tell the user, in your own words: *"If your agent supports it, choose 'always allow' for each of these the first time it asks. The whole run is read-only against your machine except for the collector container, and write operations against ClickHouse are limited to creating the ingest user and the `otel` schema."*

If the user is on the existing-collector path and does not want to run the optional telemetry check, you can drop `docker` and `curl` from the list.

**Two approvals are semantic, not prefix-based, so allowlisting won't pre-clear them.** Warn the user to expect these and approve them explicitly when they appear:

- The `clickhousectl` install in Step 3 uses `curl … | sh`, which many agent sandboxes flag as "downloading and running untrusted code" regardless of any `curl` allowlist rule. - The `CREATE USER` / `GRANT` in Step 5 may be flagged as "modifying shared production infrastructure," again independent of the `clickhousectl` prefix rule.

Neither is solved by the table above; they are one-time, intentional, and safe to approve.

Then continue.

---

## Step 2: Confirm the target service and lay down the secrets file

The user's prompt contains a service identifier, either a service ID (UUID) or a service name. Treat that value as `SERVICE_REF`.

Create a working directory and a **`0600` env file** that will hold all configuration and secrets for this run. The key names match exactly what the collector image reads, so this same file is passed straight to `docker run --env-file` (or referenced by Compose) in Step 6. Write it under a tight `umask` so the secret is never briefly world-readable:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}" mkdir -p "$WORKDIR" && chmod 700 "$WORKDIR" ENV_FILE="$WORKDIR/collector.env"

# Generate secrets WITHOUT printing them; write straight into a private file. ( umask 177 { echo "SERVICE_REF=$SERVICE_REF" echo "OTLP_AUTH_TOKEN=$(openssl rand -hex 32)" echo "CLICKHOUSE_USER=hyperdx_ingest" echo "CLICKHOUSE_PASSWORD=$(openssl rand -hex 24)Aa1-" echo "HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel" } > "$ENV_FILE" ) chmod 600 "$ENV_FILE" ls -l "$ENV_FILE" ```

Two things about these values matter and are easy to get wrong:

- **Key names are exact.** The collector reads `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_ENDPOINT`, and `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE`. Store the SQL password under `CLICKHOUSE_PASSWORD` (not a custom name); if it is missing, the collector starts with an **empty** password and dies with `code: 516, Authentication failed`. - **The password charset is constrained from three directions at once.** ClickHouse Cloud rejects passwords without at least one uppercase character and one special character, so a plain hex string fails at `CREATE USER`. At the same time, the collector's migration tool embeds the password in a connection URL, so `@`, `:`, `/`, `?`, `#`, and `%` corrupt it (symptom: `code: 516` at startup even though the password is "correct"). The recipe above is random hex (lowercase + digits) plus the suffix `Aa1-`, which adds the required uppercase, a digit, and a **URL-unreserved** special character (`-`). The OTLP token has no such rules (it is just a bearer token), so plain hex is fine for it.

The env file uses **bare `KEY=VALUE` lines with no quotes**: Docker's `--env-file` does not do shell parsing, so any quotes you add become part of the value.

On the **existing-collector path** the `OTLP_AUTH_TOKEN` is not used by your collector (auth on your receiver is your own setup); it is generated only so the same file works if you later switch to the local collector. The `CLICKHOUSE_*` values are still used: they go into the exporter config you add to your collector in Step 6.

**Every later step runs in a fresh shell, so `WORKDIR`, `ENV_FILE`, and any exported credentials do not persist, and `WORKDIR`/`ENV_FILE` are not stored inside the env file, so sourcing it can't recover them.** Begin each subsequent step's shell with this **standard preamble**, which re-derives the paths from the deterministic default, loads the saved credentials (Step 3), and loads the config:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env" [ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a ```

If you chose a non-default `WORKDIR`, set it explicitly at the top of every step (the `${WORKDIR:-…}` default only covers the standard location). Later steps refer to this as "the standard preamble".

**Confirm with the user** that `SERVICE_REF` is correct. Tell them the working directory and that `collector.env` (mode `0600`) now holds the OTLP token and the SQL password. Do **not** print either secret. If they want to see a value, point them at the file (`grep OTLP_AUTH_TOKEN "$ENV_FILE"`).

If the user supplied their own token or password, write those into the file instead of the generated ones, but keep the same `0600` discipline and make sure any custom password still meets the charset rules above.

---

## Step 3: Authenticate `clickhousectl` (separate terminal by default)

Check `clickhousectl` is on `PATH`. Run this presence check **on its own**, not chained to the installer: the `|| curl … | sh` form drags a harmless check into a compound command that sandboxes deny wholesale as an untrusted-code download.

```bash which clickhousectl ```

Only if that prints nothing, install it (the user may need to approve this explicitly, see Step 1):

```bash curl -fsSL https://clickhouse.com/cli | sh ```

Check authentication:

```bash clickhousectl cloud auth status ```

This skill needs **API key authentication**: OAuth is read-only and cannot create users or run write queries. If the `API key` row is not `Active`, the user must authenticate.

**Do not ask the user to paste their API key and secret into the chat.** Anything pasted into the conversation lives in the transcript and has to be rotated afterward. Instead, ask them to authenticate in a **separate terminal**, then tell you when they are done:

> I need a ClickHouse Cloud **Admin** API key to create the ingest user and verify the data. > Please don't paste it here. Instead: > > 1. In the [Cloud console](https://console.clickhouse.cloud), open **Organization → API keys > → New API key**, and give it the **Admin** role. (Developer-scoped keys can't provision the > per-service Query API endpoint that `cloud service query` uses.) > 2. In a **separate terminal**, run: > > ```bash > clickhousectl cloud auth login --api-key <key-id> --api-secret <key-secret> > ``` > > 3. Tell me when that's done and I'll re-check the auth status.

Poll until the API key row reports `Active`, then confirm with a real privileged call rather than trusting the status table alone. Use a **ref-agnostic** call here: `SERVICE_REF` may be a name, and `cloud service get` only accepts a UUID, so confirming with `get` would fail on a name for reasons unrelated to auth. `cloud service list` needs no ref and proves the API key works:

```bash clickhousectl cloud auth status clickhousectl cloud service list --json | jq -r '.[].name' ```

If the list returns your services, you are authenticated; continue. The actual name-or-UUID resolution of `S

--- name: clickstack-otel-collector description: Use when a user wants to wire an OpenTelemetry collector into a Managed ClickStack service on ClickHouse Cloud, either by deploying a new local collector (Docker run or Docker Compose) or by configuring their own existing collector, then send rich synthetic telemetry and verify it is visible in ClickStack. license: Apache-2.0 metadata: author: ClickHouse Inc version: "0.6.0" ---

# Set up an OpenTelemetry collector for Managed ClickStack

This skill wires an OpenTelemetry collector into a Managed ClickStack service running on ClickHouse Cloud, sends rich synthetic telemetry through it, and confirms the data is actually visible in ClickStack. It uses [`clickhousectl`](https://clickhouse.com/docs/interfaces/cli) for all cloud and SQL operations.

**Scope.** This skill supports two paths, chosen in Step 0:

1. **Deploy a new collector** locally. You can do this **two ways: individual `docker` commands, or a `docker compose` file** (recommended, fewer commands and one file to start/stop). Make the user aware of both up front and let them pick in Step 0; do not assume plain `docker`. Either way runs the ClickStack distribution of the collector, preconfigured for Managed ClickStack. 2. **Configure your own existing collector** by adding the ClickHouse exporter configuration. We give you the exact config to drop in; you reload your collector. Use this if you already run a collector in a **gateway** role.

A full Kubernetes deployment (Helm, secrets in K8s Secrets) is out of scope here; the config we generate in path 2 can be applied to a collector running anywhere.

The end state is:

- A dedicated `hyperdx_ingest` SQL user on the target service, with exactly the grants the collector needs (it creates the `otel.*` schema on first write). - A collector forwarding logs, traces, and metrics into the `otel` database on the service, either the new local ClickStack collector or your existing one. - Rich synthetic telemetry across several services, severities, span statuses, and metric types, so ClickStack's Search, Service Map, and dashboards have something real to show. - The service confirmed **awake**, and the user walked through the ClickStack onboarding in the Cloud console so they can actually *see* their data.

Secrets (the OTLP auth token and the SQL password) are generated locally, written **once** to a `0600` env file, and passed to Docker via `--env-file`. They are never pasted into the chat, never passed with `docker run -e`, and never echoed back after creation.

Follow these steps in order. Each step depends on state established by the previous one.

---

## Step 0: Choose your path

Ask the user two short questions before doing anything else, because they determine which later steps run.

**Question 1: Do you already have an OpenTelemetry collector running in a gateway role?**

- **No, set one up for me.** -> the **new-collector** path. Continue to Question 2. - **Yes, I have one.** -> the **existing-collector** path. Skip Question 2 (it does not apply), and in Step 6 you will configure their collector rather than deploy a new one.

**Question 2 (new-collector path only): Run the collector with individual Docker commands, or a Docker Compose file?**

- **Docker Compose (recommended).** Fewer commands, one file to start and stop, easiest to re-run. Best if `docker compose` is available. - **Individual Docker commands.** Use if Compose is not installed or you prefer explicit commands.

Record the answers as `COLLECTOR_PATH` (`new` or `existing`) and, for the new path, `DEPLOY_MODE` (`compose` or `run`). Refer back to them in Step 6 and Step 7.

---

## Step 1: Batch the permissions up front

Coding agents prompt for approval the first time they see each shell command. To avoid interrupting the user every few steps, ask them once, up front, to allowlist the command prefixes below (the "always allow for this project / session" option in their agent). There are no destructive operations and nothing targets anything outside this project or their ClickHouse Cloud service.

| Command prefix | Used for | Needed when | | --- | --- | --- | | `openssl rand …` | generate the OTLP token and SQL password | always | | `clickhousectl cloud …` | auth, resolve the service, run SQL via the Query API | always | | `jq …` | parse JSON from `clickhousectl` | always | | `docker …` / `docker compose …` | run/inspect the collector and the telemetry generator | new-collector path, and the optional telemetry check | | `curl …` | local health check against `localhost:13133` (and installing `clickhousectl` if missing) | new-collector path |

Tell the user, in your own words: *"If your agent supports it, choose 'always allow' for each of these the first time it asks. The whole run is read-only against your machine except for the collector container, and write operations against ClickHouse are limited to creating the ingest user and the `otel` schema."*

If the user is on the existing-collector path and does not want to run the optional telemetry check, you can drop `docker` and `curl` from the list.

**Two approvals are semantic, not prefix-based, so allowlisting won't pre-clear them.** Warn the user to expect these and approve them explicitly when they appear:

- The `clickhousectl` install in Step 3 uses `curl … | sh`, which many agent sandboxes flag as "downloading and running untrusted code" regardless of any `curl` allowlist rule. - The `CREATE USER` / `GRANT` in Step 5 may be flagged as "modifying shared production infrastructure," again independent of the `clickhousectl` prefix rule.

Neither is solved by the table above; they are one-time, intentional, and safe to approve.

Then continue.

---

## Step 2: Confirm the target service and lay down the secrets file

The user's prompt contains a service identifier, either a service ID (UUID) or a service name. Treat that value as `SERVICE_REF`.

Create a working directory and a **`0600` env file** that will hold all configuration and secrets for this run. The key names match exactly what the collector image reads, so this same file is passed straight to `docker run --env-file` (or referenced by Compose) in Step 6. Write it under a tight `umask` so the secret is never briefly world-readable:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}" mkdir -p "$WORKDIR" && chmod 700 "$WORKDIR" ENV_FILE="$WORKDIR/collector.env"

# Generate secrets WITHOUT printing them; write straight into a private file. ( umask 177 { echo "SERVICE_REF=$SERVICE_REF" echo "OTLP_AUTH_TOKEN=$(openssl rand -hex 32)" echo "CLICKHOUSE_USER=hyperdx_ingest" echo "CLICKHOUSE_PASSWORD=$(openssl rand -hex 24)Aa1-" echo "HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel" } > "$ENV_FILE" ) chmod 600 "$ENV_FILE" ls -l "$ENV_FILE" ```

Two things about these values matter and are easy to get wrong:

- **Key names are exact.** The collector reads `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_ENDPOINT`, and `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE`. Store the SQL password under `CLICKHOUSE_PASSWORD` (not a custom name); if it is missing, the collector starts with an **empty** password and dies with `code: 516, Authentication failed`. - **The password charset is constrained from three directions at once.** ClickHouse Cloud rejects passwords without at least one uppercase character and one special character, so a plain hex string fails at `CREATE USER`. At the same time, the collector's migration tool embeds the password in a connection URL, so `@`, `:`, `/`, `?`, `#`, and `%` corrupt it (symptom: `code: 516` at startup even though the password is "correct"). The recipe above is random hex (lowercase + digits) plus the suffix `Aa1-`, which adds the required uppercase, a digit, and a **URL-unreserved** special character (`-`). The OTLP token has no such rules (it is just a bearer token), so plain hex is fine for it.

The env file uses **bare `KEY=VALUE` lines with no quotes**: Docker's `--env-file` does not do shell parsing, so any quotes you add become part of the value.

On the **existing-collector path** the `OTLP_AUTH_TOKEN` is not used by your collector (auth on your receiver is your own setup); it is generated only so the same file works if you later switch to the local collector. The `CLICKHOUSE_*` values are still used: they go into the exporter config you add to your collector in Step 6.

**Every later step runs in a fresh shell, so `WORKDIR`, `ENV_FILE`, and any exported credentials do not persist, and `WORKDIR`/`ENV_FILE` are not stored inside the env file, so sourcing it can't recover them.** Begin each subsequent step's shell with this **standard preamble**, which re-derives the paths from the deterministic default, loads the saved credentials (Step 3), and loads the config:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env" [ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a ```

If you chose a non-default `WORKDIR`, set it explicitly at the top of every step (the `${WORKDIR:-…}` default only covers the standard location). Later steps refer to this as "the standard preamble".

**Confirm with the user** that `SERVICE_REF` is correct. Tell them the working directory and that `collector.env` (mode `0600`) now holds the OTLP token and the SQL password. Do **not** print either secret. If they want to see a value, point them at the file (`grep OTLP_AUTH_TOKEN "$ENV_FILE"`).

If the user supplied their own token or password, write those into the file instead of the generated ones, but keep the same `0600` discipline and make sure any custom password still meets the charset rules above.

---

## Step 3: Authenticate `clickhousectl` (separate terminal by default)

Check `clickhousectl` is on `PATH`. Run this presence check **on its own**, not chained to the installer: the `|| curl … | sh` form drags a harmless check into a compound command that sandboxes deny wholesale as an untrusted-code download.

```bash which clickhousectl ```

Only if that prints nothing, install it (the user may need to approve this explicitly, see Step 1):

```bash curl -fsSL https://clickhouse.com/cli | sh ```

Check authentication:

```bash clickhousectl cloud auth status ```

This skill needs **API key authentication**: OAuth is read-only and cannot create users or run write queries. If the `API key` row is not `Active`, the user must authenticate.

**Do not ask the user to paste their API key and secret into the chat.** Anything pasted into the conversation lives in the transcript and has to be rotated afterward. Instead, ask them to authenticate in a **separate terminal**, then tell you when they are done:

> I need a ClickHouse Cloud **Admin** API key to create the ingest user and verify the data. > Please don't paste it here. Instead: > > 1. In the [Cloud console](https://console.clickhouse.cloud), open **Organization → API keys > → New API key**, and give it the **Admin** role. (Developer-scoped keys can't provision the > per-service Query API endpoint that `cloud service query` uses.) > 2. In a **separate terminal**, run: > > ```bash > clickhousectl cloud auth login --api-key <key-id> --api-secret <key-secret> > ``` > > 3. Tell me when that's done and I'll re-check the auth status.

Poll until the API key row reports `Active`, then confirm with a real privileged call rather than trusting the status table alone. Use a **ref-agnostic** call here: `SERVICE_REF` may be a name, and `cloud service get` only accepts a UUID, so confirming with `get` would fail on a name for reasons unrelated to auth. `cloud service list` needs no ref and proves the API key works:

```bash clickhousectl cloud auth status clickhousectl cloud service list --json | jq -r '.[].name' ```

If the list returns your services, you are authenticated; continue. The actual name-or-UUID resolution of `S

--- name: clickstack-otel-collector description: Use when a user wants to wire an OpenTelemetry collector into a Managed ClickStack service on ClickHouse Cloud, either by deploying a new local collector (Docker run or Docker Compose) or by configuring their own existing collector, then send rich synthetic telemetry and verify it is visible in ClickStack. license: Apache-2.0 metadata: author: ClickHouse Inc version: "0.6.0" ---

# Set up an OpenTelemetry collector for Managed ClickStack

This skill wires an OpenTelemetry collector into a Managed ClickStack service running on ClickHouse Cloud, sends rich synthetic telemetry through it, and confirms the data is actually visible in ClickStack. It uses [`clickhousectl`](https://clickhouse.com/docs/interfaces/cli) for all cloud and SQL operations.

**Scope.** This skill supports two paths, chosen in Step 0:

1. **Deploy a new collector** locally. You can do this **two ways: individual `docker` commands, or a `docker compose` file** (recommended, fewer commands and one file to start/stop). Make the user aware of both up front and let them pick in Step 0; do not assume plain `docker`. Either way runs the ClickStack distribution of the collector, preconfigured for Managed ClickStack. 2. **Configure your own existing collector** by adding the ClickHouse exporter configuration. We give you the exact config to drop in; you reload your collector. Use this if you already run a collector in a **gateway** role.

A full Kubernetes deployment (Helm, secrets in K8s Secrets) is out of scope here; the config we generate in path 2 can be applied to a collector running anywhere.

The end state is:

- A dedicated `hyperdx_ingest` SQL user on the target service, with exactly the grants the collector needs (it creates the `otel.*` schema on first write). - A collector forwarding logs, traces, and metrics into the `otel` database on the service, either the new local ClickStack collector or your existing one. - Rich synthetic telemetry across several services, severities, span statuses, and metric types, so ClickStack's Search, Service Map, and dashboards have something real to show. - The service confirmed **awake**, and the user walked through the ClickStack onboarding in the Cloud console so they can actually *see* their data.

Secrets (the OTLP auth token and the SQL password) are generated locally, written **once** to a `0600` env file, and passed to Docker via `--env-file`. They are never pasted into the chat, never passed with `docker run -e`, and never echoed back after creation.

Follow these steps in order. Each step depends on state established by the previous one.

---

## Step 0: Choose your path

Ask the user two short questions before doing anything else, because they determine which later steps run.

**Question 1: Do you already have an OpenTelemetry collector running in a gateway role?**

- **No, set one up for me.** -> the **new-collector** path. Continue to Question 2. - **Yes, I have one.** -> the **existing-collector** path. Skip Question 2 (it does not apply), and in Step 6 you will configure their collector rather than deploy a new one.

**Question 2 (new-collector path only): Run the collector with individual Docker commands, or a Docker Compose file?**

- **Docker Compose (recommended).** Fewer commands, one file to start and stop, easiest to re-run. Best if `docker compose` is available. - **Individual Docker commands.** Use if Compose is not installed or you prefer explicit commands.

Record the answers as `COLLECTOR_PATH` (`new` or `existing`) and, for the new path, `DEPLOY_MODE` (`compose` or `run`). Refer back to them in Step 6 and Step 7.

---

## Step 1: Batch the permissions up front

Coding agents prompt for approval the first time they see each shell command. To avoid interrupting the user every few steps, ask them once, up front, to allowlist the command prefixes below (the "always allow for this project / session" option in their agent). There are no destructive operations and nothing targets anything outside this project or their ClickHouse Cloud service.

| Command prefix | Used for | Needed when | | --- | --- | --- | | `openssl rand …` | generate the OTLP token and SQL password | always | | `clickhousectl cloud …` | auth, resolve the service, run SQL via the Query API | always | | `jq …` | parse JSON from `clickhousectl` | always | | `docker …` / `docker compose …` | run/inspect the collector and the telemetry generator | new-collector path, and the optional telemetry check | | `curl …` | local health check against `localhost:13133` (and installing `clickhousectl` if missing) | new-collector path |

Tell the user, in your own words: *"If your agent supports it, choose 'always allow' for each of these the first time it asks. The whole run is read-only against your machine except for the collector container, and write operations against ClickHouse are limited to creating the ingest user and the `otel` schema."*

If the user is on the existing-collector path and does not want to run the optional telemetry check, you can drop `docker` and `curl` from the list.

**Two approvals are semantic, not prefix-based, so allowlisting won't pre-clear them.** Warn the user to expect these and approve them explicitly when they appear:

- The `clickhousectl` install in Step 3 uses `curl … | sh`, which many agent sandboxes flag as "downloading and running untrusted code" regardless of any `curl` allowlist rule. - The `CREATE USER` / `GRANT` in Step 5 may be flagged as "modifying shared production infrastructure," again independent of the `clickhousectl` prefix rule.

Neither is solved by the table above; they are one-time, intentional, and safe to approve.

Then continue.

---

## Step 2: Confirm the target service and lay down the secrets file

The user's prompt contains a service identifier, either a service ID (UUID) or a service name. Treat that value as `SERVICE_REF`.

Create a working directory and a **`0600` env file** that will hold all configuration and secrets for this run. The key names match exactly what the collector image reads, so this same file is passed straight to `docker run --env-file` (or referenced by Compose) in Step 6. Write it under a tight `umask` so the secret is never briefly world-readable:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}" mkdir -p "$WORKDIR" && chmod 700 "$WORKDIR" ENV_FILE="$WORKDIR/collector.env"

# Generate secrets WITHOUT printing them; write straight into a private file. ( umask 177 { echo "SERVICE_REF=$SERVICE_REF" echo "OTLP_AUTH_TOKEN=$(openssl rand -hex 32)" echo "CLICKHOUSE_USER=hyperdx_ingest" echo "CLICKHOUSE_PASSWORD=$(openssl rand -hex 24)Aa1-" echo "HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel" } > "$ENV_FILE" ) chmod 600 "$ENV_FILE" ls -l "$ENV_FILE" ```

Two things about these values matter and are easy to get wrong:

- **Key names are exact.** The collector reads `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_ENDPOINT`, and `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE`. Store the SQL password under `CLICKHOUSE_PASSWORD` (not a custom name); if it is missing, the collector starts with an **empty** password and dies with `code: 516, Authentication failed`. - **The password charset is constrained from three directions at once.** ClickHouse Cloud rejects passwords without at least one uppercase character and one special character, so a plain hex string fails at `CREATE USER`. At the same time, the collector's migration tool embeds the password in a connection URL, so `@`, `:`, `/`, `?`, `#`, and `%` corrupt it (symptom: `code: 516` at startup even though the password is "correct"). The recipe above is random hex (lowercase + digits) plus the suffix `Aa1-`, which adds the required uppercase, a digit, and a **URL-unreserved** special character (`-`). The OTLP token has no such rules (it is just a bearer token), so plain hex is fine for it.

The env file uses **bare `KEY=VALUE` lines with no quotes**: Docker's `--env-file` does not do shell parsing, so any quotes you add become part of the value.

On the **existing-collector path** the `OTLP_AUTH_TOKEN` is not used by your collector (auth on your receiver is your own setup); it is generated only so the same file works if you later switch to the local collector. The `CLICKHOUSE_*` values are still used: they go into the exporter config you add to your collector in Step 6.

**Every later step runs in a fresh shell, so `WORKDIR`, `ENV_FILE`, and any exported credentials do not persist, and `WORKDIR`/`ENV_FILE` are not stored inside the env file, so sourcing it can't recover them.** Begin each subsequent step's shell with this **standard preamble**, which re-derives the paths from the deterministic default, loads the saved credentials (Step 3), and loads the config:

```bash WORKDIR="${WORKDIR:-$HOME/clickstack-otel-collector}"; ENV_FILE="$WORKDIR/collector.env" [ -f "$WORKDIR/creds.env" ] && . "$WORKDIR/creds.env"; set -a; . "$ENV_FILE"; set +a ```

If you chose a non-default `WORKDIR`, set it explicitly at the top of every step (the `${WORKDIR:-…}` default only covers the standard location). Later steps refer to this as "the standard preamble".

**Confirm with the user** that `SERVICE_REF` is correct. Tell them the working directory and that `collector.env` (mode `0600`) now holds the OTLP token and the SQL password. Do **not** print either secret. If they want to see a value, point them at the file (`grep OTLP_AUTH_TOKEN "$ENV_FILE"`).

If the user supplied their own token or password, write those into the file instead of the generated ones, but keep the same `0600` discipline and make sure any custom password still meets the charset rules above.

---

## Step 3: Authenticate `clickhousectl` (separate terminal by default)

Check `clickhousectl` is on `PATH`. Run this presence check **on its own**, not chained to the installer: the `|| curl … | sh` form drags a harmless check into a compound command that sandboxes deny wholesale as an untrusted-code download.

```bash which clickhousectl ```

Only if that prints nothing, install it (the user may need to approve this explicitly, see Step 1):

```bash curl -fsSL https://clickhouse.com/cli | sh ```

Check authentication:

```bash clickhousectl cloud auth status ```

This skill needs **API key authentication**: OAuth is read-only and cannot create users or run write queries. If the `API key` row is not `Active`, the user must authenticate.

**Do not ask the user to paste their API key and secret into the chat.** Anything pasted into the conversation lives in the transcript and has to be rotated afterward. Instead, ask them to authenticate in a **separate terminal**, then tell you when they are done:

> I need a ClickHouse Cloud **Admin** API key to create the ingest user and verify the data. > Please don't paste it here. Instead: > > 1. In the [Cloud console](https://console.clickhouse.cloud), open **Organization → API keys > → New API key**, and give it the **Admin** role. (Developer-scoped keys can't provision the > per-service Query API endpoint that `cloud service query` uses.) > 2. In a **separate terminal**, run: > > ```bash > clickhousectl cloud auth login --api-key <key-id> --api-secret <key-secret> > ``` > > 3. Tell me when that's done and I'll re-check the auth status.

Poll until the API key row reports `Active`, then confirm with a real privileged call rather than trusting the status table alone. Use a **ref-agnostic** call here: `SERVICE_REF` may be a name, and `cloud service get` only accepts a UUID, so confirming with `get` would fail on a name for reasons unrelated to auth. `cloud service list` needs no ref and proves the API key works:

```bash clickhousectl cloud auth status clickhousectl cloud service list --json | jq -r '.[].name' ```

If the list returns your services, you are authenticated; continue. The actual name-or-UUID resolution of `S