--- name: karpathy-wiki-ingest description: | For a detached wiki ingester only. The main agent never loads this. Defines orientation protocol, page format, role guardrail, validator contract, manifest protocol, and deterministic completion contract. Loaded by the provider-aware runtime worker. ---

# karpathy-wiki ingest (for detached runtime ingester only)

You are a detached wiki ingester invoked by the provider-aware runtime worker. Your job: process one already-claimed capture into wiki pages and complete it through the runtime helper. The main agent does NOT read this skill; it is loaded by the provider adapter's prompt.

Perform this ingest yourself. You must not launch or delegate the work to another model or agentic CLI. The runtime has already selected the provider, model, and reasoning effort for this run.

## Deep orientation

Before any wiki write, run this 9-step orientation protocol. The goal: your view of the wiki is shaped by the actual page content, not by titles alone.

### Steps 1-3: read the wiki's current state

1. Read `<wiki>/schema.md` — current categories, taxonomy, thresholds. 2. Read `<wiki>/index.md` (or `<wiki>/<category>/_index.md` per category) — what pages exist with one-line summaries. 3. Read the last ~10 entries of `<wiki>/log.md` — recent activity.

### Steps 4-7: pick and read 0-7 candidate pages

4. **Extract candidate signals from the capture.** From the body and frontmatter, gather: - Words and phrases from the capture title (lowercase, split on non-alphanumerics). - Frontmatter `tags:` list (if any). - For `chat-only` and `chat-attached` captures: meaningful nouns and proper-noun phrases from the body. Use judgment — you're an LLM, not a regex; pick names, identifiers, technical terms, version numbers. - For `raw-direct` captures: filename basename + the file's first 200 lines.

5. **Score candidates against the index.** A page is a candidate if ANY extracted signal: - Substring-matches its title (case-insensitive), OR - Matches a tag (exact, case-insensitive), OR - Appears in its `_index.md` one-line summary.

This is a deterministic substring + tag match — no embeddings, no semantic similarity. See "Why no embeddings / vector search" below for the rationale.

6. **Pick up to 7 candidates** ordered by: - Signal-match count (descending — more matches = stronger candidate). - Title length (ascending — shorter titles rank higher for the same match count; they tend to be more general / canonical). - Tie-break: alphabetical by title.

7. **Read all picked candidates in full.** Zero is a valid count for a small or fresh wiki — see "Cold start" below.

### Steps 8-9: decide and report observations

8. **Decide**: create new page, augment an existing page, or no-op (the capture's content is already covered by an existing page). Write your decision rationale into the commit message later.

9. **Issue reporting (during steps 5-7).** While reading the index and the candidate pages, observe issues. Append each as one JSONL line to `<wiki>/.ingest-issues.jsonl` via `bash "${WIKI_PLUGIN_ROOT}/scripts/wiki-issue-log.sh"`. Do NOT fix issues inline; report only — `wiki doctor` consumes the log later.

Example invocation:

```bash bash "${WIKI_PLUGIN_ROOT}/scripts/wiki-issue-log.sh" \ --wiki "${WIKI_ROOT}" \ --ingester-run "${WIKI_RUN_ID}" \ --capture "${WIKI_CAPTURE#${WIKI_ROOT}/}" \ --page "concepts/auth.md" \ --type broken-cross-link \ --severity warn \ --detail "Links to /concepts/legacy.md which does not exist." \ --suggested-action "Remove link or create stub" ```

Issue types to watch for (the `--type` enum in `wiki-issue-log.sh`): - **broken-cross-link**: page links to `/path/foo.md` but that file does not exist. - **contradiction**: page makes a claim that contradicts another page you're reading or the capture itself. - **schema-drift**: page has `type: concept` (singular) but directory is `concepts/`; page lacks required frontmatter; page uses retired categories. - **stale-claim**: page says "as of 2025-12 library X is at version Y" and the capture or another source clearly indicates a newer version. - **tag-drift**: same concept tagged two different ways across pages. - **quality-concern**: page's `quality.overall < 3.0` was rated by ingester (not human). - **orphan**: page exists but is not linked from any `_index.md` or other page. - **other**: anything else worth noting.

Severity: - **error**: blocks ingestion (rare — only if reading the schema or a candidate page fails outright). - **warn**: default. Issue noted; ingestion proceeds. - **info**: observational; usually for tag-drift or quality-concern.

Each issue gets one JSONL line, ≤ 4096 bytes total. Over-length `--detail` is auto-truncated by the helper.

### Cold start: wikis with ≤ 7 pages

When the candidate list from step 6 is empty or near-empty (the index has fewer than 8 pages total), step 7 reads few or no pages. The role guardrail in `<wiki>/.wiki-config` becomes the PRIMARY lens, not a backup tripwire:

- **`role: project` or `role: project-pointer`**: write specifics about THIS codebase / instance / situation. Document the symptom, the pinpoint, what code path triggered it. Do NOT generalize. - **`role: main`**: write general patterns reusable across projects. Do NOT name specific apps or instances; abstract them.

In the cold-start state the index has insufficient gravity to shape the page; the role hint carries the lens. Defer cross-linking to a future ingester run when more pages exist.

### Why no embeddings / vector search

The substring + tag match in step 5 is deterministic, scriptable, testable, and cheap. Embedding-based candidate selection is the "smart" upgrade but explicitly out of scope per CLAUDE.md ("do not add: vector search — defer until genuine scaling pain"). The substring/tag approach degrades gracefully — at large scale it hits more candidates than 7, but the ranking still produces a usable top-7. When that breaks, vector search becomes worth its complexity; not before.

### Cost

3-7 extra file reads per ingestion (zero on cold-start wikis). Each page is typically 5-20 KB. The ingester is already reading the capture and the schema; this is the same order of magnitude. No measurable spawn-time impact.

## Run record (per ingestion)

`WIKI_RUN_ID` is assigned before you start. Do not generate or replace it. The runtime wrapper owns the `started`, retry, failure, and `completed` records in `<wiki>/.ingest-runs.jsonl`; it also owns the heartbeat on `WIKI_CAPTURE` and the process-slot lease. Do not write run-history records yourself.

Use `WIKI_RUN_ID` only when another deterministic helper needs to tie an observation to the current run, such as `wiki-issue-log.sh`. A successful ingest is closed by the completion helper described in step 10. The wrapper writes `completed` only after it verifies both the archive and the provider's zero exit.

## Role guardrail

Read `<wiki>/.wiki-config` to determine the wiki's role:

- `role: project` or `role: project-pointer` → you are writing for a PROJECT wiki. Document specifics: how this app handles X, where bug Y lived, what we decided for this codebase. Do NOT generalize. - `role: main` → you are writing for the MAIN wiki. Extract general patterns reusable across projects. Do NOT name specific apps or instances.

The role field is the primary lens during cold start (≤ 7 pages in the wiki) and a sanity tripwire in mature wikis (the index pull is the primary lens once enough pages exist). See "Deep orientation — Cold start" above.

If the index pulls strongly toward instance-style or pattern-style writing (you read 5+ existing pages of one shape), trust the pull. The role hint then becomes a tripwire — *"if you find yourself generalizing in a project wiki / specifying in a main wiki, stop."*

## Capture format

See `<plugin>/skills/karpathy-wiki-capture/references/capture-schema.md` for the canonical capture frontmatter contract — `capture_kind` enum, body floors, evidence rules, legacy backward-compat. Do not duplicate that schema here.

## Page format

See `references/page-conventions.md` for the canonical page frontmatter and cross-link conventions.

All generated page metadata timestamps (`created`, `updated`, `rated_at`) must be real UTC values. In shell, generate them with `date -u +%Y-%m-%dT%H:%M:%SZ`. Never append `Z` to local wall-clock output.

## Body-sufficiency check (first thing — reject if too thin)

Before any other work, measure the capture body size in bytes (content AFTER the closing `---` of frontmatter). Apply the per-`capture_kind` floor:

- `raw-direct` → no floor (body is auto-generated boilerplate). - `chat-attached` → 1000 bytes. - `chat-only` → 1500 bytes.

Legacy captures (no `capture_kind`): apply backward-compat per `<plugin>/skills/karpathy-wiki-capture/references/capture-schema.md`.

If body is BELOW its floor:

1. Add `needs_more_detail: true` to the capture's frontmatter. 2. Add `needs_more_detail_reason: "body is <N> bytes; floor for capture_kind=<K> is <F> bytes"`. 3. Rename `.md.processing` → `.md` so the next session-start drain preserves it for the main agent to expand. The dispatcher skips marked captures until the main agent removes both deferral fields. 4. Append to `log.md`: `## [<timestamp>] reject | <capture-basename> — body <N>b below <F>b floor`. 5. Exit 0. Do NOT write pages. Do NOT commit.

A thin-capture rejection is a feature, not a failure.

## Ingester steps

1. **The capture is already claimed for you.** Read `${WIKI_CAPTURE}` (it's a `.md.processing` file). If for some reason `${WIKI_CAPTURE}` is unset or missing, call `wiki_capture_claim "${WIKI_ROOT}"` to grab any pending capture as fallback.

2. **Read the orientation files**: schema.md, index.md, last 10 entries of log.md (per the Orientation section above).

3. **Read the capture body.**

4. **Copy evidence with write-staging discipline (v2.4):**

Atomic write to `raw/` requires the staging dance (skip steps 1–2 and 7 if `evidence_type` is `conversation` AND the capture has no real path):

1. Copy the evidence file to `<wiki>/.raw-staging/<basename>` (NOT directly to `raw/`). 2. Compute the new sha256. 3. Acquire `<wiki>/.locks/manifest.lock` via `bash "${WIKI_PLUGIN_ROOT}/scripts/wiki-manifest-lock.sh" ...` for the manifest write + rename block. 4. Re-read the manifest under the lock. If `raw/<basename>` already exists with the same sha256, this is a duplicate — skip (apply the sha256 short-circuit below). 5. Update the manifest entry for `raw/<basename>` (origin, sha, copied_at, last_ingested, referenced_by). 6. Write the manifest atomically (`.manifest.json.tmp` + `os.rename`) — `wiki-manifest.py build` already does this. 7. Atomically rename `<wiki>/.raw-staging/<basename>` → `<wiki>/raw/<basename>` (POSIX rename is atomic on the same filesystem). 8. Release the lock. 9. If the source file is in `<wiki>/inbox/<basename>` (raw-direct via inbox queue), `rm` it.

Crash recovery: if the ingester crashes between steps 1 and 7, a file lingers in `.raw-staging/`. The SessionStart recovery scan SKIPS `.raw-staging/` (it's a reserved dot-prefixed directory). Future cleanup is `wiki doctor`'s responsibility.

- In ALL cases (including `chat-only` / legacy `conversation`): write or update the manifest entry for the raw file in `<wiki>/.manifest.json`: ```json { "raw/<basename>": { "sha256": "<sha256 of raw file>", "origin": "<exact value of capture's `evidence` field>", "copied_at": "<iso-8601 utc, preserve if already present>", "last_ingested": "<iso-8601 utc, now>", "referenced_by": ["<pages added to this list below>"]