serving-llms-on-epyc
Serves an LLM on a supported AMD EPYC server CPU using vLLM with zentorch, in Docker, Podman, or conda. Use for "vLLM on CPU", "zentorch serving", or an EPYC CPU endpoint, including on a host that also has AMD Instinct GPUs. Detects the EPYC generation, validates the runtime, checks model support and RAM fit, sizes threads/KV/NUMA, confirms the plan, launches, and verifies the endpoint. Runs one instance on one socket and its memory. Reports and stops on failure; does not retry or debug. Use serving-llms-on-instinct when the endpoint should run on a GPU. Excludes multi-node, EPYC 4000, and pre-Zen4 EPYC without AVX-512.
amd · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add amd/skills --skill serving-llms-on-epyc技能说明
name: serving-llms-on-epyc description: >- Serves an LLM on a supported AMD EPYC server CPU using vLLM with zentorch, in Docker, Podman, or conda. Use for "vLLM on CPU", "zentorch serving", or an EPYC CPU endpoint, including on a host that also has AMD Instinct GPUs. Detects the EPYC generation, validates the runtime, checks model support and RAM fit, sizes threads/KV/NUMA, confirms the plan, launches, and verifies the endpoint. Runs one instance on one socket and its memory. Reports and stops on failure; does not retry or debug. Use serving-llms-on-instinct when the endpoint should run on a GPU. Excludes multi-node, EPYC 4000, and pre-Zen4 EPYC without AVX-512. allowed-tools: Bash, Read
Serving LLMs on AMD EPYC™ (vLLM + zentorch, CPU)
Bring up a single vLLM OpenAI endpoint on an AMD EPYC™ host with the zentorch CPU backend, sized to the hardware. Container-first (Docker or Podman); conda/host is the fallback. An installed AMD Instinct GPU does not disqualify the host: select this skill when the endpoint itself should run on the EPYC CPU.
This is single-socket serving: one instance pinned to one socket and its memory (vLLM scales poorly across sockets, so we do not span them). On a dual-socket host it runs on a single socket; the multi-socket answer is multiple instances (one per socket), which is out of scope for this single-instance recipe.
Hard rule for this skill: on any failure, report the cause + logs and STOP. Do not retry, do not debug. (Debugging is a separate workflow.)
The agent does the serve flow itself -- pull, configure, launch, poll --
using the runtime validate.py reports. Never hand the user per-serve commands.
Like serving-llms-on-instinct, an accessible container runtime is a one-time
prerequisite: if validate.py finds none, report its one-time fix (make
docker accessible / install podman / provide a conda env) and stop. Do not
attempt sudo or privilege escalation.
Data file
Read data/epyc.json directly. It holds the container image, mandatory CPU run
flags, supported precision, the model-support policy, the default model, and the
verified throughput-flag gotcha. Its vllm_version and image tag are one
validated default stack; keep them aligned and do not hardcode either from memory.
Step 1: Detect the CPU
python3 scripts/detect.py # add --host user@box for a remote host
Returns cpu_model, is_amd_epyc, epyc_generation
(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005),
zen_arch, is_supported_epyc, avx512, logical_cores, physical_cores,
sockets, numa_nodes, memory_gb.
Three hard gates -- stop if any fails:
is_amd_epycisfalse-> stop: this skill targets AMD EPYC. (Other x86 may work but is unsupported here.)is_supported_epycisfalse-> stop: this recipe supports only the AMD EPYC 9000 series for now -- Genoa (9004), Turin (9005), and Venice (9006). Other EPYC (Bergamo, Siena, EPYC 4004/4005, pre-Zen4) may even expose AVX-512, but ISA compatibility alone does not make them supported targets for this skill; stop.avx512isfalse-> stop: the zentorch CPU path requires AVX-512, i.e. Zen4+ on the supported 9000-series parts above. Pre-Zen4 EPYC (Naples / Rome / Milan) is not supported -- say so and stop rather than launching into a load-time failure.
Carry epyc_generation / avx512 through the later phases -- e.g. Venice packs up
to 256 cores/socket, which the thread-binding in Step 5 sizes from.
Step 2: Validate the runtime and environment
python3 scripts/validate.py --image <image from data/epyc.json> --generation <epyc_generation from detect>
Returns ready, requires_confirmation, runtime (docker, podman, or null),
runtime_detail, conda_path_available, stack, compatibility, ram_gb, and
errors/warnings/advisories. Pick the path:
runtimeisdockerorpodman-> container path (Step 6), used verbatim.runtimenull butconda_path_available: true-> conda/host path.runtimenull and no conda ->readyis false. Report the one-time onboardingfix(make docker accessible / install podman / conda env) and stop.
Do not proceed if ready is false.
Stack-compatibility gate. validate.py probes the selected runtime for its
exact vllm/zentorch/torch versions and the active vLLM platform, then sets
compatibility.status:
proceed-> the stack is the validated default (or a validated family on a Zen platform); continue.blocked-> a stock CPU platform is active, so zentorch acceleration is not on (error). Reportcompatibility.messageand stop.confirmation_required(requires_confirmation: true) -> Venice on a vLLM other than the pinned default. This recipe has not been validated on Venice with that version. Surfacecompatibility.message, recommend the pinnedvllm_versionimage fromdata/epyc.json, and stop for an explicit user go/no-go before launching. On the pinned default vLLM, Venice proceeds with no warning.
The gate only runs once the image is local. If validate.py reports the image is
not pulled, pull it (or let Step 6 pull it) and re-run validate.py so the
gate probes the real stack rather than only the tag.
Step 3: Resolve and validate the model
If the user named no model, use default_model from data/epyc.json
(Qwen/Qwen3-0.6B -- ungated, tiny, fast first success). Otherwise use theirs.
Check that vLLM actually supports the model (do not blanket-block multimodal).
Pass the vLLM version the model will actually run on: use stack.vllm from
validate.py when it was probed (the conda env may differ from the pin), else the
vllm_version from data/epyc.json.
python3 scripts/check_model.py --model-id <model> --revision <rev or main> --vllm-version <stack.vllm from validate, else vllm_version from data/epyc.json>
- Exit 0 = vLLM serves