quark-torch-llm-ptq
Runs an end-to-end AMD Quark post-training quantization workflow for PyTorch / Hugging Face LLMs: inspect a Hub or local model, choose a quantization plan, create reproducible artifacts, request execution approval, and produce a quantized model. Applies to Llama, Qwen, Mistral, and similar transformer LLM requests involving FP8, INT4, or another Quark scheme. Does not handle .onnx model inputs or ONNX PTQ.
amd · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add amd/skills --skill quark-torch-llm-ptq技能说明
name: quark-torch-llm-ptq description: > Runs an end-to-end AMD Quark post-training quantization workflow for PyTorch / Hugging Face LLMs: inspect a Hub or local model, choose a quantization plan, create reproducible artifacts, request execution approval, and produce a quantized model. Applies to Llama, Qwen, Mistral, and similar transformer LLM requests involving FP8, INT4, or another Quark scheme. Does not handle .onnx model inputs or ONNX PTQ.
Quark Torch PTQ
Purpose
Take a PyTorch / Hugging Face LLM from model identification through confirmed AMD Quark PTQ. Perform intake, planning, manifest generation, execution, and output verification as one self-contained workflow.
This skill stops at the quantized model. It does not accept .onnx model input,
train or fine-tune a model, or modify Quark package/source files.
Prerequisites
- Python 3.11 to 3.13, accelerator-matched PyTorch 2.2 or newer,
amd-quark[cli], anddatasets. - The required ROCm version and GPU architecture depend on the PyTorch build and selected quantization scheme. Record the actual runtime, GPU architecture (
gfx...fromgcnArchNameon AMD), host kernel, and driver, and verify support for the confirmed plan. - No container image is required. If running in a container, record its image name or digest when available and verify GPU device access.
- Inspect and preserve
HIP_VISIBLE_DEVICES,CUDA_VISIBLE_DEVICES,HSA_OVERRIDE_GFX_VERSION,PYTORCH_ROCM_ARCH, andPYTORCH_HIP_ALLOC_CONF. Include any required changes to device visibility, architecture, or memory allocation in the confirmed plan.
Inputs
- Model source: a Hugging Face repository ID or local model directory.
- Output directory.
- Quantization intent: requested precision/scheme, target hardware, accuracy priority, and optional calibration settings.
- Optional environment facts: Python, PyTorch, accelerator, available memory,
installed
amd-quark, and Transformers versions.
Do not require pre-existing workflow artifacts. Create all artifacts in the user's working directory by following this skill's local references:
references/model-intake.mdreferences/quant-plan.mdreferences/environment.mdreferences/troubleshooting.md
Outputs
Produce these three artifacts before or during execution:
model_analysis.json, validated againstreferences/contracts/model_analysis.schema.json.quant_plan.json, validated againstreferences/contracts/quant_plan.schema.json.run_manifest.yaml, validated againstreferences/contracts/run_manifest.schema.json.
The quantized model and its configuration/tokenizer files are written under the confirmed output directory. Record actual files and the final status in the manifest.
Interaction Flow
Always complete the following four steps in order. Show concrete facts, artifacts, and commands. Stop at every checkpoint and wait for the user.
Step 1 — Model intake
- Confirm whether the model source is local or remote. For a local source,
resolve it to an absolute path and verify that the directory and
config.jsonexist. For a remote source, preserve the repository ID. - Follow
references/model-intake.md. Read configuration only; do not load model weights during intake. - Determine
model_type, architecture/loading hints, hidden-layer facts, multimodal or MoE signals, default exclusions, compatibility risks, and a defensible estimate of quantizable linear layers. - Write schema-valid
model_analysis.jsonand show its summary.
Checkpoint 1 — Confirm the model analysis
Ask the user to confirm or correct the model analysis.
Do not plan quantization until the user confirms.
Step 2 — Quantization plan
- Follow
references/quant-plan.mdand use the confirmed analysis plus the user's priorities. - Select and explain the global scheme, optional KV-cache scheme, per-pattern overrides, exclusions, algorithms, and calibration data.
- Treat scheme, algorithm, and model-template support as version-dependent.
Use the current installed Quark API or
quark-cli torch-llm-ptq --helpwhen a choice needs verification; do not rely on historical list sizes. - Show a decision table, write schema-valid
quant_plan.json, and setrequires_confirmation: trueuntil approved.
Checkpoint 2 — Confirm the quantization plan
Ask the user to confirm or adjust the complete plan.
After confirmation, update requires_confirmation to false. If the user
changes a decision, rewrite and revalidate the plan before continuing.
Step 3 — Manifest and execution confirmation
Use the public quark-cli torch-llm-ptq command installed by
amd-quark[cli]. Do not import its implementation module directly or locate,
copy, generate, or patch another PTQ runner.
Build an argument-array-safe command equivalent to:
quark-cli torch-llm-ptq \
--model_dir "<MODEL_OR_ABSOLUTE_LOCAL_PATH>" \
--output_dir "<ABSOLUTE_OUTPUT_PATH>" \
--quant_scheme "<SCHEME>" \
--num_calib_data "<N>" \
--seq_len "<LENGTH>" \
--device cuda \
--no_trust_remote_code
Add only confirmed options:
--dataset <NAME>and--batch_size <N>when the plan changed them from the CLI defaults.--kv_cache_dtype <SCHEME>for confirmed KV-cache quantization.- One
--layer_quant_scheme <PATTERN> <SCHEME>per override. --quant_algo <comma-separated-list>when required.--exclude_layers <patterns...>only when overriding template defaults.--multi_devicewhen its constraints are understood.--no_trust_remote_codeunless the user explicitly accepts executing remote model code. The CLI trusts remote code when this flag is omitted.