sglang-xpu-bench
Benchmark a **running SGLang-XPU server** on an Intel GPU using `sglang.bench_serving`. Measures TTFT, TPOT, ITL, end-to-end latency, and throughput against the OpenAI-compatible endpoint. Use after sglang-xpu-run. Not for vLLM servers (use vllm-xpu-bench) or no-server PyTorch (use torch-xpu-bench).
intel · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add intel/gpu-ai-skills --skill sglang-xpu-benchAbout this skill
name: sglang-xpu-bench
description: Benchmark a running SGLang-XPU server on an Intel GPU using sglang.bench_serving. Measures TTFT, TPOT, ITL, end-to-end latency, and throughput against the OpenAI-compatible endpoint. Use after sglang-xpu-run. Not for vLLM servers (use vllm-xpu-bench) or no-server PyTorch (use torch-xpu-bench).
sglang-xpu-bench
sglang.bench_serving is SGLang's online benchmark client (the
counterpart to vllm bench serve). Speaks the OpenAI-compatible API
exposed by sglang-xpu-run.
Step 0 — verify SGLang server is running on Intel XPU
REQUIRED: Before benchmarking, you must confirm a SGLang server is running, identify its port, and verify it's using Intel XPU (not CPU fallback, not NVIDIA). This prevents benchmarking a server that silently fell back to CPU.
Run the checks below in a single shell session (later blocks reuse
$SGLANG_PID, $SGLANG_PORT, $MODEL from earlier ones).
Find the server process and its container:
# 1. Check if SGLang is running and find its container
SGLANG_PID=$(ps aux | grep -iE 'sglang|launch_server' | grep -v grep | awk 'NR==1 {print $2}')
if [ -z "$SGLANG_PID" ]; then
echo "❌ No SGLang server found running. Start one with sglang-xpu-run."
exit 1
fi
echo "✓ SGLang server found (PID: $SGLANG_PID)"
CONTAINER_NAME=$(docker ps --format '{{.Names}}' 2>/dev/null | while read name; do
if docker top "$name" -o pid 2>/dev/null | awk 'NR>1' | grep -qxF "$SGLANG_PID"; then echo "$name"; break; fi
done)
if [ -z "$CONTAINER_NAME" ]; then
echo "❌ No container matched PID $SGLANG_PID. This skill runs the bench client"
echo " via 'docker exec' because the sglang package is only installed inside"
echo " the server container. A host-only SGLang install is not supported here."
exit 1
fi
echo "✓ SGLang container: $CONTAINER_NAME"
Read the launch args once — they give you both the container-internal port
and the --device flag. The bench client runs via docker exec inside the
container, so the port must be the one SGLang binds inside the container
(host ss can't see the container-namespaced socket). The --device xpu check
is REQUIRED — it catches a server that silently fell back to CPU:
# 2. Read port + device from the launch args in one pass (REQUIRED)
ARGS=$(ps -p "$SGLANG_PID" -o args=)
SGLANG_PORT=$(echo "$ARGS" | awk '{for(i=1;i<=NF;i++) if($i=="--port" && i<NF) print $(i+1)}' | head -1)
SGLANG_PORT=${SGLANG_PORT:-30000} # sglang default
DEVICE_ARG=$(echo "$ARGS" | awk '{for(i=1;i<=NF;i++) if($i=="--device" && i<NF) print $(i+1)}' | head -1)
echo "DEVICE_ARG=${DEVICE_ARG:-none}"
if [ "$DEVICE_ARG" = "cuda" ]; then
echo "❌ Server is on NVIDIA CUDA, not Intel XPU."; exit 1
elif [ "$DEVICE_ARG" = "xpu" ]; then
echo "✓ SGLang server has --device xpu"
else
echo "⚠️ Could not confirm --device xpu (got: '${DEVICE_ARG:-none}'); relying on XPU memory check."
fi
# 3. Verify reachable from inside the container and read model name
if ! docker exec "$CONTAINER_NAME" curl -s http://127.0.0.1:$SGLANG_PORT/v1/models >/dev/null 2>&1; then
echo "❌ SGLang server not responding on container port $SGLANG_PORT."
exit 1
fi
MODEL=$(docker exec "$CONTAINER_NAME" curl -s http://127.0.0.1:$SGLANG_PORT/v1/models | python3 -c "import sys,json; print(json.load(sys.stdin)['data'][0]['id'])")
echo "✓ Responding on container port $SGLANG_PORT — model: $MODEL"
Reject NVIDIA GPU usage, then confirm XPU memory is in use (rules out CPU fallback):
# 4a. Reject NVIDIA GPU usage — scan only the compute-app PID list and match
# the whole line, so a short PID can't collide with memory/temp/other numbers
# elsewhere in nvidia-smi's output.
if command -v nvidia-smi >/dev/null 2>&1 && \
nvidia-smi --query-compute-apps=pid --format=csv,noheader 2>/dev/null \
| tr -d ' ' | grep -qxF "$SGLANG_PID"; then
echo "❌ Server is running on NVIDIA GPU, not Intel XPU."; exit 1
fi
# 4b. Verify Intel XPU memory usage — -m 18 = "GPU Memory Used (MiB)", CSV output
GPU_COUNT=$(xpu-smi discovery 2>/dev/null | grep -cE "^\| +[0-9]|Device [0-9]+:")
[ "${GPU_COUNT:-0}" -gt 0 ] || GPU_COUNT=1
XPU_IN_USE=0
for id in $(seq 0 $((GPU_COUNT - 1))); do
MEM_USED=$(xpu-smi dump -d "$id" -m 18 -i 1 -n 1 2>/dev/null | awk -F',' 'NR==2 {gsub(/ /,"",$NF); print int($NF)}')
echo " Device $id: ${MEM_USED:-0} MiB"
[ "${MEM_USED:-0}" -gt 500 ] && XPU_IN_USE=1
done
if [ "$XPU_IN_USE" -eq 0 ]; then
echo "❌ XPU memory near zero — server may have fallen back to CPU. Restart with --device xpu."
exit 1
fi
echo "✓ SGLang is confirmed running on Intel XPU"
Everything needed for the benchmark is now confirmed:
# 5. Summary — CONTAINER_NAME, SGLANG_PORT, MODEL were set above
echo "=== Ready to benchmark === Port: $SGLANG_PORT Model: $MODEL Container: $CONTAINER_NAME"
If no SGLang server is found, if it's running on NVIDIA instead of Intel XPU, or if it's fallen back to CPU, the checks exit with guidance.
Metrics
- TTFT — wall time to first generated token.
- TPOT — mean per-token time after the first.
- ITL — per-token inter-arrival; percentiles meaningful for latency SLAs.
- E2EL — wall time of one request.
- Throughput —
output_tokens / wall_seconds.
Prerequisites
A running SGLang-XPU server (per sglang-xpu-run). Step 0 above will verify the server is running, identify its port, and confirm it's using Intel XPU.
Note on ALL_PROXY: if the host has ALL_PROXY=socks://... set, the
bench client's HTTP requests may be routed through the SOCKS proxy.
Unset it before benching a local server:
unset ALL_PROXY all_proxy
Online bench
Important: The benchmark client must run inside the same container
where the SGLang server is running. The sglang package is only installed
in the container environment, not on the host.
Always use docker exec (not nsenter) to run benchmarks. The nsenter
appro