Remote Gpu Trainer
A SKILL.md-based agent skill covering the full deep learning experiment lifecycle—running jobs on owned or rented GPUs, verifying results, and delivering reproducible figures and tables.
Hanyuyuan6 · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add Hanyuyuan6/remote-gpu-trainer技能说明
name: "remote-gpu-trainer" description: "Use when running, debugging, verifying, or delivering a deep-learning experiment on an owned or rented GPU, especially AutoDL or a remote SSH host; also use for Windows + Clash/Mihomo high-port SSH banner timeouts, fake-IP, or TUN routing interference. Covers launch, checkpoint/resume, detached monitoring, OOM/NaN/convergence/data-loader failures, multi-GPU hangs, ablations, result verification, pull/teardown safety, and canonical export closure. Routes durable replicas to mirror-research-artifacts. Triggers: owned/rented GPU, AutoDL, SSH, Windows Clash/Mihomo, banner timeout, fake-IP, TUN, train/debug/verify/pull/export, 远程GPU训练/租卡, 断点续训, 消融复现, checkpoint 拉回." license: MIT metadata: last-model-review: "2026-08-10 AutoDL canonical export and generic mirror handoff; preserves 2026-07 lifecycle review findings (day completed from aca1c467, which rewrote the description and added the Windows/Clash SSH gate — the bare 2026-08 form is unparseable to the staleness hook)"
remote-gpu-trainer — the DL Experiment Lifecycle
Overview
One skill for the whole arc of a DL experiment: RUN → VERIFY → DELIVER.
- RUN — get a long GPU job to start, survive, and finish, then get the result off the box. On a
machine you own there is no meter; on a rented box the core insight is that you are a
short-term tenant on someone else's machine — so the job is to detach the work, make the result
outlive the instance, and stop the meter safely, not to provision a cluster. Platform-specific at the
edges (one
profiles/<platform>.mdowns every path, proxy, billing verb, and spot rule), invariant at the core. - Remote ownership boundary — this skill is the compute/control layer: it binds inputs, runs and
verifies compute, and closes one run into
export/<run-id>. Long-term project organization belongs toresearch-artifact-hygiene; any durable/local/cloud mirror begins only at a validated export and belongs to the genericmirror-research-artifactsskill. Never mirror a mutableactive/tree. - VERIFY — is this number a bug, a real effect, or noise? A surprising result is a hypothesis, not a fact to report. Platform-agnostic.
- DELIVER — organize the result so every shipped number/figure/table is a deterministic function of one immutable evidence layer; provenance and cross-document consistency are locked by mechanism, not by a human remembering to update three documents. Platform-agnostic.
Two stances run through VERIFY and DELIVER: user sovereignty (the science — seed count, which samples,
whether an aux channel exists — is the user's call; the skill organizes and discloses a tradeoff once,
then stops nagging) and audit → disclose, not enforce (the skill is an honest auditor, not a gate
guard — an integrity issue must surface with the conclusion it affects, but the skill never blocks the
user from shipping). Mantra: "disclose it, or don't claim it."
Route first
- RUN — own the box or rent it?
- Local (a workstation/laptop you own, no meter) →
references/run-local/andprofiles/local.md. - Rented / remote (any metered or shared box you don't own) →
references/run-remote/, and pick yourprofiles/<platform>.mdFIRST (it owns every path/verb/proxy the phases delegate to).
- Local (a workstation/laptop you own, no meter) →
- Then ALWAYS → VERIFY the result (
references/verifying/). A green run is not a real number. - When publication synthesis is requested, optionally consult
references/delivering/for legacy/non-canonical publication synthesis guidance. It never defines the artifact layout: canonical runs, hardware evidence, trust records, figures and tables belong toresearch-artifact-hygiene.
Already debugging a model that won't converge / OOMs / hangs / NaNs, regardless of where it runs? Jump straight to
references/training/(the 8-file debug layer), then come back to VERIFY before you report.
Operating principles (the spine)
The load-bearing invariants. One line each; the full cross-platform set (10 invariants for the remote
lifecycle) is in references/run-remote/principles.md — read it before Phase 0 of a remote run.
- Checkpoint-to-durable + idempotent resume is the universal spine. File-checkpoint to the durable location + unconditional load-latest-on-startup is the one mechanism that survives an SSH drop, a Slurm walltime kill, a K8s reschedule, a spot preemption, a Colab disconnect. The detach primitive (tmux / sbatch / Job) is the swappable plug; this is the invariant.
- Trust the artifact you loaded, not a log line that claims success. "synced / saved / done" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process / artifact / pixels / bytes.
- Cheap checks before expensive compute. A 1–2 batch CPU smoke (logger off) kills import/config/ shape/scale bugs for ~free, before they bill GPU-hours.
- Cost and destructive actions are the user's call. Never auto-release/terminate, never delete durable files without confirmation; if cleanup can't free space, ask to expand the disk, don't silently shrink the experiment.
- Execution permission is not task authority. Sandbox / Full Access only controls whether a command can run. Keep operational authority and scientific promotion separate: once a bounded, non-overwriting delivery objective is authorized, same-scope diagnostics, verifier/schema version bumps, tests, hashes, and control-plane repairs do not require a fresh confirmation merely because they mint a new immutable ID. Re-ask for new billable compute, destructive/irreversible actions, science-protocol changes, metric promotion, publication, or any other material scope expansion. A standing unattended contract applies only after its own activation rule is satisfied.
- Make the control plane cheap and the data plane rare.