Shard pyfixest Grids to Fix Slow GPU Regression Runs

Where this fits

You're an applied economist or ML researcher running a robustness grid: 2 outcome measures × 3 fixed-effect structures × per-domain splits × label variants, easily 70+ models. On a 100k+ row panel with high-cardinality fixed effects, each pf.feols call takes ~55 seconds despite demeaner_backend="cupy64" and an idle RTX 5080. You check nvidia-smi — 0% GPU, while top shows 1.5 cores busy. The GPU isn't the bottleneck; the CPU-side fixed costs are. This skill gives you a diagnostic checklist and a concrete fix: shard the grid across OS processes and push that mandate into your agent's prompt.

Why agents benefit

• Detects the real bottleneck: Distinguishes GPU-bound demeaning from single-threaded CPU overhead (formulaic matrix build, interaction construction, singleton detection, cluster vcov) using process inspection and GPU utilization.
• Prescribes a concrete fix: Shards the model grid across processes (e.g., multiprocessing or concurrent.futures) or uses pyfixest's multiple-estimation syntax to cut wall time by 5–10×.
• Prevents recurrence: Injects the sharding requirement into worker prompts, so future grid runs never silently fall back to sequential execution.
• Saves compute budget: Avoids wasting GPU hours on jobs that are CPU-limited, and reduces iteration cycles for robustness checks.
• Works with existing code: No API rewrites — just a wrapper around your current loop, plus a prompt-level guardrail.

Practical scenarios

Scenario 1: A 70-model grid on a 327k-row panel

You have a panel of firm-director matches with 12k director fixed effects. Running 70 models sequentially takes 65 minutes. After applying the skill, you shard the grid into 8 processes (one per CPU core), cutting runtime to ~8 minutes. The GPU remains idle, but that's fine — you're now CPU-bound in parallel.

Scenario 2: Agent-generated robustness checks in a research pipeline

Your agent automatically generates specification grids (outcome × FE structure × clustering). Without the prompt mandate, it writes a naive for loop. The skill's prompt snippet forces it to use ProcessPoolExecutor or pyfixest's estimation with multiple formulas — even if the dataset is small, the pattern scales.

Scenario 3: Debugging a "slow GPU" complaint

A colleague reports pyfixest is slow "because of CuPy." You run nvidia-smi and top — GPU at 0%, CPU at 1.2 cores. The skill's diagnostic steps confirm the root cause, and you refactor the grid to use process-level parallelism, improving throughput without touching the demeaning backend.

Add it to your agent workflow

Install the skill via:

npx skills add kennethkhoocy/applied-micro-skills --skill pyfixest-grid-sharding

Then in your worker prompt, include: "When running multiple pyfixest models, shard the grid across processes or use multiple-estimation syntax. Never loop sequentially." The skill's README provides a ready-to-copy snippet for multiprocessing.Pool with a model-grid function.

Compare before adopting

Before integrating, compare against:

• Alternatives: linearmodels or fixest in R (which has native multi-threading). Does pyfixest's feature set (e.g., fepois, cluster vcov) justify staying in Python?
• Quality signals: Check the repo's issue tracker for recent fixes around CPU overhead. The skill is maintained by Claude Code, but verify it matches your pyfixest version.
• Workflow fit: If your grid is small (<20 models) or your data is <50k rows, the overhead of process spawning may not pay off. This skill shines on large panels with many specifications.

Why it is worth tracking

This skill addresses a common but under-diagnosed performance pitfall in causal inference workflows. It has 47 stars and is actively maintained, with a clear diagnostic framework that transfers to any CPU-bound model grid (e.g., statsmodels, sklearn). Track it if you routinely run specification grids or plan to delegate such tasks to agents. For more skills like this, see OpenAgentSkill. Evaluate it whenever your GPU utilization is low but wall time is high — it will save you hours.