跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
数据与分析

dpnp-io

Reading and writing files from dpnp code on Intel CPUs and GPUs. Use when the user needs to load an array into dpnp or save a dpnp result — .npy, .npz, HDF5 via h5py, Zarr, CSV or plain text — when a file is larger than device memory and has to be read in chunks, or when they ask why dpnp has no save function of its own. Covers the NumPy conversion round trip, chunked and incremental patterns, and choosing a format by dataset size.

intel · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add intel/skills --skill dpnp-io

技能说明


name: dpnp-io description: >- Reading and writing files from dpnp code on Intel CPUs and GPUs. Use when the user needs to load an array into dpnp or save a dpnp result — .npy, .npz, HDF5 via h5py, Zarr, CSV or plain text — when a file is larger than device memory and has to be read in chunks, or when they ask why dpnp has no save function of its own. Covers the NumPy conversion round trip, chunked and incremental patterns, and choosing a format by dataset size. license: Apache-2.0 compatibility: "Requires dpnp and NumPy. HDF5 needs h5py, Zarr needs zarr, CSV parsing examples use pandas." metadata: intel-skill-type: "tool-skill" version: "1.0"

dpnp file I/O

Purpose

Gets data in and out of dpnp arrays. dpnp has no native binary file I/O: every format goes through NumPy, with dpnp.array() on the way in and dpnp.asnumpy() on the way out. This skill is that round trip, plus the chunked variants for data larger than memory and the format choice by size.

Prefer it over reaching for a dpnp.save() that does not exist, and over loading a file whole when the device cannot hold it.

When to Use This Skill

Use this skill when:

  • An array has to be loaded into dpnp from a file, or a result written out.
  • A file is larger than host or device memory and must be streamed in pieces.
  • The user is choosing between .npy, .npz, HDF5, Zarr, and CSV.
  • The user asks why dpnp will not write their format.

Do not use this skill to decide device placement or chunk sizing against device capacity — that is dpnp-memory — and do not expect it to make an I/O-bound job faster: if reading dominates, moving the compute to a device changes nothing.

Quick Start

import numpy
import dpnp

arr = dpnp.array(numpy.load("data.npy"))          # host file -> device array
result = dpnp.fft.fft2(arr) + dpnp.mean(arr)      # compute on the device
numpy.save("output.npy", dpnp.asnumpy(result))    # device array -> host file

The whole skill is that shape: NumPy load → dpnp.array() → compute → dpnp.asnumpy() → NumPy save.

Implementation Guide

  1. .npy and .npz. One array or several, with the archive closed after reading:

    with numpy.load("data.npz") as npz:
        x = dpnp.array(npz["x"])
        y = dpnp.array(npz["y"])
    
    numpy.savez("output.npz", x=dpnp.asnumpy(x), y=dpnp.asnumpy(y))
    

    Each conversion needs a full host copy of the array as well as the device copy, so a 4 GB array wants 4 GB of free RAM during the call.

  2. Chunked reads for a file larger than RAM. Memory-map the source, write each processed chunk straight into a pre-allocated output slice rather than appending to a list:

    data = numpy.load("large.npy", mmap_mode="r")
    final = numpy.empty(len(data), dtype=numpy.float64)
    chunk = 25_000_000
    
    for start in range(0, len(data), chunk):
        host = data[start:start + chunk]
        processed = dpnp.sqrt(dpnp.array(host)) * 2.0
        final[start:start + len(host)] = dpnp.asnumpy(processed)
    
    numpy.save("output.npy", final)
    

    A chunk of roughly a tenth to a fifth of free RAM is a workable start.

  3. HDF5 through h5py. h5py only speaks NumPy, so the same conversion applies, and datasets can be written incrementally when the result is too large to hold:

    import h5py
    
    with h5py.File("output.h5", "w") as handle:
        dset = handle.create_dataset("result", shape=(50_000_000,), dtype="float64")
        for start in range(0, 50_000_000, 5_000_000):
            dset[start:start + 5_000_000] = dpnp.asnumpy(compute_chunk(start))
    
  4. Zarr for very large or remote arrays. Chunked, compressed, and reachable on object storage through fsspec; read and write slice by slice:

    import zarr
    
    store = zarr.open("output.zarr", mode="w", shape=(10_000_000,),
                      chunks=(500_000,), dtype="float32")
    for start in range(0, 10_000_000, 500_000):
        store[start:start + 500_000] = dpnp.asnumpy(compute_chunk(start))
    
  5. Text and CSV. dpnp.loadtxt() returns a dpnp array directly (it delegates to numpy.loadtxt internally, and does not support structured dtypes). Anything with headers, strings, or missing values goes through numpy.loadtxt/numpy.genfromtxt or pandas first:

    import pandas
    
    frame = pandas.read_csv("data.csv")
    arr = dpnp.array(frame.values)
    numpy.savetxt("output.csv", dpnp.asnumpy(arr), delimiter=",")
    
  6. Pick the format by size. .npy/.npz below about a gigabyte, HDF5 for multi-dataset files in the gigabyte range, Zarr above that or when the data lives in cloud storage, CSV only for small human-readable exports.

Performance

No measured numbers ship with this skill. What to measure, and in which order:

  • Time the I/O and the compute separately first. If reading dominates, no device will help and the conversion cost is irrelevant either way.
  • Count conversions, not bytes. One conversion at each end of a batch of work is the pattern; one per iteration of a loop is the anti-pattern, and it is the usual reason a rewritten pipeline is no faster.
  • Chunking trades peak memory against more conversions. Compare the two on the real file rather than assuming a ratio.
  • CSV parsing is CPU-bound and dominates everything around it. Convert once to .npy or HDF5 if the same file is read repeatedly.

Gotchas & Limitations

  • There is no dpnp.save() for binary formats. dpnp.loadtxt() exists; .npy, HDF5, and Zarr all go through NumPy. Code that calls a dpnp save function fails at the call, not at review.
  • Conversion doubles peak memory. Host copy plus device copy, briefly, for every dpnp.array() and dpnp.asnumpy().
  • Accumulating chunks in a list defeats chunking. The whole point is that the full array never exists in memory; a pre-allocated output or an incremental dataset write
阅读完整来源 ↗