blob-store
This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at scale", object "versioning", storage "tiering" (hot/cold/archive), or "erasure coding" vs replication for durability. Use it whenever a design must hold large unstructured objects (photos, video, backups, logs, ML datasets) and serve them cheaply and durably, even if the user just says "where do we put the files".
proyecto26 · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add proyecto26/system-design-skills --skill blob-storeAbout this skill
name: blob-store description: This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at scale", object "versioning", storage "tiering" (hot/cold/archive), or "erasure coding" vs replication for durability. Use it whenever a design must hold large unstructured objects (photos, video, backups, logs, ML datasets) and serve them cheaply and durably, even if the user just says "where do we put the files".
Blob store
Store large, immutable, unstructured objects — images, video, backups, model weights, document blobs — in a flat namespace keyed by a string, replicated for durability and served by direct download. Getting it wrong means stuffing multi-megabyte blobs into a row-oriented database (where they bloat the working set, wreck cache locality, and cap throughput) or hand-rolling a file server that loses data on the first disk failure.
When to reach for this
Objects are large (KB to GB), written once and read many times, and you only ever
fetch them whole by key — never query inside them. Photo/video stores, user
uploads, backups, data-lake/ML datasets, static-site assets, log archives. The
access pattern is PUT key → GET key, durability matters, and the total volume is
too large or too cold to sit in a primary database.
When NOT to
Small structured records you query, filter, sort, or join — that is data-storage.
Data that needs transactions, secondary indexes, or partial updates (blobs are
replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or
caching wins). A few files on one box that never grow — the local filesystem is
fine; a blob store is operational overhead you do not need yet (YAGNI). Naming
"object storage" for a workload that is really a database is failure mode #2.
Clarify first
- Object size distribution — average and p99 size? (Decides chunking, multipart
thresholds, and whether reads stream or buffer.) →
back-of-the-envelope. - Read:write ratio and access recency — write-once/read-many? How fast does data go cold? (Drives tiering and CDN fronting.)
- Durability and availability target — how many nines of durability? Can a read briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.)
- Access control — public, private, or time-limited per-object grants? (Signed URLs.)
- Mutability and history — do objects change? Must old versions be retained (compliance, undo)? (Versioning + lifecycle.)
- Egress profile — who reads, from where, how often? (CDN offload, egress cost.)
The options
Durability scheme (how many copies, what shape)
- N-way replication — store N full copies on different nodes/racks/AZs. Use when objects are small, hot, and latency matters; simplest to reason about.
- Erasure coding (EC) — split an object into k data + m parity shards; any k
reconstruct it. Use for large/cold data at scale — same durability as replication
at ~1.4x overhead instead of 3x. (Mechanics in
references/deep-dive.md.)
Storage tier (price/latency/retrieval trade)
- Hot/standard — millisecond reads, highest $/GB. Use for actively served objects.
- Cool/infrequent — cheaper storage, retrieval fee/slightly higher latency. Use for backups and data read a few times a month.
- Archive/cold — cheapest storage, minutes-to-hours retrieval. Use for compliance retention and rarely-touched data; never for anything on a request path.
Upload path
- Single PUT — one request. Use for small objects (under the multipart threshold).
- Multipart / resumable — split into parts, upload in parallel, retry per-part, commit on completion. Use for large objects and flaky networks; the default above the threshold.
Mutation model
- Immutable + versioning — each write is a new version; deletes are tombstones. Use when history, undo, or accidental-overwrite protection matters.
- Overwrite-in-place (last-writer-wins) — simplest; no history. Use when only the latest object matters and storage of old copies is waste.
Trade-offs
| Option | What it solves | What it worsens | Change it when |
|---|---|---|---|
| N-way replication | Simple, fast reads, fast rebuild | 3x+ storage cost | Data is large/cold and cost dominates → erasure coding |
| Erasure coding | Same durability at ~1.4x storage | CPU + multi-node read on every fetch; slow small-object reads; costly rebuild | Objects are small/hot and latency matters → replication |
| Hot tier | Low-latency serving | Highest $/GB | Data goes cold and is rarely read → cool/archive |
| Archive tier | Cheapest at-rest storage | Minutes–hours to first byte; retrieval fees | Anything ends up on a latency-sensitive path → hot/cool |
| Multipart/resumable upload | Large files survive flaky links; parallel throughput | More client logic; orphaned parts cost money | Objects are small → single PUT |
| Versioning | Undo, history, overwrite protection | Storage grows silently; needs lifecycle expiry | Only latest matters → overwrite, last-writer-wins |
| Signed URLs | Offload transfer off your app; scoped access | Leaked/over-broad URLs; clock-skew expiry bugs | Content is fully public → CDN + public read |
Behavior under stress
A blob store rarely "falls over" the way a database does, but it amplifies trouble in specific ways.
- Hot object / hot prefix: a viral file or a key scheme where many writes share a
prefix concentrates load on one partition. Mitigate: front hot reads with a CDN
(
content-delivery), randomize/hash key prefixes, replicate the hot object. - Metadata-index bottleneck: the index that maps key → shard locations is the real SPOF and the throughput ceiling (millions of tiny objects hurt far more than a few huge ones). *Mitig