ddia-systems
Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions "database choice", "which database should I use", "SQL or NoSQL", "replication lag", "partitioning strategy", "consistency vs availability", "stream processing", "ACID transactions", "eventual consistency", "my queries are slow at scale", or "data is inconsistent across replicas". Also trigger when choosing a datastore, designing data pipelines, or debugging distributed-system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.
wondelai · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add wondelai/skills --skill ddia-systems技能说明
name: ddia-systems description: 'Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions "database choice", "which database should I use", "SQL or NoSQL", "replication lag", "partitioning strategy", "consistency vs availability", "stream processing", "ACID transactions", "eventual consistency", "my queries are slow at scale", or "data is inconsistent across replicas". Also trigger when choosing a datastore, designing data pipelines, or debugging distributed-system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.' license: MIT metadata: author: wondelai version: "1.4.0"
Designing Data-Intensive Applications Framework
A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.
Core Principle
Data outlives code. Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.
Scoring
Goal: 10/10. Score a data architecture by the seven Quick Diagnostic rows below: award ~1.4 points per row answered "yes" with evidence (deliberate, documented trade-off), 0 where the answer is "no" or unknown.
- 9-10: every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read/write/consistency requirements; failover tested.
- 5-6: core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.
- <=3: choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.
Report the current score, which diagnostic rows failed, and the improvements needed to reach 10/10.
The DDIA Framework
Seven domains for reasoning about data-intensive systems:
1. Data Models and Query Languages
Core concept: The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.
Why it works: Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.
Key insights:
- Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data
- Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility
- Polyglot persistence -- different stores for different access patterns -- is often the right answer
- Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates
Code applications:
| Context | Pattern | Example |
|---|---|---|
| User profiles with nested data | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document |
| Social network connections | Graph model for relationship traversal | Neo4j Cypher: MATCH (a)-[:FOLLOWS*2]->(b) for friend-of-friend |
| Financial ledger with joins | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, entries |
See references/data-models.md when picking relational vs document vs graph or evaluating schema-on-read -- adds the full trade-off matrix and query-language comparisons.
2. Storage Engines
Core concept: Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.
Key insights:
- LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification
- B-trees: in-place updates, predictable read latency, write amplification from page splits
- Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles
- Column-oriented storage dramatically improves analytical queries through compression and vectorized processing
- In-memory databases are fast because they avoid encoding overhead, not because they avoid disk
Code applications:
| Context | Pattern | Example |
|---|---|---|
| High write throughput | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes/sec |
| Mixed read/write OLTP | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups |
| Analytical queries | Column-oriented storage | ClickHouse or Parquet for scanning billions of rows, few columns |
See references/storage-engines.md when a workload is read/write-bound or you must choose indexes -- adds write/read-path diagrams, compaction strategies, column storage, and a benchmark-driven decision procedure.
3. Replication
Core concept: Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.
Why it works: Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.
**Key ins
阅读完整来源 ↗