ddia-principles
Designing Data-Intensive Applications (DDIA) distilled reference guide by Martin Kleppmann. MUST be loaded when: designing database schemas, choosing storage engines, implementing replication or partitioning, handling distributed transactions, building batch/stream processing pipelines, choosing consistency models, implementing consensus, designing data flow architectures, evaluating trade-offs between availability and consistency, encoding/serialization decisions, data modeling (relational vs document vs graph), building fault-tolerant systems, or any system design and architecture discussion involving data-intensive applications. Trigger on: database design, replication, partitioning, sharding, transactions, isolation levels, consistency, consensus, CAP theorem, batch processing, stream processing, MapReduce, Kafka, event sourcing, CDC, OLTP, OLAP, B-tree, LSM-tree, data warehouse, schema evolution, encoding formats, distributed systems, fault tolerance, leader election, quorum.
luoling8192 · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add luoling8192/skills --skill ddia-principlesAbout this skill
name: ddia-principles description: > Designing Data-Intensive Applications (DDIA) distilled reference guide by Martin Kleppmann. MUST be loaded when: designing database schemas, choosing storage engines, implementing replication or partitioning, handling distributed transactions, building batch/stream processing pipelines, choosing consistency models, implementing consensus, designing data flow architectures, evaluating trade-offs between availability and consistency, encoding/serialization decisions, data modeling (relational vs document vs graph), building fault-tolerant systems, or any system design and architecture discussion involving data-intensive applications. Trigger on: database design, replication, partitioning, sharding, transactions, isolation levels, consistency, consensus, CAP theorem, batch processing, stream processing, MapReduce, Kafka, event sourcing, CDC, OLTP, OLAP, B-tree, LSM-tree, data warehouse, schema evolution, encoding formats, distributed systems, fault tolerance, leader election, quorum.
Designing Data-Intensive Applications — Distilled Guide
Source: Martin Kleppmann, Designing Data-Intensive Applications Central thesis: Data is the core challenge of modern applications — not compute.
Part I: Foundations of Data Systems
Chapter 1: Reliability, Scalability, Maintainability
Three Pillars
| Pillar | Definition | Key Metric |
|---|---|---|
| Reliability | System works correctly even when faults occur | Fault ≠ Failure; tolerate faults, prevent failures |
| Scalability | System handles load growth gracefully | Measure with percentiles: p50, p95, p99, p999 |
| Maintainability | System is easy to operate, understand, evolve | Operability + Simplicity + Evolvability |
Fault Categories
- Hardware: Random, independent (disk, RAM, power). Mitigate with redundancy (RAID, dual power).
- Software: Systematic bugs affecting all nodes simultaneously (leap-second bug). Mitigate with process isolation, monitoring, chaos engineering.
- Human: #1 cause of outages (config errors). Mitigate with good abstractions, sandboxes, canary deployments, fast rollback.
Scalability Patterns
- Vertical (scale-up): Bigger machine. Simple but has ceiling.
- Horizontal (scale-out): More machines (shared-nothing). Complex but unlimited.
- Elastic: Auto-scale on load detection. Good for unpredictable workloads.
Twitter fan-out case study: 4.6k writes/s but 300k reads/s. Solution: pre-compute timelines (write fan-out) for most users; read-time merge for celebrities.
Performance: Use Percentiles, Not Averages
- p50 = median. p99 = tail latency matters for user experience.
- Amazon: 100ms delay = 1% revenue loss.
- Tail latency amplification: One slow backend call slows entire parallel request.
Chapter 2: Data Models & Query Languages
Model Selection Guide
| Model | Best For | Weakness |
|---|---|---|
| Relational | Structured data, complex joins, ACID transactions | Rigid schema, impedance mismatch with OOP |
| Document | Hierarchical data, flexible schema, data locality | Poor joins, many-to-many relationships |
| Graph | Highly connected data, variable-depth traversals | Less mature tooling, harder to partition |
Schema Strategy
- Schema-on-write (relational): Enforce structure at write time. Early error detection, migration cost.
- Schema-on-read (document): Interpret structure at read time. Flexible but validation burden on app.
Normalization vs Denormalization
- Normalize: Single source of truth, consistent updates, requires joins.
- Denormalize: Faster reads, risks inconsistency, update anomalies.
Trend: Models converge — PostgreSQL supports JSON, MongoDB added joins. Choose based on access patterns, not ideology.
Chapter 3: Storage & Retrieval
Storage Engine Comparison
| Feature | B-Tree | LSM-Tree |
|---|---|---|
| Write throughput | Lower (in-place update + WAL) | Higher (sequential append) |
| Read latency | More predictable | May check multiple SSTables |
| Write amplification | Higher | Lower |
| Space efficiency | Fragmentation possible | Better compression |
| Transaction support | Simpler (lock on tree node) | More complex |
| Used by | PostgreSQL, MySQL, Oracle | LevelDB, RocksDB, Cassandra |
OLTP vs OLAP
| Aspect | OLTP | OLAP |
|---|---|---|
| Access | Random, few records | Sequential scan, millions of rows |
| Users | End users | Analysts |
| Data | Current state | Historical events |
| Scale | GB–TB | TB–PB |
| Optimize for | Low latency | Throughput |
Data Warehousing
- ETL: Extract from OLTP → Transform → Load into warehouse.
- Star schema: Central fact table (events) + dimension tables (attributes). Fact tables can have 100+ columns and petabyte scale.
- Column-oriented storage: Store each column separately. Huge I/O savings when queries touch few columns. Enables bitmap encoding, run-length compression, vectorized processing.
Chapter 4: Encoding & Evolution
Format Comparison
| Format | Size (example) | Schema | Evolution | Cross-language |
|---|---|---|---|---|
| JSON | 81 bytes | Implicit | Manual | Excellent |
| Thrift | 59 bytes | Required | Field tags | Good |
| Protobuf | 33 bytes | Required | Field tags | Excellent |
| Avro | 32 bytes | Required | Name matching | Good |
Compatibility Rules
- Backward compatible: New code reads old data. (Always required)
- Forward compatible: Old code reads new data. (Required for rolling upgrades)
- Rule: Only add/remove fields with default values. Never reuse deleted field tags.
Data Flow Patterns
- Via databases: Multiple code versions coexist during rolling deploys. Data outlives code.
- Via services (REST/RPC): Servers update before clients. B