Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Data & analytics

ddia-principles

Designing Data-Intensive Applications (DDIA) distilled reference guide by Martin Kleppmann. MUST be loaded when: designing database schemas, choosing storage engines, implementing replication or partitioning, handling distributed transactions, building batch/stream processing pipelines, choosing consistency models, implementing consensus, designing data flow architectures, evaluating trade-offs between availability and consistency, encoding/serialization decisions, data modeling (relational vs document vs graph), building fault-tolerant systems, or any system design and architecture discussion involving data-intensive applications. Trigger on: database design, replication, partitioning, sharding, transactions, isolation levels, consistency, consensus, CAP theorem, batch processing, stream processing, MapReduce, Kafka, event sourcing, CDC, OLTP, OLAP, B-tree, LSM-tree, data warehouse, schema evolution, encoding formats, distributed systems, fault tolerance, leader election, quorum.

luoling8192 · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add luoling8192/skills --skill ddia-principles

About this skill


name: ddia-principles description: > Designing Data-Intensive Applications (DDIA) distilled reference guide by Martin Kleppmann. MUST be loaded when: designing database schemas, choosing storage engines, implementing replication or partitioning, handling distributed transactions, building batch/stream processing pipelines, choosing consistency models, implementing consensus, designing data flow architectures, evaluating trade-offs between availability and consistency, encoding/serialization decisions, data modeling (relational vs document vs graph), building fault-tolerant systems, or any system design and architecture discussion involving data-intensive applications. Trigger on: database design, replication, partitioning, sharding, transactions, isolation levels, consistency, consensus, CAP theorem, batch processing, stream processing, MapReduce, Kafka, event sourcing, CDC, OLTP, OLAP, B-tree, LSM-tree, data warehouse, schema evolution, encoding formats, distributed systems, fault tolerance, leader election, quorum.

Designing Data-Intensive Applications — Distilled Guide

Source: Martin Kleppmann, Designing Data-Intensive Applications Central thesis: Data is the core challenge of modern applications — not compute.


Part I: Foundations of Data Systems

Chapter 1: Reliability, Scalability, Maintainability

Three Pillars

PillarDefinitionKey Metric
ReliabilitySystem works correctly even when faults occurFault ≠ Failure; tolerate faults, prevent failures
ScalabilitySystem handles load growth gracefullyMeasure with percentiles: p50, p95, p99, p999
MaintainabilitySystem is easy to operate, understand, evolveOperability + Simplicity + Evolvability

Fault Categories

  • Hardware: Random, independent (disk, RAM, power). Mitigate with redundancy (RAID, dual power).
  • Software: Systematic bugs affecting all nodes simultaneously (leap-second bug). Mitigate with process isolation, monitoring, chaos engineering.
  • Human: #1 cause of outages (config errors). Mitigate with good abstractions, sandboxes, canary deployments, fast rollback.

Scalability Patterns

  • Vertical (scale-up): Bigger machine. Simple but has ceiling.
  • Horizontal (scale-out): More machines (shared-nothing). Complex but unlimited.
  • Elastic: Auto-scale on load detection. Good for unpredictable workloads.

Twitter fan-out case study: 4.6k writes/s but 300k reads/s. Solution: pre-compute timelines (write fan-out) for most users; read-time merge for celebrities.

Performance: Use Percentiles, Not Averages

  • p50 = median. p99 = tail latency matters for user experience.
  • Amazon: 100ms delay = 1% revenue loss.
  • Tail latency amplification: One slow backend call slows entire parallel request.

Chapter 2: Data Models & Query Languages

Model Selection Guide

ModelBest ForWeakness
RelationalStructured data, complex joins, ACID transactionsRigid schema, impedance mismatch with OOP
DocumentHierarchical data, flexible schema, data localityPoor joins, many-to-many relationships
GraphHighly connected data, variable-depth traversalsLess mature tooling, harder to partition

Schema Strategy

  • Schema-on-write (relational): Enforce structure at write time. Early error detection, migration cost.
  • Schema-on-read (document): Interpret structure at read time. Flexible but validation burden on app.

Normalization vs Denormalization

  • Normalize: Single source of truth, consistent updates, requires joins.
  • Denormalize: Faster reads, risks inconsistency, update anomalies.

Trend: Models converge — PostgreSQL supports JSON, MongoDB added joins. Choose based on access patterns, not ideology.


Chapter 3: Storage & Retrieval

Storage Engine Comparison

FeatureB-TreeLSM-Tree
Write throughputLower (in-place update + WAL)Higher (sequential append)
Read latencyMore predictableMay check multiple SSTables
Write amplificationHigherLower
Space efficiencyFragmentation possibleBetter compression
Transaction supportSimpler (lock on tree node)More complex
Used byPostgreSQL, MySQL, OracleLevelDB, RocksDB, Cassandra

OLTP vs OLAP

AspectOLTPOLAP
AccessRandom, few recordsSequential scan, millions of rows
UsersEnd usersAnalysts
DataCurrent stateHistorical events
ScaleGB–TBTB–PB
Optimize forLow latencyThroughput

Data Warehousing

  • ETL: Extract from OLTP → Transform → Load into warehouse.
  • Star schema: Central fact table (events) + dimension tables (attributes). Fact tables can have 100+ columns and petabyte scale.
  • Column-oriented storage: Store each column separately. Huge I/O savings when queries touch few columns. Enables bitmap encoding, run-length compression, vectorized processing.

Chapter 4: Encoding & Evolution

Format Comparison

FormatSize (example)SchemaEvolutionCross-language
JSON81 bytesImplicitManualExcellent
Thrift59 bytesRequiredField tagsGood
Protobuf33 bytesRequiredField tagsExcellent
Avro32 bytesRequiredName matchingGood

Compatibility Rules

  • Backward compatible: New code reads old data. (Always required)
  • Forward compatible: Old code reads new data. (Required for rolling upgrades)
  • Rule: Only add/remove fields with default values. Never reuse deleted field tags.

Data Flow Patterns

  1. Via databases: Multiple code versions coexist during rolling deploys. Data outlives code.
  2. Via services (REST/RPC): Servers update before clients. B
Read the complete source ↗