Post 01 · DDIA Ch. 1–2

Foundations

Twitter serves ~300k timeline reads/s and ~12k tweets/s — same product, opposite extremes. Before you can build for that, you need three tools: a way to see the load, a way to measure it honestly, and a data model that fits the shape.

Six visuals. Two chapters. One story: what a data-intensive app actually has to survive.

Twitter's timeline is a read-heavy workload sitting on top of a write-heavy social graph. Two ways to build it: fan-out on read (join followers' tweets at query time — cheap writes, expensive reads) or fan-out on write (materialise each user's timeline on every tweet — cheap reads, expensive writes for anyone with millions of followers). Neither is right; Twitter runs a hybrid.

Averages hide the users who suffer. If p50 is 10ms and p99 is 2s, one in a hundred requests is a full-page freeze — and those requests are usually the same power users, hitting the biggest queries, over and over. Percentiles are how you see the tail; the tail is where your reputation lives.

Everything in the book hangs off three words: reliability (works correctly under adversity), scalability (keeps working as load grows), maintainability (the next engineer can extend it without dread). Every trade-off in the coming chapters is a bet on which of the three matters most right now.

Reliability, scalability, maintainability — the framework says what you're optimising for. The next question is what you're storing. The shape of your data picks the database.

A résumé looks like a nested document: one person, arrays of jobs, arrays of schools. Store it as JSON and reads are one lookup. Normalise it into SQL tables and reads become joins — but IDs stay stable and referential integrity is free. Turn it into a graph and connections between people become first-class. Same data, three physical shapes, three different sets of queries that are cheap.

  "user_id": 42,
  "name": "Bill",
  "region": "US",
  "positions": [
     "job_title": "CEO", "company": "Aetna" ,
     "job_title": "Senator", "company": "State of CT" 
  ],
  "education": [
     "school": "Yale", "degree": "MBA" 
  ],
  "contact_info":  "email": "bill@…" 

The moment your data has real many-to-many relationships — people who share employers, tags that belong to many posts, friends of friends — the document model starts hurting. Joins that the relational model does in one query become application-level nested fetches, and consistency across the copies of a duplicated field becomes your problem. Connectedness pushes you toward relational or graph.

No universal winner. The decision is: how tree-like is my data, how connected is it, and how often does the shape change? Documents win when reads are one-object, writes are self-contained, and the schema wanders. Relational wins when joins are the norm and constraints matter. Graph wins when the interesting questions are about paths between things.