Software Engineering Architecture Lab

Level 8: Data-Intensive Systems

Explain how databases store, replicate, partition and protect data — and predict how they fail. Level 8 of 11: 15 lessons on data-intensive systems.

Outcome: Explain how databases store, replicate, partition and protect data — and predict how they fail.

A million users later, the database is the bottleneck. We open the hood: storage engines, indexes, caches, replication, partitioning, transactions, distributed-systems failure, and logs/streams with Kafka.

The pressure on ShopFlow: Checkout p95 is 1.2 s, the primary database is at 90% CPU, and analytics queries slow everything down.

Lessons in level 8

  1. Storage Engines: B-Trees vs LSM-Trees — Pages and in-place updates vs append-only logs, SSTables and compaction.
  2. Indexes: Fast Reads, Slower Writes — How an index turns a full scan into a lookup, and what every index costs.
  3. Caching: Speed at the Cost of Truth — Cache-aside, write-through, TTLs, invalidation and stampedes.
  4. Replication: Leaders, Followers & Failover — Synchronous vs asynchronous copies, and what happens when the leader dies.
  5. Replication Lag: When Your Own Write Disappears — Read-your-writes, monotonic reads and consistent prefix reads.
  6. Partitioning: Hash, Range, Hotspots & Rebalancing — Split data across machines without creating hot partitions.
  7. Transactions & What ACID Actually Promises — Atomicity, consistency, isolation, durability — precisely, and without marketing.
  8. Isolation Levels: Dirty Reads to Serializable — Anomalies, MVCC, locks, and why “read committed” is not enough for money.
  9. Lost Updates: Optimistic vs Pessimistic Locking — Two admins edit the same product; one change silently vanishes.
  10. The Trouble With Distributed Systems — Unreliable networks, unreliable clocks and partial failure.
  11. Consistency Models, CAP & Consensus — Linearizability, eventual consistency and an honest reading of CAP.
  12. Logs & Kafka: Partitions, Offsets and Ordering — An append-only log as the backbone of data flow.
  13. Consumer Groups & Delivery Semantics — Scaling consumers, rebalancing, and at-least-once vs “exactly-once”.
  14. Batch vs Stream Processing & Data Pipelines — Derived data, reprocessing, and moving data between systems.
  15. Event Sourcing & CQRS — Store what happened; build read models for how you want to query it.

Source books

  • DDIA — Martin Kleppmann

Other levels