Software Engineering Architecture Lab

Batch vs Stream Processing & Data Pipelines

Derived data, reprocessing, and moving data between systems. Lesson 14 of Level 8: Data-Intensive Systems. Based on DDIA.

Level 8 · Data-Intensive Systems · Lesson 14 of 15

Derived data, reprocessing, and moving data between systems.

Continue learning

Try it hands-on

  • Kafka Lab — Logs, partitions, keys, consumer groups and ordering.

All lessons in Data Systems

  1. Storage Engines: B-Trees vs LSM-Trees
  2. Indexes: Fast Reads, Slower Writes
  3. Caching: Speed at the Cost of Truth
  4. Replication: Leaders, Followers & Failover
  5. Replication Lag: When Your Own Write Disappears
  6. Partitioning: Hash, Range, Hotspots & Rebalancing
  7. Transactions & What ACID Actually Promises
  8. Isolation Levels: Dirty Reads to Serializable
  9. Lost Updates: Optimistic vs Pessimistic Locking
  10. The Trouble With Distributed Systems
  11. Consistency Models, CAP & Consensus
  12. Logs & Kafka: Partitions, Offsets and Ordering
  13. Consumer Groups & Delivery Semantics
  14. Batch vs Stream Processing & Data Pipelines
  15. Event Sourcing & CQRS