Designing Data-Intensive Applications Part 5: The Future of Data Systems

We have traveled a long, rigorous road across the landscape of modern data architecture.

• In Part 1, we studied single-node storage engines (LSM-Trees vs B-Trees) and foundational trade-offs in data modeling.
• In Part 2, we distributed data across networks using Replication (Leaderless/Single-Leader) and Partitioning (Consistent Hashing).
• In Part 3, we tackled the chaotic realities of concurrency, ACID transactions, MVCC, and Distributed Consensus.
• In Part 4, we explored large-scale computation over bounded batch data (Spark) and unbounded real-time stream data (Kafka & CDC).

Now, in this 5th and final installment of our Designing Data-Intensive Applications masterclass series, we step back to synthesize the big picture: How do all these components combine to form the future of software architecture?

1. Unbundling the Database

In the early days of software engineering, a single Relational Database Management System (RDBMS) like Oracle, DB2, or PostgreSQL handled every data requirement an application had:

• It was your transactional store (OLTP).
• It was your search engine (via basic LIKE '%query%' queries).
• It was your queue (via SELECT FOR UPDATE poll loops).
• It was your analytics engine (via heavy GROUP BY aggregations).

As applications expanded to internet scale, no single monolithic database could excel at all these specialized workloads.

                   MONOLITHIC PAST                      UNBUNDLED FUTURE
             ┌─────────────────────────┐           ┌────────────────────────┐
             │   Monolithic RDBMS      │           │ Primary OLTP Database  │
             │  (OLTP + Search + Cache │           └───────────┬────────────┘
             │   + Analytics Queue)    │                       │ (CDC Log)
             └─────────────────────────┘                       ▼
                                                   ┌────────────────────────┐
                                                   │ Event Log (Kafka Stream)│
                                                   └───────────┬────────────┘
                                           ┌───────────────────┼───────────────────┐
                                           ▼                   ▼                   ▼
                                    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
                                    │ Search Index │    │ Memory Cache │    │ OLAP Engine  │
                                    │Elasticsearch │    │    Redis     │    │  Snowflake   │
                                    └──────────────┘    └──────────────┘    └──────────────┘

What “Unbundling” Means

Martin Kleppmann introduces the concept of Unbundling the Database:

Instead of forcing a single database binary to execute every workload internally, we compose specialized index and storage tools together. The central event log (e.g. Apache Kafka via Change Data Capture) acts as the distributed transaction log connecting these specialized storage engines!

• Primary OLTP Store (PostgreSQL): Optimized for low-latency atomic transactions.
• Search Index (Elasticsearch/Meilisearch): Optimized for full-text inverted index searches.
• Cache (Redis): Optimized for sub-millisecond key-value reads.
• Analytics Store (Snowflake/ClickHouse): Columnar storage optimized for OLAP aggregation.

2. Primary Data vs Derived Data

To maintain sanity in an unbundled architecture, you must strictly categorize every data store into one of two roles:

                               ┌─────────────────────────┐
                               │     SOURCE OF TRUTH     │
                               │  (Primary System Log)   │
                               └────────────┬────────────┘
                                            │
                                            │ (Deterministic Transformations)
                                            ▼
                               ┌─────────────────────────┐
                               │      DERIVED DATA       │
                               │ (Caches, Search Indexes)│
                               └─────────────────────────┘

1. Source of Truth (Primary Data)

The authoritative, canonical version of your data. Writes enter the system here first. It represents the immutable record of reality. If there is a disagreement between data stores, the Source of Truth is always right.

2. Derived Data

Data that is created by taking the primary data and running a transformation function on it.

• Examples: Caches, search indexes, materialized views, predictive ML features.
• Crucial Property: Derived data is lossless and disposable! If your Elasticsearch index corrupts or your Redis cache crashes, you have lost zero business data. You simply replay the primary log stream through your transformation pipeline and rebuild the derived index retroactively!

3. Lambda Architecture vs Kappa Architecture

How should we structure data pipelines that process both historical batch data and real-time streaming data?

The Lambda Architecture (Nathan Marz, 2011)

Lambda Architecture runs two parallel computation layers:

                          ┌──> Batch Layer (Hadoop/Spark) ──> Batch View ──┐
                          │                                                 │
Incoming Data Stream ─────┤                                                 ├──> Query Result
                          │                                                 │
                          └──> Speed Layer (Storm/Samza)  ──> Real-Time View ┘

1. Batch Layer: Processes all historical data in batch mode every night for 100% accurate results (Slow, high latency).
2. Speed Layer: Processes recent data in real-time to provide low-latency estimates for the last few hours (Fast, lower accuracy).
3. Serving Layer: Merges results from the Batch View and Real-Time View at query time.

The Flaw of Lambda Architecture:

You are forced to write and maintain two separate codebases doing the exact same logical computation (e.g. Java code in MapReduce and C++ code in Storm). Bugs introduced in one layer create silent discrepancies!

The Kappa Architecture (Jay Kreps, 2014)

Kappa Architecture eliminates the Batch layer entirely!

Incoming Stream ──> [ Log-Based Event Broker (Kafka) ] ──> Stream Processor (Flink/Spark) ──> Serving View

• Everything is processed as a Stream.
• Historical data is simply retained in the event log (Kafka) for a long period.
• If you change your computation code, you simply start a new stream processor instance, reset its offset to position 0 (the beginning of time), reprocess historical events at high speed, and swap the serving view!

4. End-to-End Correctness

A fundamental mistake in software architecture is assuming that relying on ACID transactions at the database level guarantees 100% application-level correctness.

The Limits of Database Transactions

Consider an online banking application where a user clicks “Transfer $100”:

1. User clicks "Pay $100" in web UI.
2. Web server sends request to Database.
3. Database executes ACID Transaction (Debits $100) ──> SUCCESS!
4. Network drops! The HTTP response fails to reach the user's browser!
5. User sees an error spinner, gets confused, and clicks "Pay $100" AGAIN!
6. Second ACID Transaction executes (Debits ANOTHER $100)!

The database transaction worked perfectly both times! It maintained atomicity, isolation, and durability. Yet the user was incorrectly charged twice!

Database transactions alone cannot solve end-to-end problems that occur outside the database boundary.

The End-to-End Argument in System Design

Pioneered by Saltzer, Reed, and Clark in 1984, the End-to-End Argument states:

“Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level.”

To guarantee application correctness, software engineers must enforce safety at the application level:

1. Idempotency Keys

Every user action is assigned a unique, client-generated Idempotency Key (UUID):

// Client generates key on button click
const idempotencyKey = "req_9f83a2c1-84bd-4e20-9118";

// Request payload sent to server
await fetch("/api/v1/transfers", {
  method: "POST",
  headers: { "Idempotency-Key": idempotencyKey },
  body: JSON.stringify({ amount: 10000, recipient: "user_456" })
});

The database stores processed idempotency keys in a processed_requests table with a UNIQUE constraint. If a duplicate request arrives (due to a retried click or network retry), the server detects the existing key, ignores the operation, and returns the previous cached response!

2. End-to-End Checksums

Do not trust network switches, storage controllers, or memory modules to preserve data integrity. Validate SHA-256 checksums at the application boundary when data enters and leaves storage.

5. Ethical Data Systems & Responsibility

As software engineers, we are not merely building abstract graphs of boxes and arrows; we are building systems that impact human lives, financial stability, and societal privacy.

                    ┌───────────────────────────────────┐
                    │      ETHICAL DATA ENGINEERING     │
                    └─────────────────┬─────────────────┘
                                      │
         ┌────────────────────────────┼────────────────────────────┐
         ▼                            ▼                            ▼
  ┌──────────────┐             ┌──────────────┐             ┌──────────────┐
  │ DATA PRIVACY │             │ PREDICTIVE   │             │ DATA DELETION│
  │ Minimization │             │ BIAS         │             │ Right to be  │
  │ & Encryption │             │ Auditing     │             │ Forgotten    │
  └──────────────┘             └──────────────┘             └──────────────┘

1. Data Minimization: Collect only the data that is strictly necessary for the application’s core function. Never store raw user credentials or unhashed PII “just in case”.
2. Predictive Bias & Discrimination: Automated classification models trained on historical data frequently perpetuate historical human biases. Always audit algorithmic scoring outputs for fairness.
3. Data Retention & Deleting Immutable Logs: In an event-sourced architecture where logs are immutable, respecting legal regulations like GDPR’s “Right to be Forgotten” requires cryptographic erasure (encrypting a user’s data with a unique key, and destroying that key upon deletion request).

Masterclass Series Conclusion

We have completed our journey through Martin Kleppmann’s Designing Data-Intensive Applications.

Building resilient software is an art of managing trade-offs:

• You trade Simplicity for Scale.
• You trade Consistency for Availability (CAP Theorem).
• You trade Read Speed for Write Throughput (B-Trees vs LSM-Trees).

Equipped with these deep mental models, you are no longer guessing when designing software. You can deliberately select the exact right tools, algorithms, and architectures to build systems that scale reliably for years to come.

Up next: AI Engineering Part 1: Fundamentals of Foundation Models.