“The most scalable systems don’t constantly ask what’s happening. They simply react when something happens.”

Imagine you’re building an online marketplace. A customer places an order. At first, the workflow seems straightforward. The application creates the order, charges the customer’s card, reserves inventory, schedules shipment, sends a confirmation email, updates analytics, awards loyalty points, and notifies the warehouse. A traditional implementation might have the Order Service call each of these services directly.

Order Service

     │

     ├── Payment Service

     ├── Inventory Service

     ├── Shipping Service

     ├── Email Service

     ├── Analytics Service

     └── Loyalty Service

At first, this architecture feels perfectly reasonable. Each service performs its work and returns a response. As the business grows, however, the Order Service slowly becomes responsible for more and more integrations. Marketing introduces a Recommendation Service, finance adds an Accounting Service, customer success wants a CRM integration, and fraud detection joins the platform. Before long, every new feature requires modifying the Order Service. A service that originally knew only how to create orders now knows almost everything about the entire company. The result is tight coupling. Every new integration increases complexity, every downstream outage affects the original request, and every deployment becomes slightly more risky.

Eventually, developers begin asking a different question. Instead of asking:

“Which services should I call?”

They begin asking:

“What happened?”

That small change in thinking fundamentally changes the architecture. Instead of directly calling every interested service, the Order Service simply announces:

“An order has been created.”

It doesn’t care who listens and doesn’t even know who listens. It simply publishes an event, and every interested service reacts independently. The Shipping Service prepares delivery, the Email Service sends a confirmation, analytics records another sale, loyalty awards points, fraud detection evaluates the transaction, and recommendation engines update customer preferences. The Order Service never calls any of them directly.

This architectural style is known as Event-Driven Architecture (EDA). Instead of services communicating through tightly coupled request-response interactions, they communicate through events describing things that have already happened. This shift may seem subtle, but in reality, it completely changes how distributed systems evolve, scale, and recover from failure.

What Is an Event?

Before discussing Event-Driven Architecture, it’s important to understand what an event actually is. An event is simply a record that something meaningful has already happened. It is not something that might happen, and not something another service should do, but something that has already occurred. Examples include:

• Order Created
• Payment Completed
• Customer Registered
• Loan Approved
• Inventory Reserved
• Shipment Delivered

Notice the wording: every event is written in the past tense because events describe facts. Once an event has occurred, it becomes part of the system’s history. Unlike commands, events don’t tell another service what to do. They simply communicate what has already happened. This distinction is incredibly important. Consider the difference between these two messages.

Command

Charge Customer

Event

Customer Charged

The first message expects another service to perform work; the second informs the rest of the system that the work has already been completed. Commands ask; events announce. That simple distinction lies at the heart of Event-Driven Architecture.

How Event-Driven Systems Work

Imagine a borrower submits a loan application. The Loan Service validates the request, stores the application, and commits its transaction. Rather than directly calling every downstream service, it publishes a LoanApplicationSubmitted event, and from there every interested service works independently.

Loan Service

      │

LoanApplicationSubmitted

      │

──────── Event Broker ────────

      │

      ├── Risk Service

      ├── Notification Service

      ├── Fraud Detection

      ├── CRM

      └── Analytics

Notice what changed: the Loan Service no longer knows anything about these downstream systems. Adding another consumer doesn’t require modifying the Loan Service, and removing one doesn’t either. Every service simply subscribes to the events it cares about. That loose coupling is one of the defining advantages of Event-Driven Architecture.

Why This Improves Scalability

Suppose your application suddenly doubles in size. Marketing introduces three new services, operations introduces two more, and finance adds another. In a request-response architecture, every one of those integrations often requires modifying the originating service. In an event-driven architecture, nothing changes. The new services simply subscribe to existing events, and the publisher remains exactly the same. This dramatically reduces coupling and allows applications to evolve much more independently. Instead of building systems that know about one another, you’re building systems that share facts. That distinction becomes increasingly valuable as applications grow.

Event Brokers: The Backbone of Event-Driven Systems

At this point, a natural question arises. If services no longer call one another directly, how do events actually travel through the system? The answer is an event broker. An event broker acts as the central communication hub for your architecture. Instead of sending events directly to every interested service, a publisher sends the event to the broker, and the broker becomes responsible for delivering it to every subscriber. Conceptually, the architecture looks like this:

                Order Service

                     │

      OrderCreated Event

                     │

                     ▼

             Event Broker

     ┌─────────┼─────────┐

     ▼         ▼         ▼

 Inventory   Shipping   Analytics

   Service    Service     Service

Notice what has disappeared: the Order Service no longer knows how many consumers exist, doesn’t know whether the Analytics Service is online, and doesn’t know whether another team introduces a Recommendation Service next month. Its only responsibility is publishing the event. Everything else becomes someone else’s concern. Several technologies can act as event brokers. Apache Kafka is widely used for high-throughput event streaming. RabbitMQ is popular for traditional message queuing. Cloud platforms offer managed services such as Amazon EventBridge, Amazon SQS, Azure Service Bus, and Google Pub/Sub. Each has different strengths, but they all serve the same purpose: moving events from producers to consumers without tightly coupling the two.

Why Event-Driven Architecture Matters

At first glance, publishing events instead of calling APIs might not seem like a revolutionary change. In practice, however, it fundamentally changes how software evolves. Consider our online marketplace again. The Order Service publishes an OrderCreated event. Initially, only three services consume it.

• Shipping
• Email
• Inventory

A year later, the business introduces:

• Fraud Detection
• Recommendation Engine
• Customer Rewards
• CRM Integration
• Business Intelligence
• Machine Learning

The Order Service doesn’t change. It continues publishing exactly the same event, and the new services simply subscribe. This is one of the greatest strengths of Event-Driven Architecture: applications grow by adding consumers rather than modifying existing publishers, which greatly reduces the ripple effect of change.

The Trade-Offs

Like every architectural style, Event-Driven Architecture solves some problems while introducing others. One important difference is that communication becomes asynchronous. When a customer places an order, the application may respond immediately even though several downstream services are still processing events. This improves responsiveness but introduces eventual consistency.

For a short period, different services may have slightly different views of the system. The order may already exist while the reporting dashboard hasn’t yet been updated, or the shipment may still be pending while the payment has already completed. This isn’t necessarily a problem; it’s simply a different consistency model, and applications must be designed with that reality in mind. Debugging also becomes more challenging. In a request-response architecture, tracing a workflow often means following a sequence of API calls. In an event-driven architecture, the workflow is distributed across many independent services reacting to events at different times, so good observability becomes essential. Correlation IDs, distributed tracing, structured logging, and monitoring tools become increasingly valuable as systems grow.

Common Mistakes

One of the biggest mistakes teams make is publishing events for everything. Not every database update deserves an event. Good events represent meaningful business occurrences. Examples include:

• Customer Registered
• Order Completed
• Loan Approved
• Payment Received

Poor events often expose internal implementation details. Examples include:

• CustomerTableUpdated
• RowModified
• AddressFieldChanged

Consumers should care about business facts, not database implementation. Another common mistake is assuming events are delivered exactly once. In reality, duplicates can occur, messages can be delayed, and consumers can retry. This is why patterns we’ve already explored, such as Idempotency, remain critically important. Reliable event-driven systems assume events may be delivered more than once and design consumers accordingly. Finally, avoid replacing every API with events. Not every interaction should be asynchronous. Some operations naturally require an immediate response. For example:

• User authentication
• Payment authorization
• Real-time validation

A healthy architecture often combines synchronous APIs with asynchronous events, using each where it makes the most sense.

Request-Response vs Event-Driven

A useful way to compare these architectures is to think about who controls the conversation.

In a request-response system, one service explicitly asks another service to perform work, and the caller waits for an answer before continuing. In an event-driven system, a service simply announces what has already happened. Anyone interested may react, and anyone uninterested simply ignores the event. Neither architecture is universally better. Request-response communication is often simpler and easier to understand, while event-driven communication provides greater flexibility and scalability as systems become more complex. Most modern applications use both. User-facing requests frequently begin as synchronous API calls, and once the business transaction completes, the application publishes events that allow the rest of the system to react independently.

How Everything Fits Together

If you’ve followed this series from the beginning, you’ve probably noticed a pattern. Each article answered a question created by the previous one.

Notice that none of these patterns exists in isolation. An event-driven application might use:

• Idempotency to safely handle duplicate events.
• Transactions to protect local database operations.
• The Outbox Pattern to reliably publish domain events.
• Saga Pattern to coordinate long-running business workflows.
• CQRS to optimize read and write workloads independently.
• Distributed Locks where multiple application instances must coordinate exclusive work.

The real power doesn’t come from mastering one pattern. It comes from understanding how they complement one another.

Final Thoughts

Software architecture isn’t about collecting design patterns. It’s about solving real problems with the right level of complexity. Event-Driven Architecture has become popular because it reflects how modern organizations grow. Teams become independent. Services evolve at different speeds. New features appear continuously. Direct dependencies become increasingly expensive to maintain. By allowing services to communicate through events rather than tightly coupled API calls, Event-Driven Architecture enables systems that are more flexible, more scalable, and more resilient to change. Like every pattern we’ve explored, however, it isn’t a silver bullet. Small applications may never need an event broker, and a well-designed monolith may outperform a poorly designed event-driven system. Architecture should always follow business needs, not trends. The goal isn’t to build the most sophisticated system possible. It’s to build the simplest system capable of solving today’s problem while leaving room for tomorrow’s growth.

Beyond CRUD: Chapter One Complete

When we began this series, we started with a deceptively simple question:

What happens if the same request arrives twice?

From there, we explored race conditions, transactions, concurrency, distributed coordination, reliable messaging, long-running workflows, scalable read models, and event-driven communication. Each article introduced a new piece of the puzzle, and together they form a foundation for understanding how modern backend systems remain reliable under retries, failures, concurrency, and scale.

If there’s one lesson to carry forward, it’s this:

Reliable software isn’t built by avoiding failure. It’s built by expecting failure, understanding where it can occur, and designing systems that recover gracefully when it does.

That mindset, more than any single framework, language, or database, is what separates production-ready systems from code that only works under perfect conditions. The journey beyond CRUD doesn’t end here. It begins here.

“The way you write data isn’t always the best way to read it.”

Imagine you’re building an online marketplace. Every day, thousands of customers browse products, place orders, track deliveries, and review their purchase history. At the same time, administrators manage inventory, warehouse staff update shipments, finance teams generate reports, and recommendation engines continuously analyze customer behavior.

From a user’s perspective, everything appears seamless. Behind the scenes, however, the application is performing two fundamentally different kinds of work. Some requests are changing data: a customer places an order, an administrator updates inventory, a payment is processed, or a shipment is marked as delivered. Other requests simply read data: customers search for products, managers open dashboards, support agents review order histories, and executives generate monthly reports.

At first, it’s tempting to handle both using the same database tables and the same application models. After all, an order is an order. Whether you’re creating it or displaying it, why shouldn’t the same model work for both? For many applications, that’s exactly what happens. The same entity is used for inserting records, updating records, validating business rules, and serving data to the user interface.

As applications grow, however, this approach begins to reveal its limitations. The information required to create an order is often very different from the information required to display one. Creating an order might require validating inventory, calculating taxes, applying discounts, verifying payment details, and enforcing business rules. Displaying an order, on the other hand, may require customer information, shipping updates, product images, payment status, warehouse progress, and delivery estimates, all combined into a single view optimized for the user.

Trying to satisfy both responsibilities with one model often leads to increasingly complicated code. The write model becomes cluttered with fields needed only for reporting, the read model becomes constrained by rules that exist only for data modification, queries grow more complex, performance begins to suffer, and developers start adding workarounds that make the system harder to maintain.

Eventually, an important realization emerges.

The way we write data isn’t necessarily the best way to read it.

That simple observation led to a pattern known as Command Query Responsibility Segregation, more commonly called CQRS. Rather than forcing one model to satisfy two very different responsibilities, CQRS separates them completely. Commands become responsible for changing data, queries become responsible for reading data, and each side can then evolve independently, optimized for its own purpose. Before exploring how CQRS works, let’s first understand why combining reads and writes into the same model eventually becomes a problem.

Why One Model Eventually Becomes a Problem

When most applications begin, life is simple. Suppose you’re building a loan management platform. A borrower submits a loan application. The application validates the input, saves the record, and later retrieves the same information whenever a loan officer opens the application. The same model handles both writing and reading.

LoanApplication

Everything works perfectly. As the platform grows, however, different users begin asking for different views of the same information. A loan officer wants to see repayment history, guarantors, collateral, risk score, and supporting documents on one screen; finance wants reports showing outstanding balances grouped by branch; executives want dashboards displaying portfolio performance; and customers simply want to know whether their application has been approved.

Notice what’s happening: everyone is looking at the same business entity, but nobody wants exactly the same data. To satisfy these different requirements, the application gradually starts joining more tables.

Loan

JOIN Customer

JOIN Branch

JOIN RiskAssessment

JOIN LoanOfficer

JOIN Repayments

JOIN Documents

JOIN Guarantors

The queries become larger, response times increase, and the code becomes harder to maintain. Ironically, the information required to display a loan has become far more complicated than the information required to create one. This is where many systems begin struggling. The write model keeps accumulating fields that only reports need, the read model becomes constrained by business rules that only matter during updates, and eventually one model is trying to solve two completely different problems.

Commands and Queries Are Different

One of the core ideas behind CQRS is recognizing that not every request has the same purpose. Some requests change data; others simply read it. These are fundamentally different operations. A command tells the system to perform an action. For example:

• Create Order
• Approve Loan
• Reserve Inventory
• Charge Payment
• Register Customer

Commands represent intent. They usually contain business validation, enforce rules, and modify application state. A query, on the other hand, doesn’t change anything. Its only responsibility is returning information. For example:

• Get Customer Profile
• List Outstanding Loans
• View Order History
• Search Products
• Display Dashboard

Queries don’t perform business logic. They answer questions. This distinction may seem small, but in reality, it changes how applications are designed.

What Is CQRS?

CQRS stands for Command Query Responsibility Segregation. Despite the intimidating name, the underlying idea is surprisingly simple. Instead of using one model for everything, CQRS separates the write side from the read side. Commands become responsible for modifying data, and queries become responsible for retrieving it. Conceptually, the architecture looks like this.

                Application

                     │

      ┌──────────────┴──────────────┐

      │                             │

   Commands                     Queries

      │                             │

Write Model                  Read Model

      │                             │

Database                 Optimized View

The important thing to notice is that the read model no longer has to look like the write model. Each side is free to evolve independently.

The Write Model

The write model exists to protect business rules. Suppose a customer places an order. The application needs to verify inventory, calculate discounts, validate payment, reserve stock, and create the order. All of those steps belong on the write side. The write model isn’t concerned with how information will later appear on a dashboard. Its only responsibility is ensuring the business operation is correct. Think of it as the gatekeeper for your data. Nothing enters the system without passing through the write model.

The Read Model

The read model has a completely different job. Its responsibility isn’t enforcing business rules, it’s returning information as efficiently as possible. Imagine displaying an order summary. The customer expects to see:

• Order number
• Customer name
• Product images
• Shipping status
• Payment status
• Delivery estimate
• Total price

The write model probably stores all of this across multiple tables, but the read model doesn’t have to. Instead, it can store exactly the shape required by the user interface, so instead of executing six joins every time someone opens an order, the read model may already contain everything in one place.

This dramatically simplifies queries while improving performance.

Why This Improves Performance

Suppose an online store receives ten thousand requests every minute. Only five hundred of those requests create or update orders. The remaining nine thousand five hundred simply display information. Traditional CRUD applications often force both workloads through the same models and database structures. CQRS recognizes that reads and writes have completely different characteristics. Reads are usually far more frequent, while writes are usually more complicated. By separating them, each side can be optimized independently: the write model focuses on correctness, the read model focuses on speed, and neither compromises the other.

A Real-World Example

Think about YouTube. Uploading a video and watching a video are two completely different operations. Uploading requires validation, virus scanning, metadata extraction, thumbnail generation, transcoding, and storage. Watching a video requires none of those things. The viewer simply wants the video to start playing immediately. Trying to optimize both operations using exactly the same model would make little sense.

CQRS applies the same principle to business applications. The model responsible for creating data doesn’t have to be the same model responsible for presenting it. Recognizing that difference is the first step toward understanding why CQRS has become such a popular architectural pattern in modern backend systems.

At this point, we’ve separated reads from writes conceptually. The next question naturally follows:

If the read model and write model are separate, how do they stay synchronized?

Keeping the Read Model Up to Date

One of the first questions developers ask after learning about CQRS is:

“If my read model is separate from my write model, how does it stay up to date?”

The answer depends on the architecture, but in most modern systems, the read model is updated using events.

Suppose a customer places an order. The write model validates the request, checks inventory, processes payment, and commits the transaction. Once the transaction succeeds, an event such as OrderCreated is published, and one or more components responsible for maintaining the read model receive that event and update their own optimized view of the data.

Conceptually, the flow looks like this:

Customer

      │

      ▼

Command

(Create Order)

      │

      ▼

Write Model

      │

      ▼

Database

      │

      ▼

OrderCreated Event

      │

      ▼

Read Model Updated

      │

      ▼

User Queries Data

Notice something important: the user’s query never touches the write model. Instead, it reads from a model specifically designed for displaying information.

Eventual Consistency

Because the read model is updated after the write completes, there is usually a short delay before the latest information becomes visible. This is known as eventual consistency. Imagine placing an order on a large e-commerce platform. The checkout page immediately confirms that your purchase was successful, but if you refresh your order history a fraction of a second later, the new order might not appear immediately. A moment later, it does. Nothing is wrong: the write completed instantly, and the read model simply needed a short amount of time to catch up. For many applications, this tiny delay is perfectly acceptable. Users rarely notice a difference measured in milliseconds or even a few seconds, and the benefit is that reads become dramatically faster and easier to scale.

Does CQRS Require Microservices?

One of the biggest misconceptions about CQRS is that it only works in microservice architectures. It doesn’t. CQRS is simply a design pattern. A single monolithic application can separate its command handlers from its query handlers just as effectively as a distributed system. Likewise, CQRS doesn’t require Kafka, RabbitMQ, Event Sourcing, or multiple databases. Many applications implement CQRS using a single database while maintaining separate command and query models inside the same application. As systems grow, those models may eventually evolve into separate databases or services, but that’s a scaling decision, not a requirement of the pattern itself.

When Should You Use CQRS?

CQRS isn’t a solution to every problem. Many applications work perfectly well using traditional CRUD architecture. If your application has simple business rules, relatively small datasets, and straightforward queries, introducing CQRS often adds unnecessary complexity. On the other hand, CQRS becomes increasingly valuable when reads and writes have very different characteristics. Common examples include:

• High-traffic e-commerce platforms.
• Banking and financial systems.
• Loan management platforms.
• Logistics and supply chain applications.
• Reporting and analytics dashboards.
• SaaS products with complex administrative views.

In these systems, the information required to display data is often very different from the information required to modify it. Separating those responsibilities makes the application easier to optimize and easier to maintain.

CQRS vs Traditional CRUD

A useful way to understand CQRS is to compare it with the architecture most developers already know.

Neither approach is universally better. CRUD is an excellent choice for many applications. CQRS becomes valuable when the complexity of the domain begins to outweigh the simplicity of using a single model.

Common Mistakes

One mistake developers frequently make is adopting CQRS simply because they’ve heard it’s a “best practice.” Like every architectural pattern, CQRS introduces additional moving parts. You’ll often have separate models, additional event handling, and the possibility of eventual consistency. If those complexities don’t solve a real business problem, they’re simply unnecessary overhead. Another common mistake is trying to create one read model that satisfies every possible screen. The real strength of CQRS lies in allowing each query to have a model optimized for its own purpose. A customer dashboard, an administrative report, and a mobile application may each deserve different read models. Finally, don’t confuse CQRS with Event Sourcing. Although the two patterns are often used together, they solve different problems. CQRS separates reads from writes, while Event Sourcing stores state as a sequence of events. You can implement one without the other.

Bringing It All Together

Throughout this series, we’ve gradually built a collection of patterns that solve different reliability and scalability challenges.

Notice how the series has gradually expanded in scope. We began by making individual API requests reliable, then learned how to coordinate transactions, communicate between services, and manage distributed business workflows. CQRS adds another important lesson: sometimes the best way to scale an application isn’t by making one model do everything, but by giving different responsibilities to different models.

Final Thoughts

One of the biggest lessons in software architecture is that different problems deserve different solutions. Reading data and writing data may involve the same business entity, but they rarely have the same requirements. Writes prioritize correctness, validation, and enforcing business rules; reads prioritize speed, simplicity, and delivering information in the shape users actually need. CQRS embraces this difference instead of trying to hide it. For small applications, a traditional CRUD architecture is often the right choice. As systems become larger and business requirements become more demanding, separating reads from writes can lead to simpler queries, clearer responsibilities, and applications that scale far more gracefully. Like every pattern we’ve explored in the Beyond CRUD series, CQRS isn’t about making software more complicated. It’s about choosing the right level of complexity to solve the problem in front of you.

What’s Next?

So far, we’ve explored several patterns that improve reliability, scalability, and maintainability. One question still remains: how do all these patterns fit together to build applications where services communicate entirely through events instead of direct API calls?

In the next article, we’ll explore Event-Driven Architecture Explained: Building Systems That React to Events, where we’ll connect concepts like the Outbox Pattern, Saga Pattern, and CQRS into a cohesive architectural style used by many modern distributed systems.

“A transaction can roll back one database. A Saga coordinates many databases that have never even met.”

Imagine you’re building an online marketplace. A customer clicks Place Order, expecting what feels like a single operation. Behind the scenes, however, that simple button sets off a chain of events involving several independent services. The Order Service creates a new order, the Inventory Service reserves the purchased items, the Payment Service charges the customer’s card, the Shipping Service schedules delivery, and the Notification Service sends a confirmation email.

From the customer’s perspective, it’s one transaction. From your system’s perspective, it’s anything but. Each service owns its own database, each commits its own transaction independently, none of them has direct control over the others, and no single database transaction can span all of them.

Now imagine the following sequence: the Order Service successfully creates the order, the Inventory Service reserves the last laptop in stock, and the Payment Service charges the customer’s credit card. Then, just as the Shipping Service begins preparing the shipment, it discovers that delivery isn’t available to the customer’s location. The order already exists, the customer’s card has already been charged, and inventory has already been reserved, but the shipment can never be created. Unlike a traditional database transaction, there is no single ROLLBACK command capable of undoing work performed across multiple independent databases.

This is one of the defining challenges of distributed systems. As applications evolve from monoliths into microservices, business processes increasingly span services that are independently deployed, independently scaled, and independently owned. While this architecture offers tremendous flexibility, it also introduces a difficult question:

How do you maintain business consistency when a workflow spans multiple services and one of them fails halfway through?

The answer is not a larger transaction. It’s a different way of thinking. Instead of trying to make every service commit simultaneously, modern distributed systems break long-running business processes into a sequence of smaller local transactions. If every step succeeds, the workflow completes successfully. If a step fails, previously completed work is undone using carefully designed compensating actions rather than database rollbacks.

This approach is known as the Saga Pattern. Like the Outbox Pattern, the Saga Pattern embraces the reality that failures are inevitable. Rather than pretending distributed transactions behave like local database transactions, it provides a practical way to recover when things don’t go according to plan. Before we explore how Sagas work, it’s important to understand why traditional transactions stop working once your application crosses service boundaries.

Why Database Transactions Don’t Scale Across Microservices

Earlier in this series, we explored database transactions and learned how they guarantee that multiple operations either succeed together or fail together. If an online banking application deducts money from one account and credits another within the same database, a transaction ensures that both operations are treated as a single unit of work. If anything fails before the transaction commits, every change is rolled back automatically, leaving the database in a consistent state.

That model works beautifully when everything happens inside one database, but microservices change the picture completely. Imagine an order workflow involving four independent services.

Customer

      │

      ▼

Order Service

      │

      ▼

Inventory Service

      │

      ▼

Payment Service

      │

      ▼

Shipping Service

Each service owns its own database. The Order Service cannot directly roll back changes made by the Payment Service, the Payment Service cannot undo inventory reservations, and the Shipping Service has no authority over the Order database. Every service commits its own transaction independently. This independence is one of the greatest strengths of microservices, and it’s also one of their biggest challenges. Suppose the workflow progresses like this:

Create Order ✅

↓

Reserve Inventory ✅

↓

Charge Payment ✅

↓

Create Shipment ❌

At this point, three services have already committed their work, and nothing can simply “roll back.” Unlike a single database transaction, there is no global undo button. This challenge is often referred to as a distributed transaction, and solving it using traditional database techniques quickly becomes impractical.

Years ago, distributed systems experimented with approaches such as Two-Phase Commit (2PC), where every participating system agreed to either commit or roll back together. While theoretically elegant, 2PC introduced significant coordination overhead, reduced availability, increased latency, and created situations where entire systems could become blocked waiting for slow or unavailable participants. Modern cloud-native architectures generally avoid this approach. Instead of attempting to make every service commit simultaneously, they accept that each service commits independently and focus on coordinating the overall business process.

That’s exactly what the Saga Pattern does. Instead of one large transaction, a Saga is a sequence of smaller transactions linked together by business logic. If every step succeeds, the Saga completes successfully; if one step fails, previously completed steps are compensated through additional business actions designed to reverse their effects. That distinction is subtle but incredibly important: a Saga doesn’t roll back database transactions, it performs new transactions whose purpose is to restore the system to a valid business state. Understanding that difference is the key to understanding everything else about the Saga Pattern.

Understanding Compensating Transactions

One of the biggest misconceptions developers have when they first encounter the Saga Pattern is assuming it somehow provides a distributed version of ROLLBACK. It doesn’t, and in fact, that’s one of the defining characteristics of a Saga. Once a service commits its local transaction, that transaction is permanent. The database has already saved the changes, and there is no mechanism for another service to rewind history. Instead of rolling back completed work, a Saga performs compensating transactions. A compensating transaction is simply another business operation whose purpose is to undo the effects of a previous one.

Suppose an order has already been created, inventory has been reserved, and payment has been successfully processed. If shipping later fails because the customer’s address falls outside the delivery area, the system cannot ask every database to roll back, because those transactions finished long ago. Instead, the application performs a series of new operations: the payment service issues a refund, the inventory service releases the reserved stock, and the order service changes the order status from Pending to Cancelled.

Notice something important: nothing has been deleted and nothing has been rolled back. The system simply performs additional work that restores the business to a valid state.

Conceptually, the workflow now looks like this:

Create Order ✅

↓

Reserve Inventory ✅

↓

Charge Payment ✅

↓

Create Shipment ❌

↓

Refund Payment

↓

Release Inventory

↓

Cancel Order

This is the heart of the Saga Pattern. Rather than pretending failures never happened, the system accepts them and responds with carefully designed business actions. This approach is much more realistic because, in distributed systems, failures aren’t exceptional, they’re inevitable.

A Banking Example

Imagine a customer applies for a personal loan. Several independent services participate in the approval process: the Loan Service creates the application, the Credit Service performs a credit check, the Risk Service evaluates affordability, and the Notification Service informs the customer of the decision. Everything proceeds normally until the Risk Service determines that the customer’s debt-to-income ratio exceeds the organization’s lending policy. At this point, the application cannot continue. If this were a single database transaction, we would simply issue a rollback.

In a microservice architecture, however, the Loan Service has already committed the new application and the Credit Service has already stored the completed credit assessment. Neither service can magically erase its work because another service encountered a problem. Instead, the Saga performs compensating actions: the Loan Service marks the application as withdrawn, the Credit Service archives its assessment, and the Notification Service informs the customer that the application could not proceed. Each action is itself a normal transaction, and collectively they restore the overall business process to a consistent state.

Rollback vs Compensation

Although these ideas sound similar, they’re fundamentally different. A database rollback behaves like this:

BEGIN

↓

Update Balance

↓

Insert Payment

↓

Failure

↓

ROLLBACK

When the rollback occurs, the database behaves as though none of the changes ever happened. It’s as if the transaction never existed. A Saga works very differently.

Reserve Inventory

↓

Charge Payment

↓

Shipment Fails

↓

Refund Payment

↓

Release Inventory

The original payment really happened, the refund also really happened, and both become part of the permanent history of the system. That’s an important distinction, and many business domains actually require this behavior. Consider financial systems. Deleting payment records would make auditing impossible, so recording both the payment and the subsequent refund creates a complete, traceable history of what occurred. Compensation isn’t about pretending mistakes never happened, it’s about correcting them transparently.

Designing Good Compensating Actions

Writing a compensating transaction isn’t simply a matter of reversing database changes. You’re reversing business operations. For example, suppose a hotel booking system reserves a room.

The compensating action isn’t:

Delete reservation row.

It’s:

Release the room back into available inventory.

Similarly, if an airline charges a customer’s credit card, the compensation isn’t:

Delete payment record.

It’s:

Create a refund transaction.

This distinction matters because business systems are usually audited. Historical events should remain visible, and what changes is the current business state. Whenever you design a Saga, a useful question to ask is:

“If this step succeeds but a later step fails, what business action restores the system to a valid state?”

Thinking in terms of business operations rather than database updates leads to much more reliable designs.

Every Step Is Independent

Another characteristic of Sagas is that every step represents a complete, independent transaction. Suppose an order workflow consists of four services.

Order Service

↓

Inventory Service

↓

Payment Service

↓

Shipping Service

Each service commits its own database transaction before the next service begins, which means failures are isolated. If the Shipping Service experiences an outage, it doesn’t corrupt the Payment Service’s database; likewise, if the Payment Service fails, it doesn’t leave the Inventory Service with an unfinished SQL transaction.

Each service remains responsible for its own data, and the Saga simply coordinates how those independent pieces fit together. This separation of responsibility is one of the reasons microservice architectures scale so well. Services remain loosely coupled while still participating in larger business workflows.

Thinking in Business Processes

One of the biggest mindset shifts when working with Sagas is realizing that you’re no longer designing database transactions. You’re designing business processes. Database transactions answer questions like:

“How do I keep these SQL statements consistent?”

Sagas answer a much broader question:

“How does my business recover when part of this workflow succeeds and another part fails?”

That’s why Sagas often involve business concepts rather than technical ones: refunds, reservation cancellations, order cancellations, inventory releases, and account reversals. These aren’t database operations; they’re business operations that happen to involve databases. Once you begin thinking at that level, the Saga Pattern becomes much easier to understand because it mirrors how real businesses operate. Companies don’t erase history when something goes wrong, they perform additional actions to correct it, and software simply follows the same principle.

Two Ways to Coordinate a Saga

Now that we understand what a Saga is, another important question emerges: Who is responsible for coordinating all these steps?

Imagine once again that a customer places an order. The Order Service creates the order, the Inventory Service reserves the stock, the Payment Service charges the customer, and the Shipping Service prepares the shipment. If everything succeeds, the Saga completes; if one step fails, compensating transactions begin. Someone, or something, must decide what happens next.

There are two common ways to achieve this coordination:

• Choreography
• Orchestration

Both accomplish the same goal, but they do so in very different ways.

Choreography

Think about a group of experienced dancers performing together. Nobody stands at the front giving instructions. Each dancer knows exactly when to move because they respond to the music and to one another. Saga choreography works in much the same way. Instead of a central coordinator directing every step, each service reacts to events published by other services. Consider our order workflow.

Customer Places Order

        │

        ▼

Order Service

Publishes

OrderCreated

        │

        ▼

Inventory Service

Publishes

InventoryReserved

        │

        ▼

Payment Service

Publishes

PaymentCompleted

        │

        ▼

Shipping Service

Publishes

ShipmentCreated

Every service only knows two things:

• The events it listens for.
• The events it publishes.

The Payment Service doesn’t know the Shipping Service exists, and the Shipping Service doesn’t know anything about the Inventory Service. Each service simply reacts whenever an event arrives. This loose coupling is one of choreography’s greatest strengths. Because services know very little about one another, adding or removing services often becomes much easier.

Suppose your business introduces a Loyalty Service that awards reward points whenever an order is completed. Nothing else needs to change. The new service simply subscribes to the PaymentCompleted event, and the rest of the system continues operating exactly as before. This flexibility makes choreography extremely attractive in event-driven architectures. However, it comes with a cost. As systems grow, understanding the overall business workflow becomes increasingly difficult. Instead of one visible process, the Saga becomes scattered across many services.

To understand why a shipment wasn’t created, you may need to inspect logs from the Order Service, Inventory Service, Payment Service, Shipping Service, and Notification Service. The workflow still exists, it’s simply distributed across the entire system.

Orchestration

Now imagine the same dancers performing with a conductor standing at the front. Instead of reacting to one another, every performer follows instructions from a single leader. This is orchestration. Rather than allowing services to coordinate themselves, a dedicated component, often called the Saga Orchestrator, controls the entire workflow. The orchestrator tells each service what to do next.

Saga Orchestrator

        │

        ▼

Create Order

        │

        ▼

Reserve Inventory

        │

        ▼

Charge Payment

        │

        ▼

Create Shipment

If every step succeeds, the orchestrator declares the Saga complete. If a failure occurs, it explicitly instructs previous services to execute their compensating transactions.

Shipment Failed

        │

        ▼

Refund Payment

        │

        ▼

Release Inventory

        │

        ▼

Cancel Order

Unlike choreography, every decision is visible in one place. Need to understand the business process? Read the orchestrator. Need to change the workflow? Modify one component instead of updating several independent services. This centralized view makes orchestration particularly appealing for complex business processes involving many steps, conditional logic, or approval workflows. The trade-off, however, is tighter coupling. The orchestrator must understand every participating service, making it more aware of the overall system than any individual service would be in a choreographed Saga. Neither approach is universally better. The right choice depends on the complexity of the workflow and the level of control your application requires.

Choreography vs Orchestration

A useful way to compare them is to think about where the business logic lives. With choreography, the workflow is distributed across many services, and every service contributes a small piece of the overall process by responding to events. With orchestration, the workflow lives inside a single coordinator that explicitly directs every participant.

Small event-driven systems often benefit from choreography because new consumers can easily subscribe to existing events. Larger enterprise workflows frequently choose orchestration because business rules remain easier to understand and maintain.

Real-World Examples

By now, you’ve probably encountered situations where the Saga Pattern would be useful without realizing it. An online retailer processing orders, a loan management platform approving applications, an airline booking system reserving flights, or a hotel reservation platform coordinating room availability all involve multiple independent services participating in a single business process. Consider a digital lending platform. When a borrower accepts a loan offer, several services may participate:

• The Loan Service creates the loan account.
• The Disbursement Service sends funds.
• The Accounting Service records journal entries.
• The Notification Service sends an SMS.
• The Credit Bureau Service updates the borrower’s status.

If the disbursement fails because the customer’s bank account is invalid, the system shouldn’t leave behind a partially created loan. Instead, compensating actions mark the loan as cancelled, reverse accounting entries where necessary, and notify the customer that disbursement was unsuccessful. No database rollback spans all these services, and the Saga restores business consistency through coordinated business actions.

Common Mistakes

The Saga Pattern is powerful, but it’s not a silver bullet. One common mistake is trying to treat compensating transactions as database rollbacks. They aren’t. Compensation should reverse business effects, not erase history. Another mistake is making Saga steps too large. Every local transaction should remain focused and complete quickly. Long-running database transactions reduce scalability and increase the likelihood of contention. It’s also important to remember that messaging is rarely perfect. Duplicate events, delayed delivery, and retries are normal in distributed systems. Every participating service should therefore be designed with idempotency in mind. Finally, don’t introduce a Saga simply because your application uses microservices. If a workflow only involves one service and one database, a normal database transaction is usually the simpler and better solution. Sagas solve distributed coordination problems, not ordinary CRUD operations.

Saga Pattern vs Traditional Transactions

At first glance, Sagas and database transactions appear to solve similar problems. In reality, they operate at completely different levels. A database transaction guarantees consistency within a single database; a Saga guarantees business consistency across multiple independent services. One relies on rollback, the other relies on compensation. One typically completes in milliseconds, while the other may run for several minutes, or even hours, depending on the business process. They’re not competing approaches. As you’ve seen throughout this series, they complement one another. In fact, a single Saga step usually contains its own local database transaction.

Bringing It All Together

At this point in the Beyond CRUD series, we’ve gradually built a toolkit for designing reliable backend systems. Each concept answers a different engineering question.

Notice how every concept builds upon the previous one. None replaces the others, and reliable distributed systems emerge when these patterns work together.

Final Thoughts

Building software inside a single database is relatively straightforward. Building software that spans dozens of independent services is something else entirely. The challenge isn’t simply writing correct code, it’s ensuring that business processes continue making sense even when networks fail, services restart, or one step succeeds while another doesn’t.

The Saga Pattern embraces these realities rather than fighting them. Instead of relying on one enormous transaction that spans every service, it coordinates many smaller transactions while providing a structured way to recover when failures occur. That mindset has become one of the defining characteristics of modern cloud-native architecture. As your systems continue growing, you’ll discover that reliable software isn’t built by eliminating failures, it’s built by designing systems that expect failures and know exactly how to recover from them.

What’s Next?

So far, we’ve focused primarily on making writes reliable, but as applications grow, another challenge begins to emerge. Reading data efficiently often requires very different models from writing it. Should the same model be responsible for both, or should we optimize reads and writes independently?

In the next article, we’ll explore CQRS Explained: Separating Reads and Writes for Scalable Systems, a pattern that allows applications to scale, simplify complex queries, and build richer user experiences without overloading their write models.

“A database transaction can guarantee your data is correct. It cannot guarantee the rest of your system knows about it.”

Imagine you’re building an e-commerce platform. A customer places an order, and your application begins processing the request. Inside a database transaction, it creates the order, deducts the purchased items from inventory, records the payment, and commits the transaction successfully. From the perspective of the database, everything has gone exactly as planned. Every change has been saved, every business rule has been respected, and the transaction completes without error.

If this were a monolithic application, the story might end there, but modern software rarely consists of a single application. The moment an order is created, several other systems need to react. A shipping service must prepare the package, an email service needs to send an order confirmation, an analytics platform wants to record another sale, a loyalty service may award reward points, and an accounting system might need to generate journal entries. Rather than constantly querying the orders table looking for changes, these services usually rely on events published through a message broker such as Kafka, RabbitMQ, Azure Service Bus, or Amazon SQS.

A straightforward implementation seems almost obvious: once the transaction commits successfully, publish an OrderCreated event.

BEGIN TRANSACTION

↓

Create Order

↓

Reduce Inventory

↓

Record Payment

↓

COMMIT

↓

Publish OrderCreated Event

At first glance, there’s nothing wrong with this approach, until something fails. Suppose the database transaction commits successfully, permanently saving the customer’s order. A fraction of a second later, however, Kafka becomes temporarily unavailable. Perhaps RabbitMQ disconnects, or maybe the application crashes before it can publish the event.

The order now exists in the database, the customer’s payment has been processed, and inventory has already been reduced, yet none of the downstream services know the purchase ever happened. The warehouse never receives instructions to prepare the shipment, the customer never receives a confirmation email, the analytics dashboard quietly reports incorrect sales figures, and the accounting system never records the transaction.

Nothing inside the database is inconsistent. The inconsistency exists between systems.

This subtle failure is one of the most common reliability problems in distributed architectures. It’s known as the Dual-Write Problem, and it’s surprisingly easy to introduce because writing to a database and publishing an event are two completely independent operations. One can succeed while the other fails, leaving different parts of the system with different versions of reality.

In our previous articles, we’ve explored how transactions protect database operations, how isolation levels govern concurrent access to data, and how distributed locks coordinate work across multiple application servers. The Outbox Pattern builds on those concepts by solving a different challenge: ensuring that once your database commits a change, the rest of your system will eventually learn about it as well.

Before we look at the solution, let’s first understand why this seemingly simple problem is much harder than it appears.

The Dual-Write Problem

The Dual-Write Problem occurs whenever an application attempts to update two independent systems as part of a single business operation. One system is usually your database; the other is typically a message broker, search index, cache, analytics platform, or another external service. Although these updates feel like one logical operation, they are technically two completely separate actions.

Consider an online banking application. When a customer transfers money, the application updates account balances inside the database. Afterward, it publishes a MoneyTransferred event so other services can respond. A fraud detection system may inspect the transaction, a notification service may send an SMS, and an accounting platform may update its ledgers.

Conceptually, all of this represents one business event, but technically, it looks more like this:

Update Database

↓

Publish Event

The problem is that neither system knows anything about the other. Your database has no idea whether Kafka accepted the message, and Kafka has no knowledge of whether your SQL transaction committed successfully. They’re completely independent, and that independence creates four possible outcomes.

The first two outcomes are easy to reason about, but it’s the final two that create serious problems. If an event is published for data that never commits, downstream services begin processing information that doesn’t actually exist. Equally dangerous is the opposite scenario, where the database commits successfully but the event is never published. From that moment onward, every service in the architecture has a different understanding of reality. This isn’t merely an inconvenience; it’s a consistency problem that becomes increasingly difficult to detect as systems grow.

Why Transactions Can’t Solve This

A natural question follows: why not simply include the message broker inside the database transaction? Unfortunately, that’s not how database transactions work. Transactions provide atomicity because every operation is coordinated by the same database engine. The database controls when the transaction begins, when it commits, and when it rolls back. Every SQL statement participates in that process because they’re all executed by the same system.

A message broker lives outside that boundary. Kafka doesn’t participate in your PostgreSQL transaction, RabbitMQ doesn’t know your SQL Server transaction exists, and PostgreSQL has no mechanism for asking Kafka whether a message was published successfully before committing the transaction. In other words, there is no shared transaction manager coordinating both systems.

Years ago, technologies such as Two-Phase Commit (2PC) attempted to solve this problem by coordinating transactions across multiple systems. Although theoretically appealing, they introduced significant complexity, increased latency, reduced availability, and often became bottlenecks in distributed environments.

As microservices became more popular, the industry gradually moved toward simpler, more resilient approaches. Rather than trying to make two systems commit simultaneously, engineers began asking a different question:

What if we only committed to one system first and made the second system eventually consistent?

That shift in thinking led to one of the most influential patterns in modern backend architecture: the Outbox Pattern.

What Is the Outbox Pattern?

The Outbox Pattern solves the Dual-Write Problem by changing the order in which work is performed. Instead of attempting to update the database and publish an event as part of the same operation, the application first commits everything it needs to the database, including the event itself. That last part is the key. Rather than sending the event directly to Kafka or RabbitMQ, the application writes the event into a dedicated database table commonly known as the outbox.

Because the business data and the outbox record are written inside the same database transaction, they succeed or fail together. If the transaction commits, both the business data and the event are safely stored; if it rolls back, neither exists. Only after the transaction has completed does another process read events from the outbox table and publish them to the message broker.

Conceptually, the workflow changes from this:

Update Database

↓

Publish Event

to this:

BEGIN TRANSACTION

↓

Update Business Data

↓

Insert Event Into Outbox

↓

COMMIT

↓

Background Publisher Reads Outbox

↓

Publish Event

↓

Mark Event As Published

At first glance, this may seem like a small adjustment, but in reality, it completely changes the reliability characteristics of your application. The application is no longer trying to coordinate two independent systems within the same request. Instead, it commits to a single source of truth, the database, and lets another process handle communication with external systems afterward. If the message broker is temporarily unavailable, nothing is lost. The event is already safely stored inside the database and simply waits until the publisher can deliver it.

Understanding the Outbox Table

The outbox itself is usually nothing more than a normal database table. A simplified version might look like this:

Each row represents an event that should eventually be delivered. Notice the published_at column: rows where this value is NULL haven’t yet been published, while rows with a timestamp have already been successfully delivered.

The outbox isn’t intended to become a permanent event store. Instead, it acts as a reliable staging area between your transactional database and your messaging infrastructure. Once an event has been published successfully, and your retention policy allows, it can be archived or deleted.

A Complete Walkthrough

Let’s revisit our e-commerce example. A customer purchases a laptop. Inside a single transaction, the application performs three operations: first, it creates the order; second, it deducts one item from inventory; and finally, instead of publishing an event immediately, it inserts a new row into the outbox table describing what happened.

BEGIN TRANSACTION

↓

INSERT Order

↓

UPDATE Inventory

↓

INSERT Outbox Event

↓

COMMIT

At this point, the customer’s purchase has been committed successfully, and just as importantly, the event describing that purchase has also been committed. Suppose Kafka goes offline immediately afterward: the request still succeeds, nothing has been lost, and the application doesn’t need to roll back the order because the event hasn’t disappeared. It is sitting safely inside the outbox table waiting to be delivered.

A separate publisher service periodically checks the outbox. When Kafka becomes available again, it publishes the event and updates the record to indicate that delivery succeeded.

Outbox Publisher

↓

Read Unpublished Events

↓

Publish To Kafka

↓

Success?

      │

   Yes │ No

      │

Mark Published

      │

Retry Later

Notice something important: the user’s request is no longer responsible for talking to Kafka. Its only responsibility is ensuring that the event is safely recorded, and publishing becomes a separate concern. That separation dramatically improves reliability because temporary failures in external systems no longer affect the original business transaction.

Why This Works

The elegance of the Outbox Pattern comes from the fact that it reduces a distributed systems problem to a database problem. Earlier in this series, we spent considerable time discussing transactions and learned that they guarantee a group of related database operations either all succeed or all fail together. The Outbox Pattern deliberately takes advantage of that guarantee: instead of asking a database and a message broker to commit simultaneously, a task they were never designed to perform, it asks only the database to commit. Since both the business data and the outbox record live inside the same database, the transaction naturally guarantees their consistency.

Everything after that becomes a delivery problem rather than a consistency problem. Even if the publisher crashes, Kafka becomes unavailable, or the network experiences intermittent failures, the event remains safely stored. The publisher will eventually retry, and the event will eventually be delivered. The system moves from requiring immediate consistency between the database and the message broker to achieving eventual consistency in a reliable and predictable way. That single shift in mindset is what has made the Outbox Pattern one of the most widely adopted reliability patterns in modern distributed systems.

Why Not Publish Immediately After the Commit?

Some developers look at the Outbox Pattern and ask a reasonable question: “If the transaction has already committed successfully, why not simply publish the event right afterward and retry if it fails?” The problem is that retries only work if the application survives long enough to perform them.

Imagine the following sequence of events:

COMMIT Transaction

↓

Application Crashes

↓

Restart

The order exists, the event was never published, and the application has no memory that it still owes Kafka an event because that information was never persisted anywhere. With an outbox table, the event survives the crash because it was committed alongside the business data. When the publisher starts again, it simply resumes reading unpublished events from the database. The application doesn’t have to remember what happened because the database already does.

This is one of the reasons the Outbox Pattern is so resilient. It relies on durable storage rather than application memory, making it naturally tolerant of crashes, restarts, and temporary infrastructure failures.

How Events Leave the Outbox

By now, we’ve established that the application’s responsibility ends once the business data and the corresponding event have been committed to the database. The next challenge is equally important: How do those events actually reach Kafka, RabbitMQ, or another messaging system?

Broadly speaking, there are two common approaches. The first is a Polling Publisher, where a background worker periodically checks the outbox table for unpublished events. Every few seconds, or even every few milliseconds, it queries the database, publishes any pending events, and marks them as successfully delivered.

Conceptually, the process looks like this:

Background Publisher

↓

Query Outbox

↓

Find Unpublished Events

↓

Publish Event

↓

Success?

     │

 Yes │ No

     │

Mark Published

     │

Retry Later

This approach is simple to understand, easy to implement, and works well for many applications. Since the publisher operates independently of the original request, temporary failures in Kafka or RabbitMQ don’t affect users. If publishing fails, the worker simply retries later.

The second approach is Change Data Capture (CDC). Instead of periodically querying the outbox table, a CDC tool monitors the database’s transaction log and automatically detects newly inserted outbox records. As soon as a new event appears, the tool publishes it to the message broker.

One of the most popular CDC solutions is Debezium, which integrates with databases such as PostgreSQL, MySQL, and SQL Server. Rather than polling the database repeatedly, Debezium continuously streams database changes, reducing unnecessary queries while providing near real-time event publication. Both approaches solve the same problem. Polling is generally simpler and perfectly adequate for many systems, while CDC becomes attractive when applications process very large numbers of events or require lower publishing latency.

Real-World Examples

The Outbox Pattern appears in many more places than developers often realize. Once you begin recognizing the Dual-Write Problem, you start seeing it almost everywhere.

E-Commerce

A customer places an order. The transaction creates the order, updates inventory, and inserts an OrderCreated event into the outbox. Later, the publisher delivers that event to Kafka, and other services respond independently:

• Shipping prepares the package.
• Email sends the confirmation.
• Analytics records the sale.
• Loyalty awards points.

Every service receives exactly the same event without the original request needing to coordinate them.

Loan Management Systems

Suppose a borrower makes a repayment. Inside a transaction, the application updates the loan balance, records the payment, recalculates outstanding interest, and inserts a LoanRepaymentReceived event into the outbox. Once published, other systems can react independently. The accounting service posts journal entries, the notification service sends an SMS receipt, reporting dashboards update repayment statistics, and credit scoring services refresh customer profiles. The repayment itself remains fast because none of these downstream systems participate in the original transaction.

User Registration

A customer creates a new account. The application stores the user’s information and writes a UserRegistered event into the outbox. Later, other services consume the event. An email service sends a welcome message, a CRM platform creates a customer profile, a marketing system subscribes the user to onboarding campaigns, and a recommendation engine begins generating personalized suggestions. The registration endpoint doesn’t need to know anything about those systems; its only responsibility is recording that the registration occurred.

Payment Processing

Imagine a payment gateway confirms a successful transaction. The payment service updates account balances, records the payment, and inserts a PaymentCompleted event. Even if Kafka becomes unavailable immediately afterward, the payment itself isn’t lost. Once messaging resumes, the event is delivered and downstream systems continue exactly where they left off.

Benefits of the Outbox Pattern

One of the reasons the Outbox Pattern has become so widely adopted is that it solves several problems simultaneously. First, it provides reliability. Once an event has been written into the outbox, it becomes durable. Temporary network failures, message broker outages, or application crashes no longer result in permanently lost events. Second, it simplifies application code. Rather than forcing every request to coordinate database updates and message publication, the request focuses exclusively on committing business data, and event publication becomes the responsibility of a dedicated background process. Third, it improves scalability. Because event publication happens asynchronously, user requests complete more quickly, and slow downstream systems no longer delay the original transaction. Finally, it naturally embraces eventual consistency. Instead of requiring every service to update simultaneously, the system guarantees that every interested service will eventually receive the event once communication becomes available.

Common Mistakes

Like every architectural pattern, the Outbox Pattern can be implemented incorrectly.

One common mistake is assuming that publishing an event means the work is finished. Publishing only guarantees that the message reached the broker. Consumers may still fail, messages may still be retried, and downstream services must therefore remain resilient and, where appropriate, idempotent.

Another common mistake is forgetting to clean up the outbox table. If events remain forever, the table will continue growing until queries become unnecessarily expensive. Most production systems archive or remove published events after an appropriate retention period.

Developers also sometimes attempt to perform expensive business logic inside the publisher. The publisher should remain intentionally simple. Its responsibility is publishing events, not recalculating business rules or modifying application state.

Finally, don’t assume every database change requires an event. Publishing unnecessary events creates noise, increases infrastructure costs, and makes systems more difficult to understand. Good events represent meaningful business occurrences, not individual SQL statements.

Outbox Pattern vs Distributed Transactions

At first glance, the Outbox Pattern and distributed transactions appear to solve the same problem. Both attempt to coordinate work across multiple systems, but the difference lies in how they approach consistency. Distributed transactions attempt to make every participating system commit or roll back together. The Outbox Pattern accepts that this is often impractical in distributed architectures. Instead, it commits business data first and guarantees that events will eventually be delivered. This approach sacrifices immediate consistency in exchange for simplicity, resilience, and availability. For modern cloud-native applications, that trade-off is often the better engineering decision.

Outbox Pattern vs Event Sourcing

The Outbox Pattern is also frequently confused with Event Sourcing. Although both involve events, they solve entirely different problems. With the Outbox Pattern, the database remains the primary source of truth, and events simply communicate that something has happened. With Event Sourcing, events are the source of truth. Instead of storing the current state of an order or account, the system stores every event that led to its current state, and the application reconstructs state by replaying those events.

An outbox event might say:

Order Created.

An event-sourced system would permanently record every event throughout the order’s lifetime:

• Order Created
• Payment Authorized
• Inventory Reserved
• Shipment Prepared
• Order Delivered

Both patterns involve events, but only Event Sourcing uses them to reconstruct application state.

Best Practices

If you’re considering adopting the Outbox Pattern, several practices consistently lead to reliable implementations.

Keep the outbox in the same database as your business data so that both participate in the same transaction.

Design your publisher to be idempotent wherever possible. Network failures and retries are inevitable, and duplicate publication attempts should never produce incorrect results.

Monitor the outbox table. A growing backlog of unpublished events is often the earliest warning sign that something is wrong with your messaging infrastructure.

Treat event schemas as part of your public contract. Once other services depend on them, changing them carelessly becomes just as risky as changing a public API.

Finally, remember that publishing an event doesn’t guarantee it has been processed. Downstream services should always be designed to handle retries, duplicates, and temporary failures gracefully.

Bringing It All Together

At this point in the series, we’ve explored several techniques for building reliable software systems. Each one addresses a different question.

Notice the progression: the concepts don’t replace one another, they build upon one another. Modern backend systems are reliable precisely because they combine multiple patterns, each solving a different class of failure.

Final Thoughts

One of the defining characteristics of distributed systems is that communication eventually fails. Networks become unreliable, applications restart, and message brokers experience outages. Trying to eliminate these failures entirely is unrealistic.

The Outbox Pattern embraces this reality by changing the problem. Instead of attempting to guarantee that two independent systems succeed simultaneously, it guarantees that business data is safely committed first and that communication will eventually catch up. This seemingly small architectural decision dramatically improves reliability because it removes timing from the equation. Your application no longer depends on Kafka, RabbitMQ, or another messaging system being available at the exact moment a customer submits a request. Instead, it relies on something your database already does exceptionally well: storing data reliably.

The next time you find yourself writing code that updates a database and immediately publishes an event, pause for a moment and ask yourself:

“What happens if my database succeeds but my message broker doesn’t?”

If the answer is “my systems become inconsistent,” you’ve probably found a place where the Outbox Pattern belongs.

What’s Next?

So far in this series, we’ve focused on making individual operations reliable. But what happens when a single business process spans multiple microservices, each with its own database and transaction?

Imagine placing an order that requires the payment service, inventory service, shipping service, and notification service to all complete successfully. What happens if one of those services fails halfway through?

In the next article, we’ll explore the Saga Pattern, one of the most widely used approaches for coordinating long-running business transactions across distributed systems.

Where do these locks actually live?

Unlike database transactions, distributed locks cannot rely on the memory of a single application server. Remember, our application might be running on five, ten, or even hundreds of machines, and every server must consult the same source of truth before deciding whether it may perform a particular operation.

Over the years, several technologies have emerged to solve this coordination problem. Although they all provide distributed locking capabilities, they were designed with slightly different goals in mind. Let’s look at the most common ones.

Redis

For most web applications, Redis is by far the most popular choice. Originally designed as an in-memory data store, Redis is incredibly fast, making it an excellent candidate for lightweight coordination tasks.

Acquiring a lock in Redis is surprisingly straightforward. A server attempts to create a key using an atomic command that says, in effect:

“Create this key only if it doesn’t already exist.”

If Redis successfully creates the key, the server owns the lock; if the key already exists, another server is already performing the work. Because Redis executes this operation atomically, two servers can never successfully create the same lock at the same time.

A simplified example looks like this:

SET invoice-generation

Server-A

NX

EX 30

The command says:

• Create the key only if it doesn’t already exist (NX).
• Automatically expire it after thirty seconds (EX 30).

In one atomic operation, Redis both acquires the lock and ensures it won’t remain forever if the server crashes. For many applications, this is all that’s needed.

Why Redis Is So Popular

Redis has become the default choice for distributed locks because it satisfies three important requirements. First, it’s extremely fast: since Redis stores data in memory rather than on disk, lock acquisition usually takes only a few milliseconds. Second, many applications already use Redis for caching, sessions, queues, or rate limiting, so adding distributed locking often requires little additional infrastructure. Finally, Redis has mature client libraries for virtually every programming language, with Laravel, Django, Spring Boot, .NET, Node.js, and Go all providing excellent Redis support.

For the vast majority of business applications, Redis offers an excellent balance between simplicity, performance, and reliability.

The Challenge with a Single Redis Instance

Suppose your entire application depends on one Redis server. Everything works perfectly, then Redis crashes. Suddenly, no application server can acquire new locks, and even worse, if Redis loses its in-memory state during a restart, locks may disappear unexpectedly.

This introduces a new challenge: the coordinator itself has become a single point of failure. For many applications, this risk is acceptable. For others, particularly financial systems or globally distributed services, it isn’t. This challenge led to one of the most discussed topics in distributed systems: the Redlock algorithm.

The Redlock Algorithm

Redlock was proposed by Redis creator Salvatore Sanfilippo as a way to make Redis-based distributed locks more resilient. Instead of relying on one Redis server, Redlock uses multiple independent Redis instances.

Imagine five Redis servers.

Redis 1

Redis 2

Redis 3

Redis 4

Redis 5

When acquiring a lock, the application attempts to obtain it from all five servers, and the lock is considered successful only if a majority agree.

For example:

Acquire Lock

↓

Redis 1 ✅

Redis 2 ✅

Redis 3 ✅

Redis 4 ❌

Redis 5 ❌

Three out of five succeeded, so the application proceeds. If only two servers grant the lock, the operation fails because a majority wasn’t reached. This approach significantly reduces the impact of individual Redis failures.

However, Redlock is also one of the most debated algorithms in distributed systems. Some engineers argue that it’s sufficient for many practical systems, while others, including Martin Kleppmann, have published detailed critiques explaining situations where Redlock may not provide the guarantees developers expect. The important lesson isn’t that Redlock is good or bad; it’s that distributed systems involve trade-offs, and understanding those trade-offs matters more than memorizing algorithms.

ZooKeeper

Long before Redis became popular for distributed locking, many large distributed systems relied on Apache ZooKeeper. ZooKeeper was designed specifically for coordination. Rather than functioning as a cache, it provides services such as:

• Distributed locks
• Leader election
• Configuration management
• Service discovery

Think of ZooKeeper as a highly reliable coordinator for distributed applications. Its primary goal isn’t speed, it’s correctness. Large systems such as Apache Kafka, Hadoop, and HBase have historically relied on ZooKeeper to coordinate clusters of machines. If your application requires complex distributed coordination rather than simple locking, ZooKeeper remains an excellent choice.

etcd

If you’ve worked with Kubernetes, you’ve already encountered etcd, even if you didn’t realize it. Every Kubernetes cluster stores its configuration inside etcd. Like ZooKeeper, etcd is a distributed key-value store designed for coordination rather than caching. It provides:

• Distributed locks
• Leader election
• Configuration storage
• Consensus
• Service coordination

Unlike Redis, etcd prioritizes consistency over raw performance. Its API is also designed around long-lived leases, making lock management particularly elegant. Modern cloud-native applications frequently choose etcd when they already operate within Kubernetes ecosystems.

Consul

HashiCorp Consul occupies a similar space. Although many developers know Consul for service discovery, it also provides distributed locking capabilities through sessions. Organizations using Consul often rely on it for:

• Service registration
• Health checks
• Distributed configuration
• Leader election
• Distributed locks

Like ZooKeeper and etcd, Consul focuses on reliable coordination across distributed infrastructure.

Which Technology Should You Choose?

There isn’t a universal answer. Instead, the right choice depends on your application’s requirements. If you’re building a typical web application that already uses Redis, implementing distributed locks with Redis is usually the simplest and most practical solution. If you’re coordinating hundreds of services across a Kubernetes cluster, etcd may integrate more naturally with your infrastructure. If your organization already uses Consul or ZooKeeper for service coordination, leveraging those existing systems often makes more sense than introducing Redis solely for locking.

Choosing a technology isn’t just about features. It’s also about operational complexity, existing infrastructure, and the guarantees your business requires. The important thing to remember is this: distributed locks are a concept, while Redis, ZooKeeper, etcd, and Consul are simply different tools for implementing that concept. Understanding the underlying idea is far more valuable than becoming attached to a specific technology.

Do You Always Need a Distributed Lock?

Reading this article, it might be tempting to conclude that distributed locks are the solution to every concurrency problem. They’re not. In fact, many applications never need them. If your application runs on a single server, ordinary in-memory locks are often sufficient, and if your database transaction already guarantees correctness, introducing a distributed lock may simply add unnecessary complexity.

Distributed locks are powerful, but they should be introduced only when multiple independent application instances genuinely need to coordinate shared work. Like every distributed systems technique, they solve a very specific class of problems, and the best engineering decision is often knowing when not to use them.

Common Mistakes When Using Distributed Locks

Like many distributed systems concepts, distributed locks appear deceptively simple: acquire a lock, perform some work, release the lock. In practice, however, there are several subtle mistakes that can introduce bugs that are even harder to diagnose than the problem the lock was intended to solve. Understanding these pitfalls is just as important as understanding distributed locks themselves.

Assuming a Lock Lasts Forever

One of the most common mistakes is forgetting that distributed locks usually have an expiration time. Suppose a server acquires a lock with a TTL of thirty seconds, and the developer assumes the operation will always finish within that window. Months later, a new feature makes the operation take forty-five seconds. The lock expires while the first server is still working, another server acquires the same lock and begins executing the exact same task, and suddenly, duplicate work appears again.

Choosing an appropriate TTL, and renewing it for long-running tasks when necessary, is essential.

Forgetting to Release the Lock

Although expiration protects against permanent deadlocks, applications should still release locks as soon as the protected work finishes. Holding a lock longer than necessary reduces concurrency and delays other servers waiting to perform legitimate work.

A good rule is simple:

Hold the lock only for the work that genuinely requires exclusive access.

Everything else should happen outside the lock whenever possible.

Protecting Too Much Code

Developers sometimes wrap entire workflows inside a distributed lock. For example:

Acquire Lock

↓

Call External Payment API

↓

Generate PDF

↓

Upload File

↓

Send Email

↓

Release Lock

This means every other server waits while network requests, file generation, and email delivery are taking place. Often, only a small portion of the workflow actually requires exclusive access. A better approach is to keep the critical section as short as possible.

Acquire Lock

↓

Update Shared Resource

↓

Release Lock

↓

Generate PDF

↓

Send Email

The shorter the lock, the better the system scales.

Distributed Locks vs Database Locks

At first glance, distributed locks and database locks appear very similar. Both prevent concurrent operations and both coordinate access to shared resources, but they operate at completely different levels.

A database lock protects data inside the database. For example, when two transactions attempt to update the same customer record, the database can lock that row until one transaction completes. Everything happens within the database engine itself.

A distributed lock protects work performed by application servers. Instead of preventing two transactions from updating the same row, it prevents two application instances from starting the same business process. Consider generating monthly invoices: before any invoice rows even exist in the database, every server must first decide whether it should begin the job. That decision happens outside the database, and a distributed lock coordinates that decision.

An easy way to remember the difference is:

Database locks protect data. Distributed locks protect business operations.

In many systems, you’ll use both together. A distributed lock ensures only one server begins generating invoices, and database transactions then ensure every invoice is written consistently.

Distributed Locks vs Optimistic Concurrency

Another concept frequently confused with distributed locks is optimistic concurrency. Optimistic concurrency assumes conflicts are relatively rare. Instead of preventing multiple users from editing the same record, it detects whether someone else changed the data before saving.

Suppose two employees open the same customer profile, and each begins editing. The system stores a version number alongside the record. When Employee A saves, the version changes from 5 to 6. When Employee B later attempts to save, the application notices that the version has already changed. Rather than silently overwriting Employee A’s work, it rejects the update and asks Employee B to refresh the page. No locking was required; the conflict was simply detected before committing.

Distributed locks take a different approach. Instead of detecting conflicts afterward, they prevent conflicting work from starting in the first place. Neither technique is universally better: optimistic concurrency works well when conflicts are uncommon, while distributed locks work best when duplicate execution would be expensive or dangerous.

Best Practices

As with most distributed systems techniques, simplicity is your friend. If you’re considering introducing distributed locks into your application, the following guidelines will help you avoid many common problems.

Keep Critical Sections Small

Acquire the lock immediately before modifying shared resources, and release it immediately afterward. The less work performed while holding the lock, the better your system scales.

Always Use Lock Expiration

Servers crash, containers restart, and networks fail. Never assume your application will always release its lock correctly. Expiration protects the rest of the system from waiting forever.

Verify Lock Ownership

Before releasing a lock, ensure your application still owns it. Ownership checks prevent one server from accidentally deleting another server’s lock after an expiration or retry.

Design for Failure

Distributed systems should always assume that something will eventually fail. Ask yourself:

• What happens if Redis becomes unavailable?
• What happens if the server crashes?
• What happens if the network partitions?
• What happens if the lock expires unexpectedly?

Thinking through failure scenarios early often prevents painful production incidents later.

Combine Reliability Patterns

Distributed locks are rarely used in isolation. Production systems often combine multiple reliability techniques. For example:

• Idempotency prevents duplicate requests.
• Transactions guarantee atomic database updates.
• Isolation Levels provide predictable views of data.
• Distributed Locks coordinate multiple application servers.

Each technique solves a different problem. Together, they create systems that continue behaving correctly even under heavy load and unexpected failures.

Bringing It All Together

At this point in the series, we’ve explored several concepts that all contribute to building reliable backend systems. Each one addresses a different question.

Notice how these concepts complement one another. None replaces the others; reliable systems are built by combining the right tools for the right problems.

Final Thoughts

As applications grow, the biggest challenges often aren’t writing business logic. They’re coordinating work across multiple users, multiple requests, multiple transactions, and eventually multiple servers.

Distributed locks exist because modern applications are no longer confined to a single machine. They’re deployed across clusters, containers, cloud regions, and worker nodes that all need to cooperate without constantly stepping on each other’s toes.

Like transactions and isolation levels, distributed locks aren’t something you’ll use for every feature. But when you do need them, they can be the difference between a system that behaves predictably and one that quietly creates duplicate invoices, repeated payments, or inconsistent business data.

The next time you design a background job, scheduled task, or critical workflow, ask yourself one simple question:

“What happens if two servers try to do this at exactly the same time?”

If the answer is “something bad,” you’ve probably found a place where a distributed lock belongs.

What’s Next?

We’ve now covered:

• Contract Testing
• Idempotency
• Race Conditions
• Database Transactions
• Database Concurrency
• Database Isolation Levels
• Distributed Locks

So far, every concept has focused on keeping operations consistent while they’re happening. But there’s another challenge waiting. Imagine you’ve successfully updated your database inside a transaction and now need to publish an event to Kafka, RabbitMQ, or another message broker. What happens if the database commit succeeds, but publishing the event fails? Or worse, what if the event is published but the transaction rolls back?

This problem has caused countless production incidents in distributed systems. In the next article, we’ll explore The Outbox Pattern Explained: Publishing Events Without Losing Data, one of the most widely used patterns for ensuring your database and message broker stay in sync.

“Database locks protect rows. Distributed locks protect systems.”

Imagine your application has been wildly successful. What started as a simple web application running on a single server now runs across multiple machines behind a load balancer. Requests are shared between application instances, background workers process jobs independently, and scheduled tasks run on every server.

From the outside, everything looks better than ever: pages load faster, traffic scales effortlessly, and users are happy. Then one morning, your finance department calls. Every customer has received four monthly invoices.

Nothing appears wrong with the code. The invoice generation job is scheduled to run once every month, so why did it execute four times? The answer is surprisingly simple: you now have four application servers. At midnight, every server woke up, checked the scheduler, and independently decided that it was responsible for generating invoices. From each server’s perspective, everything was perfectly correct; collectively, however, they created a costly mistake.

Now imagine a different scenario. Your payment gateway sends a webhook confirming a successful payment. The webhook is delivered to one of your load-balanced servers, but a retry occurs because the payment provider doesn’t receive a response quickly enough, so another server processes the same webhook. Both servers begin updating balances, both create accounting entries, and both generate receipts.

You’ve already learned how idempotency protects against duplicate requests and how transactions ensure database operations succeed together.

But neither of those concepts answers a new question:

How do multiple application servers agree that only one of them should perform a particular piece of work?

That problem is solved by distributed locks.

As applications grow beyond a single server, distributed locks become one of the most important coordination mechanisms in modern software engineering.

Why Database Locks Are No Longer Enough

Earlier in this series, we explored database transactions and row-level locking.

Suppose two transactions attempt to update the same bank account.

Using a statement such as:

SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;

the database ensures only one transaction can modify that row at a time.

This works beautifully because both transactions are coordinating through the same database.

Now consider a different problem.

Suppose you have four application servers running the exact same code, and each server has a scheduler responsible for calculating monthly loan interest. Midnight arrives, and every server starts the same scheduled job. None of them are updating the same database row immediately. Instead, they’re deciding whether to begin an entire business process. The database has nothing to lock yet, and by the time one server begins writing data, the others have already started processing. The result is duplicated work.

Database locks are excellent at protecting individual records, but they are not designed to coordinate entire applications spread across multiple machines. This is the gap distributed locks were created to fill.

What Is a Distributed Lock?

A distributed lock is a coordination mechanism that allows multiple independent servers to agree that only one of them may perform a particular operation at a given time. Instead of protecting a single database row, a distributed lock protects an entire business activity.

Imagine a conference room with a single key. Anyone can use the room, but only the person holding the key may enter, and everyone else must wait until the key is returned. A distributed lock works in much the same way: before performing an operation, a server first attempts to acquire the lock. If the lock is available, the server proceeds; if another server already owns the lock, the operation waits, retries, or exits.

The important point is that every server asks the same central authority for permission before beginning work. That authority might be Redis, ZooKeeper, etcd, or another distributed coordination system.

A Simple Example

Imagine four application servers.

        Load Balancer
              │
   ┌──────────┼──────────┐
   │          │          │
Server A   Server B   Server C
   │          │          │
        Server D

At midnight, each server checks whether it’s time to generate invoices.

Without a distributed lock:

Server A → Generate invoices ✅

Server B → Generate invoices ✅

Server C → Generate invoices ✅

Server D → Generate invoices ✅

Four executions, four invoices, and one very unhappy finance department.

With a distributed lock:

Server A → Acquire Lock ✅

Server B → Lock Exists

Server C → Lock Exists

Server D → Lock Exists

Only Server A proceeds, and everyone else exits. The invoices are generated exactly once.

Real-World Examples

Distributed locks appear in far more places than most developers realize. Whenever multiple application instances could accidentally perform the same work, a distributed lock becomes a potential solution.

Scheduled Jobs

Suppose your loan management platform calculates accrued interest every night at midnight. Without coordination, every application server performs the calculation independently, interest may be applied multiple times, and a distributed lock ensures exactly one server performs the calculation.

Payment Processing

Imagine receiving a payment webhook from M-Pesa. Network retries cause multiple servers to receive the same notification, and without coordination, several servers may attempt to update balances simultaneously. A distributed lock allows only one server to process the payment while the others simply exit.

Inventory Management

Only one laptop remains in stock, and two servers receive purchase requests simultaneously. Each attempts to reserve the item. Although transactions protect database consistency, a distributed lock can coordinate reservation workflows across multiple application instances before they even begin modifying inventory.

Sending Emails

Marketing decides to send a promotional email to one million subscribers, and your email scheduler is deployed across five worker nodes. Without coordination, every worker starts sending the campaign and customers receive the same email five times. With a distributed lock, only one worker initiates the campaign while the others remain idle.

Report Generation

Generating annual financial reports may take several minutes. Without coordination, multiple servers might begin generating the exact same report simultaneously, wasting CPU time and increasing database load. A distributed lock ensures only one report generation process is active.

When Should You Consider a Distributed Lock?

A useful rule of thumb is to ask yourself a simple question:

What would happen if two servers performed this operation at exactly the same time?

If the answer is:

• Duplicate invoices
• Duplicate payments
• Duplicate notifications
• Duplicate accounting entries
• Duplicate interest calculations

then the operation is probably a candidate for a distributed lock.

Not every feature needs one. Most HTTP requests don’t, reading data doesn’t, and serving web pages doesn’t. Distributed locks are primarily useful for shared background work and critical business processes where duplicate execution would produce incorrect results.

Distributed Locks Are About Coordination

One misconception worth addressing early is that distributed locks replace database transactions.

They don’t. Transactions guarantee consistency inside the database; distributed locks coordinate between application servers. The two solve different problems.

In practice, many enterprise systems use both together. A server first acquires a distributed lock, then begins a database transaction. When the transaction completes successfully, the server releases the distributed lock. The lock ensures only one server performs the work, and the transaction ensures the database remains consistent while that work is being performed. Together, they provide a powerful foundation for building reliable distributed systems.

At this point, we’ve answered why distributed locks exist.

The next question is equally important:

How do multiple servers actually agree on who owns the lock?

How Distributed Locks Work

At a high level, every distributed lock follows the same basic workflow.

Before performing a critical operation, an application asks a shared coordination service for permission to proceed. If the lock is available, the application acquires it and begins its work; if another server already owns the lock, the application waits, retries later, or simply exits. Once the work has been completed, the lock is released, allowing another server to acquire it.

Although different technologies implement this process differently, the underlying idea remains remarkably simple.

Application Server

        │

Request Lock

        │

───────────────
 Lock Service
───────────────

Lock Available?

   │          │

  Yes         No

   │          │

Acquire      Wait / Retry / Exit

   │

Perform Work

   │

Release Lock

The important detail is that every application instance talks to the same lock service. Without a shared source of truth, each server would simply believe it owned the lock.

Acquiring a Lock

Imagine four servers attempting to generate monthly invoices.

Each server sends a request to Redis asking for a lock called:

invoice-generation

Redis receives the requests almost simultaneously. The first request succeeds, and Redis stores something similar to:

invoice-generation

Owner: Server A

Expires: 30 seconds

When the remaining servers ask for the same lock, Redis responds that the lock already exists. Only Server A continues. Servers B, C, and D either wait, retry after a short delay, or abandon the operation altogether.

The beauty of distributed locks lies in their simplicity: instead of every server making its own decision, they all trust a single coordinator.

Holding the Lock

Once a server has successfully acquired the lock, it proceeds with the protected operation.

This might involve:

• Generating invoices.
• Processing a payment.
• Calculating loan interest.
• Sending reminder emails.
• Synchronizing inventory.

During this period, every other server attempting the same operation sees that the lock is already owned. Rather than performing duplicate work, those servers simply back off. The lock effectively becomes a reservation saying that someone is already doing this work and others should wait.

Releasing the Lock

When the protected work completes successfully, the server releases the lock.

Acquire Lock

↓

Process Work

↓

Release Lock

Once Redis removes the lock, another server is free to acquire it if necessary. Releasing the lock is just as important as acquiring it. A lock that is never released eventually blocks every future attempt to perform that operation.

The Problem with Permanent Locks

Now consider something less pleasant.

Server A acquires the invoice-generation lock, but halfway through generating invoices, the server crashes. Perhaps the machine loses power, perhaps Kubernetes terminates the container, or perhaps someone accidentally restarts the application. The important point is that Server A never gets the opportunity to release its lock.

If the lock remained permanent, every future invoice generation attempt would fail because the system would forever believe Server A still owned the lock. This is one of the biggest differences between traditional application locks and distributed locks: distributed systems must always assume that servers can disappear without warning.

Lock Expiration (TTL)

To solve this problem, distributed locks almost always include an expiration time, often called a Time-To-Live (TTL).

Instead of storing only the lock name, the lock service stores something like:

Lock:

invoice-generation

Owner:

Server A

Expires:

30 seconds

If Server A completes successfully, it releases the lock before those thirty seconds expire. If Server A crashes, Redis automatically deletes the lock after the TTL expires, allowing another server to continue the work instead of waiting forever.

Think of it like borrowing a meeting room: rather than reserving it indefinitely, your booking automatically expires after one hour. If you forget to leave, the reservation eventually disappears and someone else can use the room. TTL prevents abandoned locks from permanently blocking the system.

Choosing the Right TTL

Choosing a lock duration isn’t as straightforward as it might seem.

Suppose generating invoices normally takes ten seconds, and a thirty-second TTL provides plenty of room for occasional delays. But what if one month the process unexpectedly takes forty-five seconds? The lock expires after thirty seconds, another server acquires it, and now both servers are generating invoices simultaneously. You’ve accidentally recreated the very problem the lock was supposed to prevent.

On the other hand, choosing an extremely long TTL isn’t ideal either. If a server crashes while holding a lock that expires after thirty minutes, every other server must wait half an hour before continuing. Finding the right TTL therefore requires understanding how long your operation normally takes, while leaving enough room for occasional delays.

Some distributed lock implementations even allow servers to periodically renew the TTL while they’re still actively working. This approach, often called a heartbeat, keeps long-running operations alive without requiring excessively long expiration times.

What Happens If Two Servers Ask at the Same Time?

One question naturally arises: what happens if two servers request the lock at exactly the same millisecond? The answer depends on the lock service.

Redis, ZooKeeper, etcd, and similar systems perform lock acquisition atomically. That means checking whether the lock exists and creating it happen as a single indivisible operation. There is never a moment when both servers successfully acquire the same lock: one request succeeds and the other fails. This atomicity is exactly what makes distributed locks reliable; without it, two servers could both believe they owned the lock, defeating the entire purpose.

Lock Ownership Matters

Imagine Server A acquires a lock. Before finishing its work, the lock expires because the TTL was too short, and Server B now acquires the same lock. Moments later, Server A finally finishes and attempts to release it. If the lock service simply deleted the lock without checking ownership, Server A would accidentally remove Server B’s lock, Server C could now acquire it, and suddenly two servers are working simultaneously again.

To prevent this, distributed lock implementations associate every lock with a unique owner identifier. When releasing a lock, the application must prove that it is still the owner; if ownership has already changed, the release request is ignored. This simple verification prevents one server from accidentally deleting another server’s lock.

Distributed Locks Aren’t Magic

It’s important to understand that distributed locks don’t eliminate failures. Servers can still crash, networks can still become partitioned, and Redis instances can still fail. Distributed locks simply provide a coordinated way for multiple application instances to make decisions despite those realities. They reduce duplicate work, improve consistency, and coordinate critical business operations, but like every distributed systems technique, they must be implemented carefully and combined with other reliability mechanisms such as transactions, retries, idempotency, and monitoring.

In the next section, we’ll explore the most common technologies used to implement distributed locks, including Redis, Redlock, ZooKeeper, etcd, and Consul, along with the strengths and weaknesses of each approach.

Now that we’ve seen the kinds of problems concurrent transactions can create, the next question is obvious:

How does a database prevent them?

The answer lies in isolation levels.

Rather than enforcing a single set of rules for every application, relational databases allow developers to choose how isolated transactions should be from one another.

This flexibility exists because different applications have different priorities.

A banking application transferring millions of shillings every day values consistency above almost everything else.

A reporting dashboard displaying website traffic may prefer speed over perfect accuracy.

Isolation levels allow the database to balance these competing requirements.

As isolation becomes stronger, transactions observe more consistent data, but the database also has to coordinate more aggressively, often reducing concurrency.

As isolation becomes weaker, transactions execute more freely, improving performance but increasing the likelihood of observing changing data.

The SQL standard defines four isolation levels.

Each one builds upon the guarantees of the previous level.

Read Uncommitted

Read Uncommitted is the weakest isolation level defined by the SQL standard.

At this level, transactions are allowed to read changes made by other transactions even if those changes haven’t yet been committed.

Returning to our banking example, imagine Alice begins transferring KES 20,000 to another account.

The database deducts the money from her balance but hasn’t yet committed the transaction.

Another transaction immediately reads Alice’s account.

Instead of seeing KES 50,000, it now sees KES 30,000, even though the transfer could still fail and be rolled back.

That second transaction has just performed a dirty read.

The advantage of Read Uncommitted is that transactions almost never wait for one another.

Because the database performs very little coordination, throughput can be extremely high.

The downside is that applications may make decisions using data that never officially existed.

For most business applications, this is unacceptable.

Imagine calculating payroll using salaries that are eventually rolled back or approving a loan based on a balance that disappears moments later.

Fortunately, very few modern relational databases actually encourage Read Uncommitted.

Many databases either discourage it entirely or internally behave more conservatively even when it’s requested.

In practice, you’ll rarely choose this isolation level for production systems.

Read Committed

Read Committed is the default isolation level in databases such as PostgreSQL, Oracle, and SQL Server.

Instead of allowing transactions to read uncommitted changes, the database only exposes data that has already been committed.

This immediately eliminates dirty reads.

Returning to Alice’s transfer, suppose another transaction checks her balance while the transfer is still running.

Instead of seeing the temporary balance of KES 30,000, it continues seeing the previously committed balance of KES 50,000 until the transfer completes.

Only after the transaction commits does the new balance become visible.

This makes Read Committed an excellent general-purpose isolation level.

Applications never observe incomplete work, while the database still allows a high degree of concurrency.

However, Read Committed doesn’t solve every problem.

Suppose your transaction reads Alice’s balance at the beginning of a report.

A few seconds later, another transaction deposits KES 100,000 into the account and commits.

If your report queries the balance again before finishing, you’ll now see a different value.

The same row has changed during your transaction.

Read Committed prevents dirty reads, but it still allows non-repeatable reads and phantom reads.

For many applications, that’s a perfectly acceptable trade-off.

Read Committed Timeline

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 150,000

COMMIT

Both values are valid.

The important difference is that Transaction A never sees incomplete or rolled-back data.

It only observes committed changes.

Repeatable Read

Suppose you’re generating an end-of-day financial report.

Your transaction calculates the total account balance across thousands of customers.

Halfway through generating the report, another transaction updates several account balances.

If your report re-reads those accounts later, the totals may no longer match the values used earlier in the report.

This is exactly the situation Repeatable Read was designed to solve.

At this isolation level, once a transaction reads a row, subsequent reads of that same row always return the same version for the lifetime of the transaction.

Even if another transaction updates the row and commits, your transaction continues working with the original version.

It’s as though your transaction receives its own private snapshot of the database.

This provides a much more consistent view of the data, making it particularly useful for reporting systems and financial calculations.

However, Repeatable Read doesn’t necessarily prevent new rows from appearing that satisfy your query conditions.

Depending on the database implementation, phantom reads may still occur, although databases like PostgreSQL use Multi-Version Concurrency Control (MVCC) to eliminate many of these anomalies without locking every row.

This is one of the reasons database behavior differs slightly across vendors.

We’ll return to that shortly.

Repeatable Read Timeline

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 50,000 ✅

COMMIT

Although the database now contains KES 150,000, Transaction A continues seeing KES 50,000 because it is working from a consistent snapshot.

Serializable

Serializable is the strongest isolation level defined by the SQL standard.

The easiest way to understand it is to imagine that every transaction runs one after another instead of simultaneously.

Internally, the database may still execute many transactions concurrently, but it guarantees that the final result is identical to some serial execution order.

Returning to our concert ticket example, suppose only one seat remains.

Customer A begins purchasing the ticket.

Customer B attempts to purchase the same seat at exactly the same time.

Under Serializable isolation, the database ensures that only one transaction succeeds.

The other transaction must either wait, retry, or fail.

The database refuses to produce a result that couldn’t happen if the transactions had executed one after another.

This provides the strongest possible consistency guarantees.

It also comes at the highest performance cost.

Serializable transactions often require additional locking, conflict detection, or transaction retries.

For systems processing large volumes of concurrent requests, this can reduce throughput significantly.

Because of this, Serializable is typically reserved for situations where correctness is absolutely critical.

Financial ledgers, securities trading systems, and certain accounting operations are common examples.

Isolation Levels at a Glance

The table makes an important pattern clear.

As isolation increases, the number of concurrency anomalies decreases.

At the same time, the amount of coordination required by the database increases.

This is why there is no universally “best” isolation level.

The appropriate choice always depends on the requirements of your application.

Database Isolation Levels in Popular Databases

One detail that often surprises developers is that not every relational database implements isolation levels in exactly the same way.

The SQL standard defines the four isolation levels, but database vendors have some flexibility in how they achieve those guarantees.

For example, PostgreSQL relies heavily on Multi-Version Concurrency Control (MVCC). Instead of locking rows aggressively, PostgreSQL keeps multiple versions of a row and allows transactions to read a consistent snapshot of the data. This approach provides excellent concurrency while maintaining strong consistency.

MySQL’s InnoDB storage engine also supports MVCC but implements certain isolation behaviors differently. In particular, its default Repeatable Read isolation level prevents many phantom reads by using a combination of snapshot reads and gap locks.

SQL Server, on the other hand, traditionally relies more heavily on locking, although it also offers snapshot-based isolation levels that can be enabled when appropriate.

As a developer, you don’t need to memorize every implementation detail.

The important lesson is this:

Always understand how your specific database implements isolation before assuming its behavior.

The SQL standard provides the vocabulary, but your database documentation explains the exact behavior.

Choosing the Right Isolation Level

After learning about all four isolation levels, it’s natural to ask:

“Which one should I actually use?”

The honest answer is:

It depends on what you’re building.

Suppose you’re developing a dashboard that displays the number of users currently online.

If the number changes while someone refreshes the page, that’s perfectly acceptable.

There’s little value in sacrificing performance just to ensure every count remains identical throughout a transaction.

Read Committed is usually more than sufficient.

Now consider a payroll system.

Calculating employee salaries requires reading thousands of records while ensuring the figures don’t change halfway through the calculation.

If one employee’s salary is updated while payroll is being processed, the final report could contain inconsistent totals.

Repeatable Read becomes a much better fit because it provides a stable snapshot throughout the transaction.

Finally, imagine a securities trading platform or a banking ledger where even a single inconsistency could have significant financial consequences.

Here, correctness is more important than throughput.

Serializable isolation is often the safest choice, even if it means transactions occasionally wait or retry.

The goal isn’t to choose the strongest isolation level.

The goal is to choose the weakest isolation level that still guarantees the correctness your application requires.

Doing so allows the database to maximize concurrency without sacrificing business integrity.

Isolation Levels and Performance

One mistake developers sometimes make is assuming higher isolation is always better.

In reality, every additional guarantee comes at a cost.

Higher isolation levels typically require the database to:

• Coordinate more transactions.
• Acquire additional locks or maintain more snapshots.
• Detect conflicts more aggressively.
• Delay or retry conflicting transactions.

As concurrency increases, these costs become more noticeable.

A high-traffic e-commerce platform processing thousands of orders every minute cannot afford unnecessary waiting if a lower isolation level already satisfies its business rules.

Likewise, a financial institution cannot sacrifice correctness simply to process a few extra transactions per second.

Finding the right balance is part of designing reliable software.

Isolation Levels vs Transactions vs Race Conditions

At this point in the series, we’ve covered three concepts that are closely related but often confused.

Let’s put them side by side.

Transactions

Transactions answer the question:

What happens if my operation fails halfway through?

They ensure a group of related database operations either all succeed together or all fail together.

Without transactions, partial updates can leave your data inconsistent.

Race Conditions

Race conditions answer a different question:

What happens if two requests modify the same data at the same time?

These problems arise because multiple users or systems interact with shared data concurrently.

The outcome often depends entirely on timing.

Transactions alone don’t eliminate race conditions.

Additional mechanisms such as locking, optimistic concurrency, or stronger isolation levels are often required.

Isolation Levels

Isolation levels answer yet another question:

While another transaction is running, what am I allowed to see?

Should your transaction observe unfinished work?

Should it continue seeing the same data even after another transaction commits?

Should it behave as though it’s the only transaction running?

Isolation levels define these rules.

Together, these three concepts form the foundation of reliable database applications.

They complement one another rather than compete.

A payment system, for example, might use:

• Idempotency to prevent duplicate payment requests.
• Transactions to ensure payment records and account balances remain synchronized.
• Read Committed or Serializable isolation to guarantee consistent reads.
• Locking to prevent concurrent modifications of the same account.

No single technique solves every reliability problem.

Reliable systems combine several techniques, each addressing a different class of failure.

Practical Advice for Backend Developers

If you’re just beginning your backend engineering journey, don’t feel pressured to master every isolation level immediately.

Instead, focus on developing the habit of asking the right questions whenever you design a feature.

For example:

• Could another user modify this data while I’m reading it?
• If the same query runs twice, should it return the same result?
• What happens if another transaction inserts new rows before mine finishes?
• Is perfect consistency necessary, or is slightly stale data acceptable?
• Would optimistic concurrency or explicit locking be a better solution?

Thinking through these questions early in the design process often prevents bugs that are incredibly difficult to diagnose later in production.

Many concurrency issues aren’t caused by writing incorrect code.

They’re caused by making incorrect assumptions about how multiple users interact with the same data simultaneously.

Final Thoughts

Database isolation levels are often presented as a collection of definitions that developers are expected to memorize.

In reality, they’re much simpler than they first appear.

They’re simply different answers to one question:

How much should one transaction be allowed to observe while another transaction is still working?

Lower isolation levels prioritize concurrency, allowing more users to interact with the database simultaneously.

Higher isolation levels prioritize consistency, ensuring every transaction sees a predictable view of the data.

Neither approach is universally correct.

The right choice depends entirely on the problem you’re solving.

As your applications grow, understanding isolation levels becomes increasingly important because concurrency is no longer the exception—it’s the norm.

Every online store, banking application, inventory system, booking platform, and loan management system eventually reaches a point where multiple transactions compete for the same data.

The developers who understand isolation levels don’t simply build applications that work.

They build applications that continue working correctly under real-world load.

What’s Next?

In this series we’ve explored:

• Idempotency
• Race Conditions
• Database Transactions
• Database Isolation Levels

We’ve learned how to protect our systems from duplicate requests, concurrent updates, partial failures, and inconsistent reads.

But one challenge still remains.

Everything we’ve discussed assumes our application is running on a single database.

What happens when your application is running on ten servers, each processing requests simultaneously?

A normal database lock isn’t always enough.

In the next article, we’ll explore Distributed Locks Explained: Coordinating Work Across Multiple Servers, where we’ll see how systems like Redis, ZooKeeper, and etcd help ensure that only one application instance performs a critical operation at a time.

“Transactions guarantee that your work completes correctly. Isolation levels determine what everyone else is allowed to see while that work is happening.”

In the previous article, we explored database transactions and learned how they ensure multiple database operations either succeed together or fail together. Transactions protect applications from partial execution, preventing situations where money is deducted from one account without being deposited into another or where inventory is reduced without successfully creating an order.

But transactions solve only part of the problem.

Modern applications rarely have just one user interacting with the database at a time. Thousands of customers may be placing orders, updating records, making payments, or querying reports simultaneously. Each of these actions runs inside its own transaction, and more often than not, several transactions are accessing the same data at exactly the same time.

This raises an important question:

What should one transaction be allowed to see while another transaction is still running?

Imagine opening your banking application to check your account balance.

At the exact same moment, your employer’s payroll system is depositing your monthly salary into the same account.

Should your transaction see the new balance immediately?

Should it continue seeing the old balance until the salary transaction finishes?

Should it wait until the payroll transaction completes before showing you anything at all?

Each answer is technically valid depending on how the database is configured.

Now imagine an online store where only one laptop remains in stock.

Customer A begins placing an order.

Before their transaction finishes, Customer B checks the product page.

Should Customer B still see one laptop available?

Should they see zero?

Should they wait until Customer A’s purchase either succeeds or fails?

Again, the answer depends on the database’s isolation level.

Isolation levels define the rules governing how concurrent transactions interact with one another. They determine whether one transaction can observe another transaction’s work before it has been completed, whether repeated reads always return the same result, and whether new rows appearing during a transaction should be visible immediately.

Although isolation levels are often introduced as an advanced database topic, they’re really about one simple idea:

How much of another transaction’s work should your transaction be allowed to see?

The answer has significant consequences for both correctness and performance.

In this article, we’ll explore why isolation levels exist, the concurrency problems they solve, the four SQL standard isolation levels, and how to choose the right one for your application.

Why Isolation Exists

To understand isolation, imagine you’re reading a book in a library.

Halfway through chapter three, someone walks over, quietly replaces several pages with new ones, and walks away.

You continue reading without realizing anything changed.

The beginning of the chapter describes one story.

The ending describes another.

Nothing makes sense.

Databases can experience a remarkably similar problem.

When multiple transactions execute simultaneously, each transaction may be reading data while another transaction is actively changing it.

Without rules governing these interactions, applications could make decisions based on incomplete information, outdated values, or data that is eventually discarded.

Isolation exists to prevent these situations.

Rather than allowing every transaction unrestricted access to every change happening in the database, the database controls what each transaction can observe and when it can observe it.

Think of it as putting walls between transactions.

Some walls are very thin.

Transactions can see almost everything happening around them.

Other walls are much thicker.

Transactions operate almost as though they’re the only users of the database.

The thicker the wall, the more isolated the transaction becomes.

The Trade-Off Between Consistency and Performance

At first glance, it might seem obvious that every database should simply use the highest possible isolation level.

After all, if stronger isolation produces more consistent data, why wouldn’t every system choose it?

The answer lies in performance.

Imagine a supermarket with only one checkout counter.

Every customer waits patiently in line.

Because only one cashier is serving customers, inventory updates happen one at a time.

Mistakes are rare.

Unfortunately, the queue becomes enormous.

Now imagine opening ten checkout counters.

Customers move much faster.

However, all ten cashiers are now updating the same inventory system simultaneously.

Keeping everything synchronized becomes much more difficult.

Databases face exactly the same challenge.

Higher isolation levels provide stronger guarantees about data consistency, but they often require additional locking, coordination, and waiting.

Lower isolation levels allow more transactions to execute concurrently, increasing throughput and reducing latency, but they also increase the likelihood that transactions observe changing data.

Isolation levels are therefore a balancing act between two competing goals:

• Consistency, ensuring every transaction sees predictable and reliable data.
• Concurrency, allowing as many users as possible to interact with the system simultaneously.

Different applications make different choices.

A banking application processing financial transfers typically prioritizes correctness over raw performance.

An analytics dashboard generating sales reports might tolerate slightly older data if it means thousands of users can run reports simultaneously without slowing the system.

Neither approach is universally correct.

The appropriate isolation level depends entirely on your business requirements.

Concurrency Anomalies: The Problems Isolation Levels Exist to Solve

Isolation levels were not invented simply to make databases more complicated.

They exist because concurrent transactions can produce behaviors that most developers would consider surprising—or even dangerous.

These unexpected behaviors are collectively known as concurrency anomalies.

Every isolation level is essentially a trade-off between preventing these anomalies and maintaining good performance.

Before discussing the isolation levels themselves, it’s important to understand the problems they are designed to solve.

The four anomalies you’ll encounter most often are:

• Dirty Reads
• Non-Repeatable Reads
• Phantom Reads
• Lost Updates

Each represents a different way concurrent transactions can interfere with one another.

Let’s begin with the simplest.

Dirty Reads

Imagine Alice has KES 50,000 in her account.

She initiates a transfer of KES 20,000 to another account.

The banking system begins processing the transaction.

The first step deducts the money from Alice’s balance.

Before the transaction finishes, another process—perhaps an ATM balance inquiry or an online banking session—checks Alice’s account.

At that moment, it sees a balance of KES 30,000.

Everything seems perfectly normal.

Then something unexpected happens.

The transfer fails because the destination account no longer exists.

The database rolls back the transaction.

Alice’s balance immediately returns to KES 50,000.

The second transaction has now made a decision based on information that never officially existed.

It observed data that was eventually discarded.

This is known as a Dirty Read.

A dirty read occurs when one transaction reads data written by another transaction before that transaction has been committed.

The easiest way to understand it is to imagine reading someone’s unfinished draft before they’ve decided whether to keep or delete it.

The version you read may never become the final version.

Making business decisions based on that draft could lead to incorrect outcomes.

Fortunately, most modern relational databases prevent dirty reads by default because they are rarely desirable in business applications.

The SQL standard still defines them because they help explain the spectrum of isolation levels.

Timeline of a Dirty Read

Transaction A                     Transaction B

BEGIN

Balance = 50,000

↓

Update Balance = 30,000

                               Read Balance = 30,000 ❌

↓

Transfer Fails

↓

ROLLBACK

Balance returns to 50,000

Transaction B has observed a value that disappeared moments later.

From the perspective of the database, that balance never officially existed.

Yet another transaction already acted as though it did.

This is precisely the type of inconsistency isolation levels are designed to prevent.

Non-Repeatable Reads

Suppose you’re building an online banking application.

A customer opens the app and views their account balance. At that moment, the database reports a balance of KES 50,000.

The customer decides to transfer KES 40,000 to another account, but before confirming the transfer, the application performs one final balance check to ensure sufficient funds are still available.

This seems like a perfectly reasonable workflow.

However, between the first balance check and the second, another transaction deposits KES 100,000 into the same account.

When the application performs the second query, the balance is no longer KES 50,000.

It’s now KES 150,000.

Nothing is technically wrong.

The second transaction committed successfully.

The balance genuinely changed.

The surprising part is that the same transaction read the same row twice and received two different answers.

This phenomenon is known as a Non-Repeatable Read.

Unlike a dirty read, the second transaction isn’t reading uncommitted data. Every value it sees has been permanently committed to the database.

The inconsistency comes from the fact that another transaction modified the row while the first transaction was still running.

Imagine reading yesterday’s newspaper while someone keeps replacing pages with today’s edition.

The information isn’t incorrect.

It’s simply inconsistent because the document changed while you were reading it.

For many applications, this isn’t a problem.

If you’re refreshing a weather dashboard or checking the number of users currently online, it’s perfectly acceptable for values to change between two queries.

However, systems that rely on a stable snapshot of data—such as financial reporting, payroll processing, or end-of-day reconciliation—often require the same query to return the same result throughout the entire transaction.

Timeline of a Non-Repeatable Read

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 150,000 ❌

COMMIT

Transaction A never modified the balance itself.

It simply asked the same question twice and received two different answers because another committed transaction changed the underlying data in the meantime.

Real-World Example: Updating a Customer Profile

Consider an insurance application where a customer service representative opens a customer’s profile.

The representative spends several minutes reviewing the information before approving a policy update.

Meanwhile, another employee updates the customer’s phone number and address.

When the representative finally clicks Save, the application may now be working with information that is different from what was originally displayed.

Depending on how the application handles these changes, it could accidentally overwrite newer data or make decisions using outdated information.

This isn’t a database bug.

It’s simply the natural consequence of multiple users interacting with the same record at the same time.

Applications that require users to work with a consistent view of the data often use higher isolation levels or optimistic concurrency controls to detect these situations before committing changes.

Phantom Reads

Now let’s consider a different scenario.

Instead of reading a single row twice, imagine you’re querying an entire collection of rows.

Suppose you’re generating a report showing all loan applications submitted today.

Your first query returns:

Loan Applications Submitted Today

-------------------------------

Loan #101

Loan #102

Loan #103

Total: 3

While your report is still running, another loan application is submitted and committed to the database.

A few moments later, your transaction performs the exact same query again.

This time the results look different.

Loan Applications Submitted Today

-------------------------------

Loan #101

Loan #102

Loan #103

Loan #104

Total: 4

Notice what changed.

None of the existing rows were modified.

Instead, an entirely new row appeared.

This is called a Phantom Read.

A phantom read occurs when the same query returns a different set of rows because another transaction inserted, updated, or deleted records that match the query’s search criteria.

Think of it like counting the number of people in a room.

You count 20 people.

While you’re writing the number down, someone walks into the room.

You count again.

Now there are 21 people.

Nothing about the original twenty people changed.

The difference is that a new “phantom” appeared between your two observations.

Timeline of a Phantom Read

Transaction A                     Transaction B

BEGIN

SELECT *

WHERE loan_date = TODAY

Returns 3 rows

                               BEGIN

                               INSERT Loan #104

                               COMMIT

SELECT *

WHERE loan_date = TODAY

Returns 4 rows ❌

COMMIT

Unlike a non-repeatable read, where an existing row changes, phantom reads involve the appearance or disappearance of entire rows.

Why Phantom Reads Matter

Imagine you’re calculating today’s total revenue for financial reporting.

Your reporting transaction begins at 5:00 PM and starts aggregating sales.

While it’s still processing, new sales continue being recorded.

Different parts of the report may now be working with different datasets.

The total revenue calculated on page one might not match the detailed transaction list generated on page five because new rows appeared while the report was still executing.

In reporting systems, this can produce confusing and inconsistent results.

Higher isolation levels solve this problem by ensuring the transaction sees a consistent snapshot of the data throughout its lifetime, even if other transactions continue inserting new rows.

Lost Updates

The final concurrency anomaly is perhaps the most dangerous because it silently discards valid work.

Imagine two warehouse employees looking at the same inventory record.

The system currently shows:

Laptop Stock = 10

Employee A sells one laptop.

Employee B also sells one laptop at almost exactly the same time.

Both employees read the current stock before making their update.

Each calculates the new quantity as:

10 - 1 = 9

Employee A saves.

The inventory becomes:

9

A fraction of a second later, Employee B saves.

The inventory is still:

9

One of the updates has effectively disappeared.

The correct inventory should now be 8, but because both transactions started from the same original value, one update overwrote the other.

This is known as a Lost Update.

Unlike the previous anomalies, nothing appears obviously wrong.

No errors occur.

No constraints are violated.

The database happily accepts both updates.

The problem is that one user’s work has unintentionally replaced another’s.

Lost updates are one of the primary reasons databases provide row locking, optimistic concurrency control, and stronger isolation levels.

Without these protections, applications that receive many simultaneous updates—such as inventory systems, banking platforms, or booking applications—can slowly drift away from reality without anyone noticing.

Now that we understand the problems, the next question is obvious: How do databases prevent them? That’s exactly what we’ll cover in Part 2.”

“Transactions are not just a database feature—they’re one of the fundamental building blocks of reliable software.”

Imagine you’re transferring KES 10,000 from your savings account to a friend.

From your perspective, it’s a single action. You tap “Send Money”, authenticate the transaction, and wait for confirmation. Behind the scenes, however, the banking system performs several independent operations. It verifies that you have sufficient funds, deducts the amount from your account, credits your friend’s account, records the transaction in a ledger, updates account balances, and generates a receipt.

Each of these operations is important, but together they represent a single business action: a money transfer.

Now imagine the server crashes immediately after deducting the money from your account but before crediting your friend.

The result is disastrous. Your balance has decreased, your friend never receives the money, and unless additional recovery mechanisms exist, the system is left in an inconsistent state. From a customer’s perspective, the money has simply disappeared.

The same problem appears outside banking.

An e-commerce application might create an order, reduce inventory, process a payment, generate an invoice, and send a confirmation email. If the payment succeeds but the order creation fails, the customer has paid for a product that the system doesn’t believe exists. Likewise, in a loan management system, a repayment may update the outstanding balance, post accounting entries, and generate a receipt. If only some of those updates complete, financial records quickly become unreliable.

These problems aren’t caused by bad algorithms or poor business logic. They’re caused by partial execution when only part of a larger operation succeeds.

This is precisely the problem database transactions were designed to solve.

A transaction ensures that multiple database operations behave as a single unit of work. Either every operation succeeds together, or every operation is rolled back as though nothing ever happened. There is no halfway point where your system is left in an inconsistent state.

For backend developers, understanding transactions is just as important as understanding APIs or databases themselves. They are the foundation upon which reliable financial systems, booking platforms, inventory systems, healthcare applications, and countless other business-critical systems are built.

What Is a Database Transaction?

A database transaction is a collection of one or more database operations that the database treats as a single logical operation.

Instead of thinking about individual SQL statements, think about the business process they represent.

Suppose a customer purchases the last laptop in your online store. That single purchase might require your application to:

• Create an order.
• Deduct one item from inventory.
• Reserve the shipment.
• Record the payment.
• Create an invoice.

Although these are separate SQL statements, they represent one business event. Either all of them should succeed, or none of them should.

That’s exactly what a transaction guarantees.

Without transactions, every statement executes independently. If statement number four fails, the previous three remain committed, leaving your data inconsistent.

With transactions, the database waits until you’re satisfied that every operation has completed successfully. Only then are the changes permanently saved.

Understanding Transactions Through a Simple Example

Let’s return to the banking example.

Alice wants to transfer KES 10,000 to Bob.

A simplified version of the SQL might look like this:

UPDATE accounts
SET balance = balance - 10000
WHERE account_id = 1;

UPDATE accounts
SET balance = balance + 10000
WHERE account_id = 2;

At first glance, this seems perfectly reasonable.

But imagine the database crashes immediately after the first statement executes.

Alice’s balance has already been reduced.

Bob’s balance has not increased.

The system now contains incorrect financial data.

This is why production systems rarely execute related operations independently.

Instead, they wrap them inside a transaction.

BEGIN;

UPDATE accounts
SET balance = balance - 10000
WHERE account_id = 1;

UPDATE accounts
SET balance = balance + 10000
WHERE account_id = 2;

COMMIT;

If every statement succeeds, the database executes the COMMIT, making the changes permanent.

If any statement fails before that point, the application issues a ROLLBACK, and the database restores itself to exactly the state it was in before the transaction began.

To the outside world, it appears as though the failed transfer never happened.

This “all-or-nothing” behavior is what makes transactions so valuable.

The Lifecycle of a Transaction

Every transaction follows a predictable lifecycle.

BEGIN
   │
Execute SQL Operations
   │
Everything Successful?
   │
 ┌─ Yes ─────────────┐
 │                   │
COMMIT          Changes Saved
 │
 └─ No ──────────────┐
                     │
                 ROLLBACK
                     │
          Database Restored

The process begins with BEGIN, which tells the database to temporarily hold all modifications rather than immediately committing them.

The application then performs one or more operations. These could involve inserting records, updating balances, deleting data, or modifying relationships between tables.

If every operation succeeds, the application calls COMMIT. At that point, the database permanently saves all changes.

If anything goes wrong along the way, perhaps a validation error, a database constraint violation, or even an unexpected server failure, the transaction is rolled back, discarding every change made since BEGIN.

The beauty of this model is that the database itself guarantees consistency. Developers don’t need to manually undo every failed operation because the database handles that responsibility.

The Four ACID Properties

When developers discuss transactions, you’ll almost always hear the term ACID.

Despite sounding intimidating, ACID simply describes the guarantees that modern relational databases provide when executing transactions.

Atomicity

Atomicity means that a transaction is indivisible.

Either every operation succeeds, or none of them do.

Returning to our banking example, it makes no sense for money to be deducted from one account without being added to another. The transaction must succeed completely or fail completely.

Think of flipping a light switch.

The light cannot be half on.

Similarly, a transaction cannot be half completed.

Consistency

Consistency ensures that every transaction leaves the database in a valid state.

Business rules should always remain true.

If your application enforces that inventory can never become negative, then a successful transaction should never violate that rule.

Likewise, if your accounting system requires every journal entry to balance, no committed transaction should ever leave the ledger unbalanced.

Consistency isn’t about preventing bugs in your application logic; it’s about ensuring that completed transactions respect the rules defined by your database and your business.

Isolation

Isolation becomes important when multiple users interact with the system simultaneously.

Imagine two customers attempting to purchase the last available ticket for a concert.

Without proper isolation, both requests may read the inventory before either updates it. Both believe the ticket is available, and both complete the purchase.

You’ve now sold the same seat twice.

Isolation ensures that concurrent transactions don’t interfere with one another in ways that produce inconsistent results.

In our previous article, we discussed race conditions situations where multiple requests compete to modify the same data. Isolation is one of the database mechanisms used to prevent those concurrency problems.

We’ll explore isolation levels in greater depth in the next article because they deserve an entire discussion of their own.

Durability

Durability guarantees that once a transaction has been committed, the changes are permanent.

Even if the server loses power immediately after the commit, the database ensures that committed data survives.

Modern databases achieve this through techniques such as write-ahead logging, transaction logs, and crash recovery.

For developers, the important takeaway is simple: once the database confirms a successful commit, you can trust that the data has been safely stored.

Transactions Solve Partial Failures: Not Every Problem

One misconception among newer developers is that transactions magically solve every data consistency problem.

They don’t.

Transactions protect against partial execution.

Suppose your application updates three tables and crashes after updating the second one. A transaction ensures that the database rolls everything back, preventing inconsistent data. However, transactions don’t automatically solve concurrency problems.

Imagine two users attempting to withdraw money from the same account simultaneously. Each transaction independently checks the balance before either completes. If both see the same balance and both proceed, the final result may still be incorrect depending on your isolation level. This isn’t a transaction problem. It’s a concurrency problem.

That’s why understanding race conditions and transactions together is so important. They solve different classes of reliability issues.

Common Places You’ll Use Transactions

Transactions appear almost everywhere in modern backend systems.

Payment processing is perhaps the most obvious example. Charging a customer’s card, recording the payment, updating invoices, and generating accounting entries should either all succeed or all fail together.

Inventory management systems use transactions to ensure stock counts remain accurate even when multiple customers are purchasing products simultaneously.

Booking platforms rely on transactions to reserve hotel rooms, airline seats, or event tickets without creating conflicting reservations.

Loan management systems use transactions when posting repayments, updating outstanding balances, calculating accrued interest, and recording accounting entries.

Healthcare systems use them to ensure patient records, prescriptions, billing information, and appointment schedules remain synchronized.

Any time a business operation spans multiple database changes, a transaction is usually involved.

Common Mistakes Developers Make

One of the most common mistakes is keeping transactions open for too long.

Imagine starting a transaction, calling a third-party payment API, waiting several seconds for a response, and only then committing the transaction.

During that entire period, database resources may remain locked, reducing performance for other users.

A better approach is to perform external API calls before starting the transaction whenever possible, keeping the transaction focused solely on database operations.

Another common mistake is assuming transactions automatically protect against concurrent updates. As we’ve already seen, concurrency introduces an entirely different set of challenges that require locking strategies or appropriate isolation levels.

Finally, developers sometimes forget that transactions should represent business operations not individual SQL statements. Wrapping every single query in its own transaction rarely provides meaningful benefits.

Transactions in Modern Frameworks

Fortunately, most frameworks make transactions straightforward to use.

Laravel offers the DB::transaction() helper.

Django provides transaction.atomic().

Entity Framework supports BeginTransactionAsync().

Spring Boot uses the @Transactional annotation.

Although the syntax differs, the underlying principle never changes. The framework simply tells the database when to begin the transaction, when to commit it, and when to roll it back if something goes wrong.

Understanding the concept matters far more than memorizing framework-specific syntax.

Transactions, Idempotency, and Race Conditions

If you’ve been following this series, you may have noticed that each concept addresses a different reliability challenge.

Idempotency protects against duplicate requests by ensuring that repeating the same request doesn’t produce duplicate side effects.

Race conditions occur when multiple requests compete to modify shared data simultaneously, leading to unpredictable outcomes.

Transactions ensure that a group of related database operations either all succeed together or all fail together.

Reliable backend systems typically rely on all three.

Imagine a payment API.

Idempotency prevents customers from being charged twice if they retry a request.

Transactions ensure that charging the customer, recording the payment, and updating account balances either all succeed or all fail.

Proper concurrency control ensures that two simultaneous payment requests don’t corrupt shared data.

Each concept complements the others rather than replacing them.

Final Thoughts

Transactions are one of the reasons relational databases remain so powerful. They provide developers with a reliable mechanism for preserving data integrity even when failures occur.

As systems become larger and more distributed, failures become inevitable. Servers crash, networks fail, APIs time out, and users submit requests simultaneously. Transactions don’t eliminate those realities, but they ensure your database remains consistent when they happen.

Whenever you’re implementing a feature that modifies multiple pieces of related data, pause for a moment and ask yourself:

What happens if this operation fails halfway through?

If the answer is “my system ends up in an inconsistent state,” then you’ve almost certainly found a place where a database transaction belongs.

In the next article, we’ll build on this foundation by exploring database isolation levels and why two perfectly valid transactions can still interfere with one another when they run at the same time.

The website shows:

Only 1 ticket remaining.

At exactly the same moment, someone else clicks Buy.

Both of you complete payment.

Both of you receive confirmation emails.

But there was only one ticket.

How did two people successfully purchase the same seat?

Now imagine the same thing happening in a banking application.

Two ATM withdrawals happen at almost the same time.

Both check the balance before either transaction finishes.

Both think there’s enough money.

Both approve the withdrawal.

The account ends up with a negative balance.

Neither application is necessarily “broken.”

Instead, they suffer from one of the most common problems in software engineering:

Race conditions.

Race conditions are among the hardest bugs to reproduce because they don’t happen every time. They may only appear under heavy traffic, high concurrency, or perfect timing. Everything works during testing until your application reaches production.

In this article, we’ll explore what race conditions are, why they happen, how they relate to idempotency, and the techniques developers use to prevent them.

What Is a Race Condition?

A race condition occurs when two or more operations access and modify the same piece of data at the same time, and the final result depends on the order in which they execute.

The important phrase is:

The outcome depends on timing.

That’s what makes race conditions so dangerous.

Sometimes everything works.

Sometimes everything breaks.

The exact same code can produce different results simply because two requests happened a few milliseconds apart.

A Simple Analogy

Imagine two people standing in front of a cookie jar.

The jar contains exactly one cookie.

Both people look inside.

Both see one cookie.

Both reach in.

Both believe they’ll get the cookie.

Reality says otherwise.

Only one cookie exists.

The mistake happened because both people checked the state before either updated it.

Software behaves exactly the same way.

A Real Banking Example

Suppose an account has:

Balance = $100

Two withdrawal requests arrive simultaneously.

Request A → Withdraw $80

Request B → Withdraw $50

Without synchronization:

Request A
↓

Read Balance ($100)

------------

Request B

↓

Read Balance ($100)

------------

Request A

Balance = $20

------------

Request B

Balance = $50

Depending on timing, the final balance might be:

$20

or

$50

or

-$30

None of these outcomes are guaranteed.

This is a race condition.

Why It Works During Development

Most developers test applications alone.

One request.

One browser.

One user.

Everything works perfectly.

Production is different.

Imagine:

• 5,000 users
• Hundreds of requests per second
• Multiple application servers
• Database replication
• Network latency

The probability of two requests colliding becomes much higher.

Race conditions often appear only after an application becomes successful.

Ironically, scaling your application can reveal bugs that never existed during development.

Race Conditions vs Idempotency

If you’ve read my previous article on idempotency, if not, read it first here. You might wonder whether they’re the same thing.

They’re related, but they solve different problems.

Idempotency answers:

What if the same request is sent twice?

Example:

POST /payments

↓

Retry

↓

POST /payments

The solution is an Idempotency-Key.

The same request produces the same result.

Race conditions answer:

What if different requests happen at the same time?

Example:

User A buys last ticket

↓

User B buys last ticket

These are two legitimate requests from different users.

An idempotency key won’t help because the requests are not duplicates.

Race conditions require synchronization, not deduplication.

Real-World Examples

Airline Seat Booking

Only one seat remains.

Two customers purchase it simultaneously.

Without proper locking:

• Seat sold twice
• Refund required
• Customer frustration

E-Commerce Inventory

Stock:

Laptop

Quantity = 1

Two customers purchase simultaneously.

Without protection:

Inventory becomes:

-1

Now you’ve sold a product you don’t have.

Loan Approval Systems

Imagine two loan officers reviewing the same application.

Officer A:

Approve.

Officer B:

Reject.

If both updates happen simultaneously without coordination, the final loan status depends entirely on timing.

Coupon Redemption

Promotion:

First 100 customers only

If multiple requests update the redemption count simultaneously, you might accidentally issue:

103

105

110

coupons.

How Race Conditions Happen

Most race conditions follow this pattern:

Read

↓

Modify

↓

Write

The danger lies between Read and Write.

Example:

Read Balance

↓

Calculate New Balance

↓

Update Balance

If another request changes the balance between those steps, your calculation becomes outdated.

This is known as a Lost Update problem.

Solution 1: Database Transactions

Transactions ensure multiple operations succeed or fail together.

Example:

BEGIN;

SELECT balance
FROM accounts
WHERE id = 1;

UPDATE accounts
SET balance = balance - 80
WHERE id = 1;

COMMIT;

Transactions protect data consistency.

However, they don’t automatically eliminate every race condition.

Isolation levels matter too.

Solution 2: Row-Level Locks

Many relational databases allow locking a row while it’s being updated.

Example:

SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;

Now:

Request A locks the row.

Request B must wait.

Only after Request A finishes can Request B continue.

This guarantees consistent updates.

Solution 3: Optimistic Locking

Instead of preventing conflicts, optimistic locking detects them.

Imagine a version number.

Account

Balance = 100

Version = 5

Update:

WHERE Version = 5

If another request updates the row first:

Version = 6

Your update fails.

The client retries with fresh data.

Optimistic locking works well when collisions are relatively rare.

Solution 4: Distributed Locks

What if your application runs on multiple servers?

Server A

↓

Database

↑

Server B

A normal in-memory lock won’t work because each server has its own memory.

Instead, developers use distributed locking systems like:

• Redis (Redlock)
• ZooKeeper
• etcd
• Consul

These coordinate access across multiple application instances.

Solution 5: Atomic Database Operations

Sometimes you don’t need to:

Read

↓

Calculate

↓

Write

Instead, let the database perform the update atomically.

Bad:

SELECT quantity;

quantity--;

UPDATE products;

Better:

UPDATE products

SET quantity = quantity - 1

WHERE quantity > 0;

Now the database guarantees consistency.

Detecting Race Conditions

One reason race conditions are difficult is that they rarely appear during manual testing.

Ways to uncover them include:

• Load testing with concurrent users
• Stress testing
• Running parallel integration tests
• Simulating delayed responses
• Chaos engineering
• Monitoring production logs

If a bug only appears “sometimes,” concurrency should be one of your first suspects.

Best Practices

When designing systems that handle shared data:

• Keep transactions short.
• Avoid long-running locks.
• Prefer atomic database operations where possible.
• Use optimistic locking when contention is low.
• Use pessimistic locking for critical resources.
• Test with concurrent requests, not just sequential ones.
• Understand your database’s transaction isolation levels.

Most importantly, always ask:

“What happens if two users do this at exactly the same time?”

Final Thoughts

Race conditions aren’t caused by bad developers, they’re caused by systems becoming concurrent.

As applications grow, users interact simultaneously, background jobs overlap, and services communicate in parallel. Timing becomes unpredictable.

That’s why building reliable software isn’t just about writing correct logic for one request. It’s about ensuring your logic remains correct when hundreds or thousands of requests happen together.

If idempotency protects you from duplicate requests, race condition handling protects you from competing requests.

Together, they form two of the most important building blocks for designing resilient APIs and distributed systems.

The next time you write code that reads, modifies, and writes shared data, pause for a moment and ask:

“What happens if someone else does this at the exact same time?”

If you don’t know the answer, you’ve just found your next engineering problem to solve.

You click the “Pay Now” button.

Nothing happens.

After a few seconds, you assume the request failed, so you click the button again.

And again.

A few minutes later, you discover you’ve been charged three times.

What happened?

From the user’s perspective, the payment seemed to fail. From the server’s perspective, however, it successfully processed every request it received.

This is one of the most common problems in distributed systems, and it’s exactly why idempotency exists.

Whether you’re building payment systems, booking platforms, inventory management software, or any API that changes data, retries are inevitable. Networks fail, clients time out, mobile connections drop, and users double-click buttons.

A well-designed API should survive these retries without creating duplicate side effects.

In this article, we’ll explore what idempotency is, why it matters, and how to implement it in your own APIs.

What Is Idempotency?

In simple terms, an idempotent operation can be performed multiple times without changing the final result beyond the first successful execution.

For example:

Turn on the light.
Turn on the light again.
Turn on the light again.

The light is still ON.

Nothing new happened after the first request.

The final state remains the same.

That’s idempotency.

Now compare it with this:

Deposit $100
Deposit $100
Deposit $100

Your account balance increases by $300.

This operation is not idempotent because every request changes the system.

Why Retries Happen

Many developers assume users only send one request.

Reality is different.

Requests are retried because of:

• Slow internet connections
• Gateway timeouts
• Reverse proxies
• Mobile network interruptions
• Browser refreshes
• Double-clicking buttons
• Client retry mechanisms
• Load balancers
• Microservice communication failures

Imagine this timeline:

Client -------- POST /payments --------> API

             Payment succeeds

API -------- 200 OK --------X

(Response never reaches client)

Client waits...

Client retries.

POST /payments again.

The client believes the payment failed.

The server already completed it.

Without idempotency…

The payment happens twice.

HTTP Methods and Idempotency

HTTP itself distinguishes between idempotent and non-idempotent methods.

GET

GET /users/10

Read the user.

Call it once.

Call it 100 times.

Nothing changes.

Idempotent

PUT

PUT /users/10
{
   "name": "Billy"
}

Replacing the same resource repeatedly produces the same result.

Idempotent

DELETE

DELETE /users/10

Delete the user.

Deleting an already deleted user doesn’t delete them twice.

The final state is still:

User does not exist.

Idempotent

POST

POST /orders

Create a new order.

Call it twice.

You now have two orders.

Not idempotent

This is why POST requests often require additional protection.

Why Payment APIs Use Idempotency Keys

Payment providers like Stripe popularized the use of Idempotency Keys.

The idea is simple.

The client generates a unique identifier.

Example:

Idempotency-Key:
6ab89d3b-acde-4d71-b20d-483d8d0ef091

Every retry sends the same key.

POST /payments

Idempotency-Key:
6ab89d3b-acde-4d71-b20d-483d8d0ef091

When the server receives the request:

1. Check if this key already exists.
2. If not, process the payment.
3. Save both the key and the response.
4. Return the response.

If the same request arrives again with the same key:

Instead of charging the customer again…

Return the previously stored response.

Client

POST /payments
Key: ABC123

↓

Server

Charge customer

↓

Store

ABC123 → Payment #456

↓

Return success

---

Retry

POST /payments
Key: ABC123

↓

Lookup

ABC123 exists

↓

Return Payment #456

No second charge.

Implementing Idempotency

A common workflow looks like this.

Step 1

Receive request.

POST /orders

Headers

Idempotency-Key:
XYZ987

Step 2

Search database.

SELECT *
FROM idempotency_keys
WHERE key = 'XYZ987'

Found?

Yes.

Return stored response.

Done.

Step 3

Not found?

Create the resource.

Create Order

Step 4

Store:

Key

Response

Status Code

Timestamp

Now every retry returns the same response.

Example in Node.js (Express)

app.post("/payments", async (req, res) => {
    const key = req.header("Idempotency-Key");

    const existing = await Idempotency.findOne({ key });

    if (existing) {
        return res.status(existing.status).json(existing.response);
    }

    const payment = await processPayment(req.body);

    await Idempotency.create({
        key,
        status: 201,
        response: payment,
    });

    return res.status(201).json(payment);
});

Python (FastAPI)

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/payments")
async def create_payment(request: Request):
    key = request.headers.get("Idempotency-Key")

    existing = await Idempotency.find_one(key=key)

    if existing:
        return JSONResponse(
            content=existing.response,
            status_code=existing.status,
        )

    body = await request.json()
    payment = await process_payment(body)

    await Idempotency.create(
        key=key,
        status=201,
        response=payment,
    )

    return JSONResponse(content=payment, status_code=201)

C# (ASP.NET Core)

app.MapPost("/payments", async (
    HttpRequest request,
    IdempotencyStore store,
    PaymentService payments) =>
{
    var key = request.Headers["Idempotency-Key"].ToString();

    var existing = await store.FindAsync(key);
    if (existing is not null)
    {
        return Results.Json(existing.Response, statusCode: existing.Status);
    }

    var body = await request.ReadFromJsonAsync<PaymentRequest>();
    var payment = await payments.ProcessAsync(body!);

    await store.CreateAsync(new IdempotencyRecord(key, 201, payment));

    return Results.Json(payment, statusCode: StatusCodes.Status201Created);
});

Go

func createPayment(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")

    existing, err := idempotency.FindOne(r.Context(), key)
    if err == nil && existing != nil {
        w.WriteHeader(existing.Status)
        json.NewEncoder(w).Encode(existing.Response)
        return
    }

    var body PaymentRequest
    if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    payment, err := processPayment(r.Context(), body)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    if err := idempotency.Create(r.Context(), IdempotencyRecord{
        Key:      key,
        Status:   http.StatusCreated,
        Response: payment,
    }); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(payment)
}

Laravel

Route::post('/payments', function (Request $request) {
    $key = $request->header('Idempotency-Key');

    $existing = Idempotency::where('key', $key)->first();

    if ($existing) {
        return response()->json($existing->response, $existing->status);
    }

    $payment = processPayment($request->all());

    Idempotency::create([
        'key' => $key,
        'status' => 201,
        'response' => $payment,
    ]);

    return response()->json($payment, 201);
});

The logic is surprisingly simple.

The complexity comes from storing and managing the keys correctly.

Where Should Idempotency Keys Be Stored?

Options include:

Database

Best for most applications.

Pros:

• Persistent
• Reliable
• Easy to query

Cons:

• Slightly slower

Redis

Excellent for high-volume APIs.

Pros:

• Extremely fast
• TTL support
• Easy expiration

Many APIs automatically expire keys after 24 hours.

In-Memory

Useful only during development.

Not recommended for production.

Restarting the server loses everything.

Common Mistakes

Reusing Keys

Every logical operation should have its own unique key.

Bad:

ABC123

used today

used tomorrow

Good:

New checkout

↓

Generate new UUID

Ignoring Request Differences

Suppose the first request is:

$50

The retry is:

$500

Same key.

Different body.

The server should reject this request because the key is being reused for a different operation.

Never Expiring Keys

Keeping millions of old keys forever wastes storage.

Most APIs expire them after:

• 24 hours
• 48 hours
• 7 days

depending on business requirements.

Real-World Use Cases

Idempotency is valuable anywhere duplicate requests could have costly consequences.

Examples include:

• Payment processing
• Bank transfers
• Order creation
• Hotel reservations
• Flight bookings
• Ticket purchases
• Subscription billing
• Inventory updates
• Webhook processing
• Email sending
• Message queues

If performing the same action twice could create an incorrect outcome, idempotency is worth considering.

When You Don’t Need Idempotency

Not every endpoint needs an idempotency key.

For example:

GET /posts

No state changes.

No duplicates.

No problem.

Likewise, endpoints such as:

• Search
• Filtering
• Reading reports
• Viewing profiles

are already naturally idempotent.

Reserve idempotency mechanisms for operations where retries could create unintended side effects.

Final Thoughts

Idempotency isn’t just an implementation detail—it’s a reliability feature.

In distributed systems, retries are normal. Networks are unreliable, users click buttons more than once, and clients retry requests automatically. Instead of hoping those situations never happen, design your APIs to handle them gracefully.

By using idempotency keys, storing responses, validating retries, and choosing the right storage strategy, you can prevent duplicate orders, repeated payments, and other costly errors.

A resilient API isn’t one that never receives duplicate requests.

It’s one that produces the correct outcome even when duplicate requests inevitably arrive.

The next time you design a POST endpoint, ask yourself:

“What happens if this request is sent twice?”

If the answer is “something bad,” it’s probably time to add idempotency.

I wasn’t thinking about traffic, search engine rankings, personal branding, or becoming a better writer. I simply wanted a place to share my thoughts, document what I was learning, and build something that belonged to me.

Like many developers, I started blogging because I was learning new things every day and wanted to keep track of my progress. At the time, I had no idea how much writing would teach me—not just about technology, but about consistency, communication, patience, and personal growth.

Now, after publishing my first 72 blog posts, I’ve learned lessons that extend far beyond blogging itself. Some lessons came from successes, while others came from mistakes and moments of frustration.

In this article, I want to share the most important lessons I’ve learned from publishing my first 72 blog posts.

1. Consistency Beats Perfection

One of the biggest mistakes I made early on was trying to make every article perfect.

I would spend hours tweaking sentences, rewriting paragraphs, and overthinking every detail. Sometimes I delayed publishing because I felt an article wasn’t “good enough.”

Over time, I realized that perfection is the enemy of progress.

The articles that helped me grow weren’t necessarily my best articles. They were simply the articles I published.

Consistency creates momentum.

Publishing one article every week for a year is far more valuable than spending months trying to create a single masterpiece.

The biggest breakthrough came when I stopped chasing perfection and started focusing on showing up consistently.

2. Writing Clarifies Thinking

Many times, I thought I understood a topic until I tried to explain it in writing.

That’s when the gaps in my knowledge became obvious.

Writing forced me to organize my thoughts, simplify complex ideas, and identify areas where my understanding was incomplete.

The process taught me that learning and explaining are not the same thing.

If I couldn’t explain a concept clearly, I probably didn’t understand it as well as I thought.

This lesson made me a better learner and a better developer.

3. Nobody Reads Your First Posts—And That’s Okay

One of the hardest realities for new bloggers is that almost nobody reads your early content.

I remember publishing articles and checking analytics repeatedly, hoping to see visitors.

Most of the time, there were very few.

At first, that felt discouraging.

But eventually I realized something important:

The purpose of your first blog posts isn’t to attract thousands of readers.

The purpose is to learn how to write.

Your first posts are practice.

Every article improves your skills, your confidence, and your ability to communicate.

The audience comes later.

4. The Habit Matters More Than Motivation

Motivation is unreliable.

Some days you’ll feel inspired.

Other days you’ll have no desire to write.

If blogging depends entirely on motivation, consistency becomes impossible.

One of the most valuable lessons I learned is that habits outperform motivation.

The writers who succeed aren’t necessarily the most talented.

They’re the ones who continue writing even when they don’t feel like it.

Creating a writing habit transformed blogging from an occasional activity into a regular part of my routine.

5. Every Post Doesn’t Need to Be Revolutionary

Early in my blogging journey, I believed every article needed a unique insight or groundbreaking idea.

I was wrong.

Many of my most useful posts covered topics that had already been discussed countless times.

The difference was that I shared my own perspective and experience.

Your value doesn’t come from inventing entirely new ideas.

It comes from explaining ideas through your own lens.

There’s always someone who needs the explanation that only you can provide.

6. Writing Builds Confidence

Publishing online can feel intimidating.

You’re sharing your thoughts with strangers.

You’re exposing your ideas to criticism.

You’re putting your work in public.

At first, this can be uncomfortable.

But over time, each article builds confidence.

You become more comfortable expressing your opinions.

You stop worrying about being perfect.

You gain confidence in your ability to communicate and contribute valuable ideas.

That confidence eventually extends beyond blogging into other areas of life and work.

7. Blogging Is a Long-Term Game

One of the most important lessons I learned is that blogging rewards patience.

Results rarely happen overnight.

Traffic grows slowly.

Authority builds gradually.

Opportunities emerge unexpectedly.

Many people quit because they expect immediate results.

The reality is that blogging is similar to investing.

Small contributions made consistently over time eventually compound into significant outcomes.

The key is staying committed long enough to experience those benefits.

8. Publishing Teaches More Than Consuming

Before I started blogging, I spent most of my time consuming content.

I watched tutorials.

I read articles.

I followed courses.

While these activities were helpful, publishing taught me far more.

Creating content forces you to think deeply, research thoroughly, and communicate clearly.

It transforms you from a consumer into a creator.

That shift changed the way I learn forever.

9. Your Blog Becomes a Personal Archive

One unexpected benefit of blogging is having a permanent record of your journey.

Looking back at my older articles shows how much I’ve grown.

I can see:

• How my thinking evolved.
• What I was learning.
• Problems I was solving.
• Mistakes I made.

The blog becomes more than a collection of articles.

It becomes a timeline of personal growth.

10. Small Efforts Compound Over Time

When I published my first article, it felt insignificant.

The same was true for my second, third, and tenth article.

But by the time I reached fifty posts, those small efforts had accumulated into something meaningful.

Fifty articles represent:

• Hundreds of hours of learning.
• Hundreds of hours of writing.
• Dozens of lessons learned.
• A growing body of work.

This reminded me that success is rarely the result of a single big action.

It’s usually the result of many small actions repeated consistently over time.

11. The Real Reward Isn’t Traffic

When people think about blogging success, they often think about page views, followers, and analytics.

Those metrics matter, but they’re not the greatest reward.

The greatest reward is who you become during the process.

Blogging made me:

• A better writer.
• A better learner.
• A better communicator.
• A more disciplined creator.
• A more thoughtful developer.

Those benefits are far more valuable than any traffic number.

Looking Ahead

Reaching 72 blog posts feels like an important milestone, but it also feels like the beginning.

There’s still so much to learn.

So many topics to explore.

So many experiences to share.

The goal isn’t simply to publish more articles.

The goal is to continue learning, improving, and documenting the journey.

If the first 72 posts taught me anything, it’s that growth comes from consistency, curiosity, and the willingness to keep showing up.

Conclusion

Publishing my first 72 blog posts taught me lessons that extend far beyond writing.

It taught me patience when growth was slow.

It taught me discipline when motivation disappeared.

It taught me confidence when self-doubt appeared.

Most importantly, it taught me that meaningful progress is often the result of small actions repeated consistently over time.

If you’re thinking about starting a blog, my advice is simple:

Start now.

Don’t wait until you’re an expert.

Don’t wait until everything is perfect.

Write your first post.

Then write your second.

Then your third.

One day, you’ll look back and realize those small steps created something much bigger than you ever imagined.

I did an article about importance of writing for developers which you can check out.

You’ve been staring at the same bug for 2 hours. You’ve restarted the server. Cleared cache. Added random console.logs. Somehow… it still doesn’t work.

At some point, you stop coding and start guessing.

And that’s the real problem.

The issue isn’t the bug. It’s that nobody actually teaches debugging as a skill.

The Way Most Developers Debug

Let’s be honest. Most of us learned debugging like this:

• Sprinkle console.log everywhere
• Change random lines and hope something works
• Copy-paste error messages into Google
• Restart everything “just in case”

It sometimes works.

But it’s slow, frustrating, and unreliable.

It’s not debugging.

It’s trial and error disguised as progress.

What Debugging Actually Is

Here’s the mindset shift that changes everything:

Debugging is not about fixing code. It’s about finding where your mental model diverges from reality.

You think the system works one way.

Reality says otherwise.

Your job is to close that gap.

The Debugging Mindset

Before tools, before techniques, this is what matters most.

1. Assume Your Assumptions Are Wrong

If something doesn’t work, at least one thing you believe is false.

Your job is to find it.

2. Narrow the Problem Space

Bad debugging:

“Something is wrong with the app”

Good debugging:

“The issue happens only when this function runs after login”

3. Reproduce Before Fixing

If you can’t reliably reproduce the bug, you don’t understand it.

And if you don’t understand it, your fix is luck—not skill.

4. One Change at a Time

If you change 5 things and it works… which one fixed it?

You don’t know.

That’s how bugs come back later.

5. Understand Before You Patch

Quick fixes feel good.

Understanding the root cause makes you dangerous (in a good way).

A Repeatable Debugging Process

This is where things become practical.

Step 1: Reproduce the Bug

Make it happen consistently.

Click button → error appears  
Refresh → still happens  
Different browser → still happens

If it’s inconsistent, your first task is to find the pattern.

Step 2: Define Expected vs Actual

Write it down clearly.

Expected: API returns user data  
Actual: API returns empty array

This step alone eliminates confusion.

Step 3: Isolate the Problem

Shrink the scope.

• Comment out unrelated code
• Remove layers (UI → API → DB)
• Test pieces independently

Think of it like this:

[ UI ] → [ API ] → [ Database ]

Which layer is lying?

Step 4: Form a Hypothesis

Be explicit:

“I think the API is returning empty data because the query filter is wrong.”

Now you’re not guessing—you’re testing a theory.

Step 5: Test the Hypothesis

Use targeted tools:

• Logs
• Breakpoints
• Network inspector

Example:

console.log("User ID:", userId)

But intentional—not random.

Step 6: Fix and Verify

Fix it.

Then confirm:

• Does it work in all cases?
• Did you break something else?

Step 7: Understand the Root Cause

This is where most devs stop too early.

Don’t just fix it—explain it:

“The bug happened because the state updated asynchronously, and we read it too early.”

Now you’ve learned something reusable.

Real Example: “The API Is Broken” (But It’s Not)

Let’s walk through a real scenario.

The Bug

Frontend shows: No data available

Initial Assumption

“The API is broken.”

Step 1: Check Network Tab

You open DevTools → Network:

API returns correct data

So… not the API.

Step 2: Check State

console.log(data)

It logs:

[]

Empty array.

Step 3: Trace the Flow

useEffect(() => {
  fetchData()
}, [])

Inside fetchData:

setData(response.data)
console.log(data) // still empty

The Problem

React state updates are asynchronous.

You’re logging before state updates.

The Fix

useEffect(() => {
  fetchData()
}, [])

useEffect(() => {
  console.log(data)
}, [data])

The Lesson

The bug wasn’t in the API. It was in your mental model of how state updates work.

Tools That Actually Help

Not everything is about tools—but the right ones matter.

1. Browser DevTools (Underrated Powerhouse)

• Network tab → verify API calls
• Console → inspect runtime values
• Application tab → check storage

2. Breakpoints (Game Changer)

Instead of spamming logs:

Pause execution and inspect state live

3. Intentional Logging

Bad:

console.log("here")

Good:

console.log("User after login:", user)

4. Stack Traces

Read them.

They literally tell you:

• Where the error happened
• What triggered it

5. Rubber Duck Debugging

Explain the bug out loud.

Yes, seriously.

You’ll often solve it mid-explanation.

Problem → Hypothesis → Test → Learn → Repeat

Common Debugging Traps

Avoid these and you’ll already be ahead of most devs:

Fixing Symptoms Instead of Causes

You silence the error… but the bug is still there.

Changing Too Many Things at Once

Now you don’t know what worked.

Ignoring Error Messages

The error is literally telling you what’s wrong.

Read it.

Assuming the Bug Is “Weird”

It’s almost never weird.

It’s misunderstood.

“It Works on My Machine”

This is not a flex.

It’s a clue.

A Better Mental Model

Expectation ≠ Reality  
        ↓  
Investigate the gap

That’s debugging.

Final Thought

The best developers aren’t the ones who write perfect code. They’re the ones who can quickly understand why things break.

Debugging isn’t a side skill.

It is the job.

Heads-up: v2 uses different endpoints and JSON than classic Accounts CRUD. Always confirm API version, preview flags, and SDK support in Stripe’s documentation before shipping production code.

What is Stripe Connect?

Stripe Connect is for platforms that move money between buyers, sellers, and your platform. You connect connected accounts to your platform account so you can:

• Collect payments from customers
• Pay out to sellers or service providers
• Take application fees where appropriate

Stripe holds the payments rail; you own product and UX choices.

Two ideas at once: “style” (Standard / Express / Custom) and “shape” (v2 configurations)

The original guide compares Standard, Express, and Custom as who owns onboarding, dashboards, and compliance.

Accounts v2 adds a separate axis: what capabilities a single Account has, expressed as configurations:

You can assign one or more configurations to the same Account. That is the core promise of v2: one identity, multiple roles, less manual ID mapping.

The Standard / Express / Custom choice is still real: it describes OAuth vs Stripe-hosted onboarding vs fully custom UI. On v2 you implement those experiences while creating and updating Accounts through the v2 surface (where supported).

Standard vs Express vs Custom (unchanged product trade-offs)

On v2, you still make these product choices—but you attach merchant / customer / recipient configurations to match what each connected business must do.

Architectural overview

Fund flow is unchanged at a high level:

What changes in v2 is object modeling:

• One Account can represent both “seller” and “buyer of your SaaS” if you add merchant and customer configurations.
• APIs that took customer may accept customer_account with an Account ID—see using Accounts as customers.

Accounts v2 primitives (before code)

Stripe’s v2 Account objects differ from v1’s flat Account create. You typically send:

• identity — country, entity type, business details.
• configuration — nested blocks such as merchant, customer, recipient, each with capabilities you request (card_payments, balance / payout capabilities as named in current docs).
• defaults — currency, locales, responsibilities (fees / losses collector), etc.
• include — ask the API to return nested sections (e.g. configuration.merchant, requirements).

Responses may omit fields unless you include them—see Stripe’s note on includable response values.

API version: v2 calls often require a specific Stripe-Version header (including preview versions while the API evolves). Set this from the exact value in Stripe’s Accounts v2 docs for your integration window.

Implementation: creating an Account on v2 (HTTP-first)

Official examples are REST + JSON. Below is illustrative—copy field names, capability keys, and version from the current docs, not from this blog alone.

cURL (canonical pattern from Stripe)

Stripe documents POST /v2/core/accounts with a JSON body. Conceptually:

curl -X POST https://api.stripe.com/v2/core/accounts \
  -H "Authorization: Bearer sk_test_..." \
  -H "Stripe-Version: <version-from-stripe-docs>" \
  -H "Content-Type: application/json" \
  --data '{
    "contact_email": "seller@example.com",
    "display_name": "Example Seller",
    "identity": {
      "country": "us",
      "entity_type": "company",
      "business_details": { "registered_name": "Example Co" }
    },
    "configuration": {
      "merchant": {
        "capabilities": {
          "card_payments": { "requested": true }
        }
      },
      "customer": {
        "capabilities": {
          "automatic_indirect_tax": { "requested": true }
        }
      }
    },
    "defaults": {
      "currency": "usd",
      "responsibilities": {
        "fees_collector": "stripe",
        "losses_collector": "stripe"
      },
      "locales": ["en-US"]
    },
    "include": [
      "configuration.customer",
      "configuration.merchant",
      "identity",
      "requirements"
    ]
  }'

This shows the mental model: one create, multiple configurations, explicit includes.

Python (generic HTTP client)

Until your stripe SDK exposes stable helpers for every v2 path, httpx or requests keeps you aligned with the docs:

import os
import requests

def create_account_v2():
    r = requests.post(
        "https://api.stripe.com/v2/core/accounts",
        headers={
            "Authorization": f"Bearer {os.environ['STRIPE_SECRET_KEY']}",
            "Stripe-Version": "<version-from-stripe-docs>",
            "Content-Type": "application/json",
        },
        json={
            # ...same structure as the cURL example...
        },
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

PHP (Laravel-style)

Use Guzzle or Laravel Http::withHeaders([...])->post(...) with the same URL, Stripe-Version, and JSON body. Keep secrets in config, not in source control.

C# (.NET)

Use HttpClient with StringContent(json, Encoding.UTF8, "application/json") and the same headers. Deserialize the JSON response into your own DTOs that track id, requirements, and configuration.* state.

OAuth / Standard accounts: Stripe currently directs platforms that authenticate with OAuth to connected accounts to continue using v1 for that path. Treat Standard as documented in the v1 guide until your OAuth + v2 story is explicitly supported for your use case—see Accounts v2 and OAuth.

Onboarding links (Express-style flows)

For Express-like experiences you still send users through Stripe-hosted onboarding where the product allows it. With a connected account ID returned from v2 (acct_...), Account Links (v1 resource) remain the usual tool for account_onboarding—the same pattern as the v1 article, but the account value may come from a v2 create. Verify compatibility for your API version in Stripe’s docs.

Python (v1 Account Links API, account id from v2 create):

import stripe
stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

link = stripe.AccountLink.create(
    account=connected_account_id,
    refresh_url="https://yourapp.com/reauth",
    return_url="https://yourapp.com/complete",
    type="account_onboarding",
)

Custom-style integrations on v2

Custom still means: you own KYC collection, ToS acceptance, and ongoing verification. On v2 you express that by:

• Sending complete identity data the API requires.
• Requesting merchant / recipient capabilities your product needs.
• Handling requirements and webhooks the same way you would for Custom on v1—only the payload shape differs.

You may still use tos_acceptance-style fields where the v2 schema maps them; follow Stripe’s v2 reference for exact property names.

Using Accounts as customers (customer_account)

Where you used customer=cus_..., many flows accept customer_account=acct_... for an Account that has the customer configuration. Example pattern from Stripe (conceptual):

curl https://api.stripe.com/v1/setup_intents \
  -u "sk_test_...:" \
  -H "Stripe-Version: <version-from-stripe-docs>" \
  -d customer_account=acct_123 \
  -d "payment_method_types[]=card" \
  -d confirm=true \
  -d usage=off_session

Details and supported objects live under using Accounts as customers.

Checking balances

For many Connect operations, connected account scoping is unchanged. Python:

balance = stripe.Balance.retrieve(stripe_account=account_id)

PHP:

$balance = \Stripe\Balance::retrieve([], ['stripe_account' => $accountId]);

C#:

var balance = await stripe.Balance.GetAsync(
    new BalanceGetOptions(),
    new RequestOptions { StripeAccount = accountId }
);

Confirm in Stripe’s docs whether your v2 account IDs behave identically for every Balance and v1 helper you rely on during migration.

Compliance responsibilities

The Standard / Express / Custom compliance split from the original article still applies who collects KYC and who owns disputes. v2 can reduce duplicate identity collection when you add configurations to an existing Account instead of opening a second Customer record.

Choosing a path

Strategic considerations

• Time-to-market: Standard (when OAuth fits) < Express < Custom.
• API surface: v2 adds configuration discipline—plan for migration from v1, not an eternal split (Stripe discourages maintaining both versions simultaneously). - Reference
• SDKs: Expect to use HTTP for some v2 paths until your language SDK is fully aligned.

Final thoughts

Accounts v2 does not erase Standard / Express / Custom—it repackages how you represent connected users in the API. Start from Connect and the Accounts v2 API, add using Accounts as customers when the same legal entity both sells and buys from your platform, and keep the v1 Connect guide handy for OAuth flows and legacy snippets until you have fully moved.

Either way, you still trade off control, compliance, and complexity—only the object model got a long-overdue upgrade.

“It worked locally. Tests passed. But production still broke.”

If you’re building distributed systems with multiple services and frontends, you’ve likely encountered this (whether using .NET + Angular, Node.js + React, Python + Vue, or any combination):

• A backend change gets deployed
• The frontend suddenly breaks
• No tests warned you

The issue isn’t always logic.

It’s often a broken contract between systems.

The Problem: Silent API Breakages

In a typical setup:

• Service Provider: An API or microservice
• Service Consumer: A frontend, app, or another service
• Communication: JSON over HTTP (or any protocol)

Everything depends on one thing:

The consumer and provider agreeing on what data looks like

A Real Example

Let’s use a .NET API and Angular frontend as an example (though this applies to any tech stack).

API Provider (Initial Version)

public class UserDto
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
}

[HttpGet("{id}")]
public IActionResult GetUser(int id)
{
    return Ok(new UserDto
    {
        Id = 1,
        Name = "Billy",
        Email = "billy@example.com"
    });
}

Consumer Interface (Angular Example)

export interface User {
  id: number;
  name: string;
  email: string;
}

Everything works perfectly.

Then a “Small” Change Happens

public class UserDto
{
    public int Id { get; set; }
    public string FullName { get; set; } // renamed
    public string Email { get; set; }
}

Production Result

user.name //  undefined

The UI breaks.

Why Didn’t Traditional Tests Catch This?

• Unit tests → passed (backend logic is fine)
• Integration tests → passed (they used the old model)
• The API still returns valid JSON

But the contract between systems changed—and traditional tests don’t verify that.

What is Contract Testing?

In Contract Testing, a contract is a formal specification of expected behavior and communication rules between two or more components, services or systems.

Contract testing ensures that your service provider always matches what the consumer expects.

A contract defines:

• Request format
• Response structure
• Required fields
• Data types

In Simple Terms

The consumer (frontend, app, or service) defines expectations The provider (API or service) must satisfy them

Consumer vs Provider

Consumer

• Calls the service/API
• Defines expected structure
• Examples: Angular frontend, React app, mobile app, another microservice

Provider

• Returns the data
• Must not break expectations
• Examples: .NET API, Node.js backend, Python service, GraphQL endpoint

How Contract Testing Works

Instead of relying only on integration tests:

1. Angular defines expectations
2. A contract file is generated
3. .NET verifies the contract

Flow

Consumer Test (e.g., Angular)
      ↓
Generates Contract
      ↓
Saved as JSON/YAML
      ↓
Provider verifies against contract (e.g., .NET API)

Practical Example

(.NET backend + Angular frontend as an example)

Step 1: Define Expectations in Consumer (Angular example)

const expectedUser = {
  id: 1,
  name: "Billy",
  email: "billy@example.com"
};

it("should fetch user correctly", async () => {
  const user = await userService.getUser(1);
  expect(user).toEqual(expectedUser);
});

Generated Contract (Simplified)

{
  "request": {
    "method": "GET",
    "path": "/api/users/1"
  },
  "response": {
    "body": {
      "id": 1,
      "name": "Billy",
      "email": "billy@example.com"
    }
  }
}

Step 2: Verify in Provider (.NET API example)

Install Pact:

dotnet add package PactNet

Provider Test

[Fact]
public void VerifyPact()
{
    var pactVerifier = new PactVerifier();

    pactVerifier
        .ServiceProvider("UserApi", "http://localhost:5000")
        .WithFileSource(new FileInfo("pacts/userapi-angular.json"))
        .Verify();
}

If Provider Breaks the Contract

public string FullName { get; set; }

• The test fails immediately

• You catch the issue before deployment

Where Contract Testing Fits

E2E Tests
(user journeys)

Integration Tests
(service interactions)

Contract Tests 
(API agreements)

Unit Tests
(business logic)

Why This Matters in Real Projects

Safer Refactoring

Change DTOs, schema, or API responses without fear. Contract tests verify nothing broke.

Independent Development

Consumer and provider teams move faster. Changes are caught instantly, not in production.

Faster Debugging

Failures clearly show what broke

Stronger CI/CD Pipelines

Consumer Build → Generate Contract
Provider Build → Verify Contract
Deploy → Only if both pass

This works with any tech stack.

Common Mistakes

Over-Specifying Data

Bad:

"name": "Billy Okeyo"

Good:

"name": "string"

Testing Everything

Only validate fields your frontend actually uses

Ignoring Versioning

Breaking contracts without versioning leads to production issues

Best Practices

Keep Contracts Minimal

Focus only on required fields

Version Your API

/api/v1/users
/api/v2/users

Automate in CI/CD

Contracts should be generated and verified automatically

Use Realistic Data

Avoid unrealistic mocks

Final Takeaway

Unit tests verify logic Integration tests verify systems E2E tests verify user flows

Contract tests verify agreements

“Most production bugs aren’t failures… they’re misunderstandings between systems.”

Contract testing eliminates those misunderstandings.

Examples include:

• Sending emails
• Generating reports
• Processing uploaded files
• Running background analytics
• Sending notifications

Running these tasks during an HTTP request can make your application slow and unreliable.

This is where Celery and Redis come in.

Together they allow you to run background jobs asynchronously without blocking your main application.

The Problem with Synchronous Tasks

Imagine a user submits a request that triggers an operation that takes 10 seconds.

For example:

• Generating a financial report
• Parsing a large document
• Sending multiple emails

If your application processes this synchronously:

User Request → Django → Long Task → Response

The user waits for the entire process to finish.

This leads to:

• Slow responses
• Poor user experience
• Possible request timeouts

Introducing Celery

Celery is a distributed task queue that allows you to run background jobs outside the request-response cycle.

Instead of executing tasks immediately, Django sends the job to a queue.

A worker then processes it asynchronously.

The flow becomes:

User Request
    ↓
Django
    ↓
Queue Task
    ↓
Immediate Response
    ↓
Celery Worker
    ↓
Executes Job

This makes your application fast and scalable.

Why Redis?

Celery requires a message broker to manage task queues.

Redis is commonly used because it is:

• Extremely fast
• Lightweight
• Easy to deploy
• Perfect for queues and caching

Redis stores the tasks until workers pick them up.

Example: Sending Email in the Background

Instead of sending email directly in a Django view:

send_mail(
    "Welcome",
    "Thanks for signing up",
    "noreply@example.com",
    [user.email],
)

You create a Celery task:

from celery import shared_task
from django.core.mail import send_mail

@shared_task
def send_welcome_email(email):
    send_mail(
        "Welcome",
        "Thanks for signing up",
        "noreply@example.com",
        [email],
    )

Then call it asynchronously:

send_welcome_email.delay(user.email)

The user gets an immediate response while the email is processed in the background.

Common Use Cases for Celery

Celery is useful for many production tasks:

Email sending

• Welcome emails
• Notifications
• Password resets

Data processing

• Financial calculations
• AI processing
• Data pipelines

Scheduled tasks

• Daily reports
• Cleaning expired sessions
• Updating analytics

Running Celery in Production

A typical Django production stack might look like this:

Users
  ↓
Nginx
  ↓
Gunicorn
  ↓
Django App
  ↓
Redis (Broker)
  ↓
Celery Workers

Each component plays a role:

• Nginx handles web traffic
• Gunicorn runs Django
• Redis manages task queues
• Celery workers execute background jobs

Scaling Celery Workers

One of Celery’s biggest advantages is scalability.

If tasks increase, you simply add more workers.

celery -A project worker --loglevel=info --concurrency=4

More workers mean faster task processing.

Final Thoughts

Celery and Redis are essential tools for running Django applications at scale.

They allow you to:

• Improve response times
• Handle heavy workloads
• Build scalable architectures
• Run background processing reliably

If your Django application handles tasks that take more than a few seconds, moving them to Celery is one of the best architectural decisions you can make.

Images need previewing. CSV files need parsing. Excel files need validation. Large files need handling. And suddenly your “simple upload” becomes a full feature.

In this guide, we’ll build a practical and production-ready file upload system using:

• Angular (Frontend)
• Django + Django REST Framework (Backend)

We’ll cover:

1. Image uploads with preview
2. CSV uploads and parsing
3. Excel uploads and processing
4. Validation and security best practices

Backend Setup (Django + DRF)

1. Install Dependencies

pip install djangorestframework pillow openpyxl pandas

• pillow → Image processing
• openpyxl → Excel support
• pandas → CSV & Excel parsing

2. Configure Media Files

settings.py

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

urls.py (project level)

from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    # your urls
]

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Part 1: Image Upload

Django Model

from django.db import models

class Profile(models.Model):
    name = models.CharField(max_length=100)
    image = models.ImageField(upload_to='profiles/')

Serializer

from rest_framework import serializers
from .models import Profile

class ProfileSerializer(serializers.ModelSerializer):
    class Meta:
        model = Profile
        fields = '__all__'

View

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, FormParser
from rest_framework import status

class ProfileUploadView(APIView):
    parser_classes = [MultiPartParser, FormParser]

    def post(self, request):
        serializer = ProfileSerializer(data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Angular Frontend (Image Upload)

HTML

<input type="file" (change)="onFileSelected($event)" accept="image/*" />
<img *ngIf="previewUrl" [src]="previewUrl" width="200" />
<button (click)="upload()">Upload</button>

Component

selectedFile!: File;
previewUrl: string | ArrayBuffer | null = null;

onFileSelected(event: any) {
  this.selectedFile = event.target.files[0];

  const reader = new FileReader();
  reader.onload = () => {
    this.previewUrl = reader.result;
  };
  reader.readAsDataURL(this.selectedFile);
}

upload() {
  const formData = new FormData();
  formData.append('name', 'Billy');
  formData.append('image', this.selectedFile);

  this.http.post('http://localhost:8000/api/upload/', formData)
    .subscribe(res => console.log(res));
}

Part 2: CSV Upload and Parsing

Django View for CSV

import pandas as pd
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser

class CSVUploadView(APIView):
    parser_classes = [MultiPartParser]

    def post(self, request):
        file = request.FILES['file']

        if not file.name.endswith('.csv'):
            return Response({'error': 'Invalid file type'}, status=400)

        df = pd.read_csv(file)

        # Example processing
        total_rows = len(df)
        columns = list(df.columns)

        return Response({
            'rows': total_rows,
            'columns': columns
        })

Angular CSV Upload

uploadCSV(file: File) {
  const formData = new FormData();
  formData.append('file', file);

  this.http.post('http://localhost:8000/api/upload-csv/', formData)
    .subscribe(res => console.log(res));
}

Part 3: Excel Upload (.xlsx)

Django View for Excel

class ExcelUploadView(APIView):
    parser_classes = [MultiPartParser]

    def post(self, request):
        file = request.FILES['file']

        if not file.name.endswith('.xlsx'):
            return Response({'error': 'Invalid file type'}, status=400)

        df = pd.read_excel(file)

        summary = {
            'rows': len(df),
            'columns': list(df.columns)
        }

        return Response(summary)

Validation & Security Best Practices

1. Limit File Size

In settings.py:

DATA_UPLOAD_MAX_MEMORY_SIZE = 5242880  # 5MB

2. Validate File Type Properly

Don’t rely only on extension.

if file.content_type not in ['image/jpeg', 'image/png']:
    return Response({'error': 'Invalid image format'}, status=400)

3. Rename Uploaded Files

import uuid

def upload_to(instance, filename):
    ext = filename.split('.')[-1]
    return f"uploads/{uuid.uuid4()}.{ext}"

4. Handle Large Files with Streaming

For very large CSV files, process in chunks:

for chunk in pd.read_csv(file, chunksize=1000):
    # process chunk
    pass

Bonus: Returning Processed Data

Sometimes you don’t just upload — you process and return a result file.

Example: Generate processed CSV

from django.http import HttpResponse

response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="processed.csv"'
df.to_csv(response, index=False)
return response

Final Thoughts

File uploads are not just about saving files.

They are about:

• Validation
• Security
• User experience
• Scalability

When done correctly, they become powerful data pipelines inside your application.

If you’re building admin systems, reporting tools, learning platforms, or fintech dashboards — mastering uploads is essential.

One pattern stood out immediately.

I bought a MacBook, and at the time, it felt sufficient. It solved the problem I had: performance, portability, and a solid development environment. From a purely functional standpoint, nothing else was required.

But that sense of “enough” didn’t persist.

Shortly after, I justified getting an iPhone partly for convenience, partly for the ecosystem. Once that was in place, the Apple Watch began to feel like a logical extension: notifications, health tracking, tighter integration. And somehow, even after all that, an Apple TV entered the setup as if the system was still incomplete.

Individually, each purchase was defensible. Collectively, they formed a pattern I hadn’t consciously designed.

What struck me during this reflection was not the spending itself, but the realization that my definition of completeness kept shifting. Satisfaction behaved like a moving target, always one accessory away.

Later, I came across a concept that explained this behavior with surprising precision: The Diderot Effect.

What Is the Diderot Effect?

The Diderot Effect describes a behavioral phenomenon where acquiring a new item triggers a cascade of related purchases, driven by the need for consistency rather than necessity.

The concept originates from Denis Diderot, an 18th-century philosopher who, after receiving a luxurious new robe, felt compelled to replace much of his existing furniture because it no longer “matched” the new standard he had introduced.

In modern terms, the Diderot Effect is essentially scope creep applied to consumption.

Why the Diderot Effect Resonates Strongly in Tech

Technology ecosystems are uniquely optimized to amplify this effect.

1. Ecosystem Lock-In as a Design Strategy

Modern tech products are rarely designed as standalone components. They are built as interdependent systems:

• Phone ↔ laptop ↔ watch ↔ TV
• Cloud services ↔ subscriptions ↔ hardware
• Accessories that unlock “full functionality”

Once you introduce a single high-tier component into your stack, the rest of your setup begins to feel like technical debt.

2. Identity-Driven Tooling

In tech, tools are not just utilities they are signals. Owning certain devices, editors, keyboards, or setups subtly communicates:

• Competence
• Professionalism
• “Seriousness” about the craft

The internal logic becomes:

If I’m this kind of developer, my tools should reflect that.

This is where the Diderot Effect stops being about objects and starts being about identity coherence.

3. Optimization Culture

Engineers are trained to optimize systems. Unfortunately, that mindset often spills into consumption:

• Latency → upgrade
• Minor friction → replace
• Marginal improvement → justify cost

Not every inefficiency needs a hardware solution but the instinct is deeply ingrained.

Real-World Tech Examples of the Diderot Effect

• Buying a new laptop → external monitor → mechanical keyboard → standing desk
• Upgrading a phone → earbuds → watch → chargers in every room
• Improving a dev setup → paid tools → subscriptions → cloud services

Each step feels incremental. The aggregate cost rarely does.

The Hidden Costs Engineers Often Overlook

1. Financial Leakage Through “Reasonable” Decisions

Most Diderot-driven purchases pass basic rational checks. The problem isn’t irrationality it’s unbounded justification.

2. Perpetual Dissatisfaction

When standards continuously shift upward, stability disappears. You’re always upgrading the environment instead of leveraging it.

3. Loss of Intentional System Design

Your setup evolves reactively rather than architecturally.

Managing the Diderot Effect as a Technical Thinker

The solution isn’t abstinence it’s intentional system design.

1. Introduce Decision Latency

Treat secondary purchases like production changes:

• Add a delay (48 hours, a week, or a sprint)
• Re-evaluate the actual requirement

Most impulses decay with time.

2. Define System Boundaries

Before upgrading, explicitly define the scope:

“This purchase solves this problem. Nothing else changes.”

Boundaries prevent cascade failures.

3. Separate Functional Debt from Aesthetic Debt

Ask:

• Is this blocking productivity?
• Or does it simply look inferior relative to a new baseline?

Most Diderot chains are triggered by aesthetics masquerading as inefficiency.

4. Budget Upgrades as Projects, Not Reactions

Instead of ad-hoc improvements:

• Plan upgrades quarterly or annually
• Allocate fixed budgets
• Close the project when scope is met

No silent expansions.

5. Be Wary of Identity-Based Justifications

Phrases like:

• “At my level, I should have…”
• “This setup doesn’t reflect where I am now…”

These are signals that the Diderot Effect is driving the decision, not necessity.

When the Diderot Effect Can Be Used Intentionally

Not all cascades are bad.

Applied deliberately, the effect can reinforce positive systems:

• Fitness → nutrition → sleep optimization
• Learning → better tools → deeper focus
• Workspace upgrades that genuinely reduce friction

The difference lies in conscious orchestration versus unconscious drift.

Final Thoughts

The Diderot Effect is not a failure of discipline, it’s a predictable system behavior triggered by introducing a higher standard into an existing environment.

In tech, where optimization and cohesion are prized, this effect is especially potent.

Reflecting on my own experience, from a single MacBook purchase to a fully integrated ecosystem made one thing clear: “Enough” must be explicitly defined, or it will keep moving.

As engineers, we design systems for a living. Our consumption habits deserve the same level of intentionality.

If you told me at the beginning of 2025 that I’d spend the year writing about scalable systems, testing strategies, outages, performance optimizations, and developer excuses I would’ve believed you. If you also told me I’d still be debugging things that “worked yesterday,” I would’ve believed you even faster.

This article is a recap of everything I wrote this year: the lessons, the wins, the bugs, and the “how did this even pass code review?” moments. Think of it as a Spotify Wrapped, but for technical articles, minus the judgment (okay, maybe a little).

System Design & Scalability — Because Your App Will Grow (Whether You’re Ready or Not)

Summary: This year, I spent a lot of time talking about scalable backend systems — not because every app needs to handle millions of users, but because every app eventually meets its first “why is the server on fire?” moment.

• How to Build Scalable Backend Systems with Python, C#, PHP and Dart A practical guide to designing systems that won’t collapse the moment your app gets featured on Twitter… or worse, WhatsApp groups.

Key lesson: Scalability isn’t about overengineering — it’s about not regretting your life choices later.

Testing — Trust Issues, But Make Them Automated

Summary: Testing dominated a good part of the year, mainly because nothing builds trust issues faster than code that looks correct but isn’t.

This series walked through testing from the basics to full CI/CD integration — because “it works on my machine” is not a deployment strategy.

• Unit, Integration, and End-to-End Tests — Part 1 Where we learn that tests are not the enemy — flaky tests are.
• Unit Testing in Depth — Part 2 Small tests, big confidence.
• Integration Testing in Depth — Part 3 When components finally talk to each other… and start arguing.
• End-to-End Testing in Depth — Part 4 Testing your app the same way users break it.
• The Testing Pyramid & CI/CD — Part 5 The moment you realize automation saves both time and sanity.

Key lesson: Tests don’t slow you down — debugging production issues does.

Performance & Tooling — Making Code Faster Without Sacrificing Sleep

Summary: Performance optimization showed up in different forms this year — from modern runtimes to build optimizations. Because sometimes the app isn’t slow… it’s just doing unnecessary work very enthusiastically.

• Diving into WebAssembly: What It Is and Why It Matters For when JavaScript alone just isn’t fast enough.
• Tree Shaking in TypeScript Removing code you forgot existed but was still shipped to production.

Key lesson: If users say your app is slow, they’re probably right. The logs just haven’t confessed yet.

Engineering Culture — Because Code Is Written by Humans (Flawed Ones)

Summary: Not everything this year was serious architecture talk. Some articles leaned into the very human side of software development — where excuses are plentiful and accountability is… negotiable.

• Top 10 Developer Excuses When Code Breaks A humorous breakdown of things we’ve all said — and what actually went wrong.

Key lesson: If you’ve never blamed the cache, the network, or “some weird edge case,” are you even a developer?

Industry Case Studies — Learning the Hard Way (So You Don’t Have To)

Summary: One of the most impactful pieces this year was breaking down a real-world outage — because nothing teaches engineering humility like watching large systems fail in creative ways.

• What Engineers Can Learn From the Cloudflare Outage A reminder that one bad configuration can humble the biggest companies.

Key lesson: Distributed systems don’t fail loudly — they fail politely, globally, and at the worst possible time.

What 2025 Really Taught Me

If I had to summarize the year in engineering truths:

• Scalability matters before users complain.
• Tests are cheaper than apologies.
• Performance issues hide in plain sight.
• Systems fail — preparation determines whether you panic or recover.
• Developers are predictable creatures with unpredictable bugs.

Looking Ahead

In the next year, expect:

• More real-world case studies
• Deeper dives into distributed systems and cloud patterns
• Practical articles you can actually apply on Monday morning

Thank you for reading, sharing, and occasionally finding bugs in my examples (yes, I see you).

Here’s to another year of writing code, fixing mistakes, and pretending we knew what we were doing all along.

On November 18, 2025, the internet shook a little.

Cloudflare, the massive networking and security platform powering millions of websites globally, experienced a major outage that resulted in widespread 5xx errors across the internet. For several hours, key services like CDN routing, Workers KV, Access authentication, and even Cloudflare’s own dashboard were degraded.

In their official incident report, Cloudflare broke down exactly what happened, and the root cause was surprisingly small and deceptively simple:

A configuration file grew larger than expected, violated a hidden assumption in the proxy code, and triggered runtime panics across the global edge.

This incident is a powerful case study in modern distributed systems. Let’s break down what went wrong — and why engineers everywhere should take note.

What Actually Happened? A Chain Reaction From One Oversized File

The root cause began with a permissions change in Cloudflare’s ClickHouse database cluster, which led to duplicate rows in a dataset used by their Bot Management engine. That duplication caused the generated “feature file”, a config-like file that proxies rely on to double in size.

Here’s where assumptions came back to haunt them:

• Cloudflare’s proxy engine expected a maximum of 200 features.
• The new file exceeded that limit.
• An .unwrap() in their Rust-based FL2 proxy assumed the file would always be valid.
• That assumption failed — the code panicked — resulting in cascading 5xx failures.

As Cloudflare noted in their report, this caused two types of breakage across their edge:

1. FL2 proxies (new engine): Panicked → produced 5xx errors
2. FL proxies (old engine): Failed to process bot scores → defaulted to zero → broke logic in Access, rules, authentication, and security products

To make things even more confusing, Cloudflare’s status page (hosted externally) briefly went offline too, creating a misleading early hypothesis that the outage was a massive DDoS attack.

It wasn’t. It was configuration drift.

Why This Seemingly Small Bug Became a Big Internet Event

Config files have become “just another part of the deployment pipeline,” especially in cloud platforms where machine-generated metadata drives features. But Cloudflare’s outage shows:

• Config is not static
• Config can be corrupted
• Config needs validation just like code

Because this file was distributed globally across tens of thousands of Cloudflare servers, a single flawed generation step caused a worldwide issue within minutes.

Distributed systems amplify mistakes — both good ones and bad ones.

Key Engineering Lessons We Should All Learn

1. Never rely on hidden assumptions

Cloudflare’s proxy code assumed the feature file would never exceed a certain size. That “should never happen” moment is often the birthplace of outages.

Lesson: Add explicit limits, schema checks, and sanity validations to all config ingestion paths.

2. Configuration is part of your software supply chain

The feature file was generated, replicated, and consumed automatically — no human intervention. That makes it just as risky as code.

Lesson: Treat configuration pipelines as first-class citizens: test them, validate them, gate them.

3. Build for graceful degradation, not hard crashes

A single .unwrap() took down parts of the internet.

Lesson: Fail softly. If config is invalid, degrade safely, skip rules, or revert to defaults — don’t panic.

4. Feature flags and kill switches are essential

Cloudflare themselves acknowledged the need for a more robust kill-switch system in their follow-up plans.

Lesson: Every modern engineering team should have:

• global feature kill switches
• fast configuration rollback
• manual override options

5. Monitoring needs context, not just alarms

Many engineers watching the outage thought it was an external attack. Alerting didn’t tell them where the failure was coming from, just that everything was “down.”

Lesson: Monitoring should distinguish between:

• origin failure
• edge failure
• config failure
• auth failure
• internal propagation issues

Context reduces misdiagnosis.

6. Observability tools must be lightweight during crises

During recovery, Cloudflare reported that their debugging systems started consuming high CPU, which slowed other mission-critical services.

Lesson: Your troubleshooting tools must use minimal resources when the system is stressed.

7. Transparency helps the whole industry learn

Cloudflare’s detailed post-mortem is a model for engineering culture. Their openness helps engineers worldwide understand real failure modes in large-scale systems.

Lesson: Share your failures. They help others avoid the same mistakes.

The Bigger Picture: Why This Incident Matters

Cloudflare’s outage wasn’t just a story about a config file. It was a reminder of how fragile the modern internet can be:

• We automate everything
• We trust our pipelines
• We deploy continuously
• We assume schemas won’t change
• We rely on millions of machines making the same decision correctly

But when the assumptions underneath those systems break, failures propagate at machine speed.

For engineers, SREs, DevOps teams, and platform architects, this incident underscores a fundamental truth:

Your system is only as resilient as your least-validated assumption.

Final Thoughts

Cloudflare’s outage was a blip in the timeline of the internet, but the lessons are timeless.

Distributed systems will always fail, the question is how gracefully they fail, how quickly they recover, and how deeply the organization learns from the event.

Cloudflare’s transparency, analysis, and remediation steps set a strong example for engineering teams everywhere.