Distributed systems inevitably face failures. Unavailability of external services, network latency, infrastructure overload, and temporary errors are all part of the reality of any production environment.
Distributed systems inevitably face failures.
Unavailability of external services, network latency, infrastructure overload, and temporary errors are all part of the reality of any production environment.
For this reason, modern microservices architectures need to be designed with the assumption that failures will happen.
The goal is not to avoid failures, but to ensure the system keeps operating even when part of its components run into trouble.
This project was built to demonstrate fundamental resilience concepts using:
The scenario simulates an order platform where one service receives requests and another processes payments through an external integration prone to failures.
Even in the face of errors, slowness, or temporary unavailability of the external provider, the system keeps operating and guarantees that no order is lost.
Source code:
The proposal is to demonstrate how microservices can keep operating by combining a few widely used architectural patterns.
During the simulation run it's possible to observe:
The architecture was built using independent services that communicate exclusively through queues.
Orders Service
↓
RabbitMQ
↓
Payments Service
↓
External Gateway
The orders service has no direct dependency on the payments service whatsoever.
This means that even if payment processing is unavailable, orders keep being accepted and stored in the queue.
Responsible for receiving orders.
Example:
{
"orderId": "123",
"amount": 150.00
}
Upon receiving the order:
The actual processing happens afterward.
RabbitMQ acts as the decoupling layer.
Its role is to:
Even if the payments service is temporarily unavailable, messages remain stored.
Responsible for consuming orders and processing payments.
Flow:
Queue
↓
Consumer
↓
External Gateway
↓
Result
This is exactly where the simulated failures happen.
To represent situations common in real environments, an external service with a configurable failure rate was created.
Examples:
70% failure rate
or
5% failure rate
This makes it possible to observe how the architecture behaves under different levels of instability.
Imagine the following scenario:
External gateway unavailable
Without any protection mechanism:
Order
↓
Timeout
↓
Retry
↓
Timeout
↓
Retry
The result would be:
To avoid this scenario, the Circuit Breaker pattern was implemented.
The goal is to temporarily stop new calls when an excessive number of failures is detected.
Normal operation.
Order
↓
Gateway
All requests are allowed.
After reaching the configured limit of consecutive failures.
Order
↓
Circuit Open
↓
Immediate failure
No call is sent to the gateway.
The system fails fast and protects the external integration.
After the waiting period.
Test request
↓
Gateway
If the call succeeds:
HALF_OPEN → CLOSED
If it fails:
HALF_OPEN → OPEN
One of the main goals of this demonstration is to guarantee that no order is lost.
Three mechanisms were used for this.
All messages are published as persistent.
channel.sendToQueue(queue, buffer, {
persistent: true
});
This guarantees survival even after RabbitMQ restarts.
A message is only confirmed after successful processing.
Success
↓
ACK
Otherwise:
Failure
↓
NACK
Problematic messages don't return immediately to the main queue.
Flow:
Orders
↓
Failure
↓
DLQ
↓
Retry Queue
↓
Orders
This approach avoids aggressive reprocessing loops.
The run lasted approximately:
115 seconds
During this period:
| Metric | Value |
|---|---|
| Orders sent | 20 |
| Total attempts | 66 |
| Approvals | 20 |
| Failures | 46 |
| Retries | 46 |
| Orders lost | 0 |
Despite:
46 failures
the final result was:
20 orders approved
Every single order reached its destination.
No message was lost.
The gateway was configured with:
70% failure rate
Observed events:
t+0s
Unstable gateway
t+9s
Circuit Breaker opens
CLOSED → OPEN
From this point on, the system stops sending requests to the gateway.
After the configured timeout:
OPEN → HALF_OPEN
A test request is executed.
The gateway is still failing.
Result:
HALF_OPEN → OPEN
The system remains protected.
A dynamic change was made at runtime:
70% → 5%
in the failure rate.
Without restarting containers.
Without interrupting the environment.
After a few successful attempts:
OPEN
↓
HALF_OPEN
↓
CLOSED
The Circuit Breaker returns to its normal state.
Throughout the run, several transitions occurred.
CLOSED → OPEN
OPEN → HALF_OPEN
HALF_OPEN → OPEN
OPEN → HALF_OPEN
HALF_OPEN → CLOSED
This demonstrates exactly the expected behavior of the pattern.
The circuit protects when needed and returns automatically once the dependency recovers.
Without the Dead Letter Queue, the behavior would be:
Failure
↓
Immediate retry
↓
Failure
↓
Immediate retry
Known as:
Retry Storm
This pattern can bring entire systems down.
With DLQ:
Failure
↓
Retry Queue
↓
Wait 30 seconds
↓
New attempt
The system gains time to recover.
The demonstration highlights several architectural advantages.
The orders service keeps operating even when payment fails.
External problems don't bring down the entire application.
The Circuit Breaker prevents overloading services that are already unstable.
No manual intervention is required.
The system recovers on its own.
No message is lost.
Even in the face of dozens of failures.
New consumers can be added without changing the architecture.
In distributed architectures, failures are not exceptions.
They are expected events.
That's why patterns such as:
should be part of the architectural design from the very start.
These mechanisms turn temporary failures into controlled delays, preventing data loss and systemic unavailability.
The simulation demonstrated how a microservices-based architecture can keep functioning even in the face of severe failures in external dependencies.
During the run, the gateway showed an extremely high failure rate, the Circuit Breaker kicked in several times, and dozens of processing attempts failed.
Even so, thanks to the use of persistent queues, Dead Letter Queues, controlled retries, and explicit processing confirmation, every single order was delivered successfully.
That is exactly the goal of resilience in distributed systems:
Not to prevent failures from happening, but to guarantee that the business keeps running despite them.