Resilience Under Load Stress: A Comparative Analysis of Circuit Breaker, Retry, and Bulkhead in Node.js Microservices

Introduction

This is a resilience case study: the payment-service simulates a real memory leak that grows until it hits the container's memory limit, enters a crash-loop (OOM-kill + automatic restart) under sustained load, and the order-service evolved, branch by branch, to handle it: retry with exponential backoff and jitter, circuit breaker, and bulkhead.

• Scope: Distributed Systems / Partial Failure Tolerance.
• Languages and Stack: Node.js (Express), Docker Compose, k6 (Grafana) for load testing.
• Business Impact: $p(95)$ latency dropped 14x (from 3005ms to 204ms) and request throughput increased by 53%. The system adopted a Fail-Fast strategy, prioritizing protection of the Event Loop's resources over retaining zombie calls.

Source Code and Load Reports: Full repository on GitHub with the 4 evolutionary branches, the k6 scripts, and Docker Compose instructions:
🔗 github.com/marcelo3macedo/node-microservices-resilience-patterns

The Problem

The starting point of this case study is the feature/01-base-service-unstable branch: k6 fires load requests against the order-service, which in turn calls the payment-service to confirm each order. The catch is that the payment-service is an unstable service, with a real memory leak built in on purpose. Every request it receives allocates and retains a buffer, so the process's RSS (physical memory) grows continuously under load until it hits the container's memory limit (256MB), and Docker kills the process with an OOM (exit code 137).

“payment-service RAM during the test (2 leak → OOM-kill → restart cycles)” is an interactive chart, available only in the full version of the article.

RAM climbs almost linearly to ~250MB, drops around t+31s (OOM-kill + restart), and climbs again until the end of the test.

The order-service simply forwards the call to the payment-service with a 3s timeout, with no protection whatsoever. The result under load: 286 requests, 16.43% error rate, p(95) of 3004.54ms, with the timeout being hit in almost 1 out of every 6 orders, concentrated exactly in the window where the payment-service's memory is at its limit.

Solution Architecture

Initial situation:

This is an interactive diagram, available only in the full version of the article.

Final architecture:

This is an interactive diagram, available only in the full version of the article.

The order-service is the gateway that calls the payment-service synchronously over HTTP. Across four branches, it gained layers of protection:

Decision Analysis

Detailed information on the main decisions in this case study:

When applying the Circuit Breaker, why did the error rate rise from 2.58% to 7.84%?

Looking at the data in the Metrics and Results section, the absolute error rate increased from the earlier branches (2.58% in retry+backoff) to the more advanced ones (7.84% in circuit-breaker and 7.00% in bulkhead).

This is not a regression, but rather the expected behavior of the Circuit Breaker and Bulkhead patterns:

• Fail-Fast vs. Holding Connections: Instead of keeping requests stuck waiting for an unstable service's timeout (which would exhaust order-service's resources), the architecture opts to reject calls quickly.
• The Positive Impact: Even though the raw error count goes up, $p(95)$ latency drops drastically, freeing up the application's CPU/memory to keep serving healthy traffic from other modules.

In high-availability systems, failing fast to protect the overall health of the cluster is preferable to trying to save individual requests at any cost.

Practical Implementation

Retry with exponential backoff and jitter

• File: order-service/utils/retry.js
• Branch: feature/02-pattern-retry-backoff

Jitter prevents multiple requests from retrying at the exact same instant against a service that just came back up:

function backoffDelay(attempt, baseDelayMs, maxDelayMs) {
  const exponential = Math.min(maxDelayMs, baseDelayMs * 2 ** attempt);
  return Math.random() * exponential;
}

if (response.ok || !isRetryableStatus(response.status) || isLastAttempt) {
  return response;
}
const delay = backoffDelay(attempt, baseDelayMs, maxDelayMs);
await sleep(delay);

Circuit breaker

• File: order-service/utils/circuit-breaker.js
• Branch: feature/03-pattern-circuit-breaker

The CLOSED → OPEN → HALF_OPEN state machine is the "magic" of the pattern: while open, canAttempt() cuts off the call without even touching the payment-service:

canAttempt() {
  if (this.state !== STATE.OPEN) return true;
  if (Date.now() - this.openedAt >= this.openDurationMs) {
    this.state = STATE.HALF_OPEN;
    return true;
  }
  return false;
}

onFailure() {
  this.failureCount += 1;
  const shouldOpen = this.state === STATE.HALF_OPEN || this.failureCount >= this.failureThreshold;
  if (shouldOpen) {
    this.state = STATE.OPEN;
    this.openedAt = Date.now();
    this.failureCount = 0;
  }
}

Bulkhead

• File: order-service/utils/bulkhead.js
• Branch: feature/04-pattern-bulkhead-isolation

A simple counter of active calls; with no queue configured, the excess is rejected immediately instead of waiting:

async run(fn) {
  if (this.active >= this.maxConcurrent) {
    if (this.queue.length >= this.maxQueue) {
      throw new BulkheadRejectedError(
        `Bulkhead "${this.name}" full (${this.active}/${this.maxConcurrent} running, ${this.queue.length}/${this.maxQueue} queued)`
      );
    }
    await new Promise((resolve) => this.queue.push(resolve));
  }
  this.active += 1;
  try { return await fn(); } finally { this.active -= 1; }
}

Metrics, Results, and Lessons Learned

Numbers for each run (one per branch) are detailed in each branch's results/report.txt and results/k6-summary.json files.

“Latency (ms) by percentile, per branch” is an interactive chart, available only in the full version of the article.

“HTTP error rate (%) per branch” is an interactive chart, available only in the full version of the article.

“Confirmed vs. failed orders, per branch” is an interactive chart, available only in the full version of the article.

“Throughput (requests/s), per branch” is an interactive chart, available only in the full version of the article.

In the last branch, with all three patterns active at the same time, you can see exactly which mechanism intercepted each type of degradation:

“Resilience mechanisms triggered on branch 04 (by event)” is an interactive chart, available only in the full version of the article.

Observations

• Retry + automatic restart are a combo, not isolated pieces: of the 26 calls that failed on the first attempt in the (retry-backoff) branch, 16 were saved because the next attempt already hit the restarted payment-service. Retry alone wouldn't save a sustained outage.
• p(95) latency dropped 14x (3005ms → 205ms), but not monotonically across every metric: the maximum latency of the (bulkhead) branch (2562ms) is higher than that of the (circuit-breaker) branch (1371ms), because the bulkhead with no queue lets anything not rejected go straight through, even if the payment-service happens to be in a bad moment of its memory cycle.
• Why did Maximum Latency rise in Bulkhead (2562 ms) compared to Circuit Breaker (1371 ms)?
  • The difference between the maximum latencies comes from how each pattern handles calls to a degraded service (payment-service mid-memory-bottleneck cycle):
    • Circuit Breaker (Fail-Fast): Upon detecting the error rate, the circuit "opens" and starts rejecting calls immediately (within a few milliseconds), without even forwarding them to the target service. Since no request waits on the slow processing of the unstable service, the recorded maximum latency stays low (1371 ms).
    • Bulkhead Without a Queue (Limited Concurrency): The Bulkhead only limits how many calls run simultaneously (e.g., a maximum of 10).
      • Overflow (11th onward): Rejected instantly with a 503 error.
      • Allowed (the first 10): Get through to the payment service. However, if the service happens to be extremely slow (about to run Out Of Memory), those few allowed requests sit waiting for a response, which drives the isolated maximum latency up to 2562 ms.
• Error rate isn't the metric that should be optimized in isolation. The (circuit-breaker and bulkhead) branches accept more errors (7.84% / 7.00%) than the (retry backoff) branch (2.58%) in exchange for much more predictable latency and isolation between services. The right call depends on what the business values more: fewer failed orders, or a system that never jams up in a cascade.
• Delivery guarantee is a different architectural problem. None of these three patterns delivers 100% of orders under a sustained payment-service outage — that would require an asynchronous layer (queue/outbox), outside the scope of this case study.

The full code, with the four branches, the frozen READMEs for each stage, and the raw reports from each run, is at:

• github.com/marcelo3macedo/node-microservices-resilience-patterns.