Graceful Degradation & Rate Limit Bypass with Redis in Distributed Systems

Introduction

This is a technical case study based on a real-world integration scenario with the Google Sheets API, focused on solving quota scarcity and high error-rate problems.

In the original scenario, direct load requests quickly blew past the Google ecosystem's limit (300 requests/minute), triggering cascading failures (HTTP 429 - Too Many Requests). To solve the bottleneck, a mock service in Go (sheets-mock-api) was built to faithfully reproduce the real limits, and the Graceful Degradation with Redis pattern was implemented in the main Laravel application (main-service-api).

• Scope: Distributed Systems / Rate Limiting & Fault Tolerance / Caching.
• Languages & Stack: PHP (Laravel), Go (sheets-mock-api), Redis, Docker Compose, k6 (Grafana) for load testing.
• Business Impact: The error rate dropped from 78.3% to 0%, the request throughput sustained by the application rose significantly, and real calls to the external API fell by more than 95%, strictly respecting the provider's quota (300 req/min).

Source Code and Load Reports: Full repository on GitHub containing the services, k6 scripts, and Docker Compose instructions: 🔗 php-graceful-degradation-rate-limit

The Problem

The starting point (branch: feature/01-direct-api-unstable) of the case study is the scenario with no resilience protection: k6 fires load requests against main-service-api (Laravel), which in turn queries sheets-mock-api (Go) directly for every request it receives.

sheets-mock-api implements a strict Rate Limiting algorithm via Leaky/Token Bucket calibrated for 300 requests per minute (5 req/s on average), mirroring the real read/write quota limits of the Google Sheets API.

When the request rate from main-service-api exceeds this ceiling, the external API immediately starts responding with HTTP 429 Too Many Requests. Without a handling layer or graceful degradation, main-service-api passes these errors straight through to end clients.

During the first 160s, the system operates normally with 100% success, but as soon as traffic peaks at t=180s, the Google Sheets limit of 300 req/min is exhausted, collapsing the integration and generating 100% errors (HTTP 429 / 502) in that window. Even with temporary Token Bucket recoveries as traffic drops, the buildup of requests triggers new bottlenecks (as seen at t=240s), totaling 160 failures (12.7% global error rate).

The Solution Architecture

Initial Situation:

Final Architecture:

main-service-api continuously counts the calls sent to the external API within 1-minute sliding windows. As long as consumption stays at or below 50% of the quota (up to 150 req/min), the application operates in Direct Mode, passing reads and writes through synchronously to sheets-mock-api with no need to go through intermediate layers.

The architecture's intelligence kicks in the moment the quota crosses the 50% trigger. From that point on, the system automatically activates Graceful Degradation mode, changing the execution flow to protect the external quota without compromising the user experience.

For read operations, the service stops making HTTP calls to the Go API and instead queries the snapshot kept in the Redis cache, merging it in real time with pending changes to instantly deliver perfectly up-to-date data. For writes, instead of risking a breach of the 300 req/min limit with synchronous writes, the mutation is recorded in a delta queue (delta queue) in Redis and immediately reflected in the local read cache.

With this separation, as soon as the sliding window resets and quota consumption returns to safe levels (< 50%), a background worker steps in to asynchronously drain the delta queue, persisting all accumulated writes to the Google Sheets API at a controlled pace, without generating new request spikes.

Evolution

Decision Analysis

Detailed information about the main decisions in this case study:

Practical Implementation

Atomic Counter and Quota Check

• Snippet: main-service-api/app/Services/GoogleSheetsQuotaService.php
• Branch: feature/02-redis-quota-counter

Instead of resetting the count on whole minutes (which can create spikes at window boundaries), GoogleSheetsQuotaService implements a second-by-second, 60-second Sliding Window algorithm. Every second generates an individual key in Redis with a 60s TTL, and total consumption is calculated by summing the keys in the current 60-second window via MGET.

public function registerRequest(): int
{
	$now = time();
	$key = self::KEY_PREFIX . $now;

	$current = Redis::incr($key);
	if ($current === 1) {
		Redis::expire($key, self::TTL_SECONDS);
	}

	return $this->getQuotaConsumed();
}

...

public function isDegraded(?int $consumed = null): bool
{
	$consumed = $consumed ?? $this->getQuotaConsumed();
	return $consumed > $this->getThreshold();
}

Observability Injection via Middleware

• Snippet: main-service-api/app/Http/Middleware/QuotaTrackerMiddleware.php
• Branch: feature/02-redis-quota-counter

The middleware intercepts the request, queries GoogleSheetsQuotaService to check whether we've hit the 50% quota threshold, and injects state-control headers into the response sent back to the client/k6.

public function handle(Request $request, Closure $next): Response
{
	$consumed = $this->quotaService->registerRequest();
	$isDegraded = $this->quotaService->isDegraded($consumed);

	$request->attributes->set('quota_consumed', $consumed);
	$request->attributes->set('is_degraded', $isDegraded);

	$response = $next($request);

	$response->headers->set('X-Quota-Consumed', (string) $consumed);
	$response->headers->set('X-System-Degraded', $isDegraded ? 'true' : 'false');

	return $response;
}

Read Orchestration and Interception with Delta Merging

• Snippet: main-service-api/app/Services/GoogleSheetsService.php
• Branch: feature/03-graceful-read-degradation

GoogleSheetsService queries GoogleSheetsQuotaService to evaluate the quota state in real time and decides whether to perform a direct, synchronous lookup against the Go Google Sheets API or activate the Graceful Degradation flow, combining the last valid copy (snapshot) with pending changes held in the local buffer.

public function getRows(string $sheet = 'orders'): array
{
	$isDegraded = $this->quotaService->isDegraded();

	if ($isDegraded) {
		return $this->getDegradedMergedRows($sheet);
	}

	return $this->getDirectRows($sheet);
}

Asynchronous Drain Worker and Background Synchronization

Snippet: main-service-api/app/Console/Commands/DrainQuotaBufferCommand.php Branch: feature/04-delta-queue-async-worker

DrainQuotaBufferCommand acts as a background worker (long-running process) responsible for continuously monitoring quota consumption in Redis. It guarantees a paced, safe drain of the atomic write queue (sheets_write_buffer) to the Google Sheets API, only once the quota settles at a safe level ($\le 150\text{ req/min}$).

public function handle(GoogleSheetsQuotaService $quotaService, GoogleSheetsService $sheetsService): int
{
	$sheet = $this->option('sheet');
	$loop = $this->option('loop');
	$sleep = (int) $this->option('sleep');

	$this->info("Iniciando Worker de Drenagem Assíncrona para a folha [{$sheet}]...");

	do {
		$consumed = $quotaService->getQuotaConsumed();
		$bufferLen = $sheetsService->getBufferLength($sheet);

		if ($consumed <= 150 && $bufferLen > 0) {
			$this->info("Cota Normalizada ({$consumed} req/min). Drenando {$bufferLen} itens pendentes do buffer...");
			
			$drained = $sheetsService->drainBuffer($sheet, 20);
			
			$this->info("Drenados {$drained} itens do buffer com sucesso.");
		}

		if ($loop) {
			sleep($sleep);
		}
	} while ($loop);

	return Command::SUCCESS;
}

Metrics, Results, and Lessons Learned

Observed Takeaways

• Strict Quota Preservation (Rate-Limit): The Google Sheets API's request limit was never exceeded, since activating degraded mode stopped synchronous external calls as soon as consumption hit the safety margin.
• Full Data Consistency: Information stayed perfectly up-to-date and coherent for the user, since every mutation/change recorded in the buffer was merged in real time with the Redis snapshot during reads.
• Zero Error Rate (100% Success): There were no failures or flow interruptions, guaranteeing 100% success processing every request sent and completely eliminating HTTP 429 and HTTP 502 errors.
• Computational Resource Overhead: Although the CPU and memory increase was moderate in the simulation (given the scale of the load test), running degraded mode requires more processing and in-memory data retention than the direct flow.
• Importance of the Dynamic Trigger and Scalability: The metrics highlight the importance of degraded mode not staying active all the time, kicking in exclusively during traffic peaks. Additionally, in sustained high-load scenarios, the environment will need to be scaled horizontally/vertically to support the additional resource consumption.

The complete code, with all four branches, the frozen READMEs for each stage, and the raw reports from each run, is available at:

• https://github.com/marcelo3macedo/php-graceful-degradation-rate-limit