Distributed Microservices Orchestration & Resilience Engine

The call to Search gets up to 3 attempts. Between attempts, the wait doubles instead of staying fixed — 1 second, then 2 — so a struggling service isn't hit with the exact same request rate it was already failing under.

User validation also retries, but with no circuit breaker behind it — a rejected or slow user check isn't the failure mode this project set out to protect against.

Three states: CLOSED (normal), OPEN (reject instantly, no network call), and HALF_OPEN (let exactly one probe through after cooldown). 3 consecutive failures trip it open; the cooldown is 10 seconds.

One detail that isn't obvious from the state diagram alone: because retry rechecks the breaker fresh on every attempt rather than once up front, the breaker's state can change mid-retry — a request can genuinely be the one whose own third attempt is what trips it.

Search is the one call deliberately built with a failure switch, and it's the dependency that sits on both the normal lookup path (SearchFlights, which returns a fixed flight list) and the streaming path (StreamFlightPrices, which produces changing prices over the stream) — a more realistic dependency to stress than a single boolean check, even though neither call is backed by a real search engine or datastore.

Retry wrapping the breaker, instead of the reverse, has a real consequence once the breaker is open: a request still runs its full 3 attempts and their waits before learning each one failed instantly, costing roughly 3 seconds for a request that never reached the network. Reading the retry logic or the breaker logic alone wouldn't predict that — it only shows up once you trace one request through both.

Without traces and metrics, a tripped breaker is invisible from the outside — it just looks like some requests got faster (rejected instantly) and others got slower (retried), with no explanation connecting the two. Observability is what turns "requests started failing" into "the breaker opened because of 3 consecutive Search failures, and here's the exact trace where that happened."

These calls happen constantly and automatically between backend services, not once per user click, so an explicit typed contract mattered more than the convenience of a plain JSON endpoint. Service methods and message shapes are defined once in .proto files and generated into typed client/server code for all three services.

Alternative considered: Use REST/JSON between services, since it's the more common default for web APIs.

Tradeoff: The shared protobuf schema and generated stubs reduce contract drift and make incompatible message shapes explicit earlier, and HTTP/2 gives the streaming call somewhere Python doesn't have to build itself — at the cost of codegen tooling and losing the "just open it in a browser or curl it" debuggability plain JSON has.

I implemented the breaker directly, in plain Python, because I wanted to understand the state transitions myself and see exactly how it interacts with retries, rather than hide that behavior behind a library. It's scoped to the Search call only — not User validation, not the streaming price feed — since Search is the dependency deliberately made flaky for testing (the exact thresholds and states are covered above).

Alternative considered: Use an existing circuit-breaker library, and/or apply it uniformly to every outbound call.

Tradeoff: A custom implementation is easier to inspect but lacks the maturity, edge-case coverage, and configurability of a production resilience library.

I structured the retry loop to call through the circuit breaker fresh on every attempt, rather than check it once up front, because I wanted a request's own retries to be able to trip the breaker mid-sequence — a scenario the walkthrough above shows directly.

Alternative considered: Check the circuit breaker once up front, and only enter the retry loop at all if it's currently closed.

Tradeoff: Checking once up front would fail faster once the breaker is already known to be open, at the cost of a request no longer being able to be the one whose own retries trip it.

I scoped mTLS to the two hops the Orchestrator itself initiates (User, Search) because backend-to-backend trust was specifically what this project set out to explore — the invariants above call out the client-facing edge as the known gap rather than smoothing over it.

Alternative considered: Also put the Orchestrator's own inbound port behind mTLS, or at minimum plain TLS.

Tradeoff: In a real deployment, the client-facing edge would typically sit behind its own protection — a load balancer or API gateway — which this project doesn't include; securing every hop would close this gap entirely, it just wasn't what got built here.

"Is this user allowed to book" is answered by a single string comparison against one hardcoded user_id — no database, no real user store, no actual authentication behind it.

Alternative considered: Wire up a real user lookup, even a minimal one.

Tradeoff: Keeping User validation trivial kept the project's actual focus — circuit breakers, retries, mTLS, tracing, metrics — from getting diluted by an unrelated auth system, at the cost of a demo that's less realistic on that one dimension.

Prometheus, Grafana, and Jaeger run under one docker compose up. The three gRPC services run as local Python processes.

Alternative considered: Containerize everything, services included.

Tradeoff: The observability stack is static infrastructure that doesn't change while developing — Compose removes real setup friction there for free. Leaving the services as local processes keeps the edit-and-restart loop fast during active development, at the cost of not yet proving the whole system also behaves correctly containerized — that's listed as future work below.