Distributed Microservices Orchestration & Resilience Engine

gRPC
Protobuf
OpenTelemetry
Jaeger
Prometheus
Grafana
Docker Compose

Real service meshes don't fail cleanly — one struggling service can quietly drag down everything that calls it, and "is this connection actually trustworthy" is a question every hop has to answer for itself. This project builds a small 3-service system specifically to explore those two problems hands-on: what a circuit breaker actually looks like mid-trip, and what "secure" really covers once you trace every hop honestly instead of just the ones that were easy to lock down.

Foundations

Everything below builds up, one idea at a time, to what this project actually is. Click around — most of it is meant to be played with, not just read.

Network request

A network request is one program asking another program — possibly on a completely different machine — to do something or return information, then waiting for a response. Every call in this project goes over the network like this, even between services sitting in the same repo.

Service-to-service vs. browser-to-server

A browser calling a web server usually answers one page view, started by a person clicking something. Two backend services calling each other happens constantly and automatically — this project's Orchestrator calls User and Search dozens of times a second, not once per click.

Browser → server

Service → service

gRPC vs. REST

REST, the common web-API style, sends plain JSON text over HTTP to URL-style paths. gRPC defines an exact contract up front — function names, their inputs and outputs — so a call looks almost like calling a regular function, just across the network.

REST

gRPC

Retry / exponential backoff

Retrying means trying a failed call again instead of giving up immediately. Exponential backoff means waiting progressively longer between attempts, instead of retrying instantly and pounding a struggling service even harder.

Click a point on the timeline to see what's happening there. Total span shown: 4s.

Circuit breaker

A circuit breaker is a safety switch: after enough failures in a row, it stops even trying to call a struggling service for a cooldown period, instead of hammering something that's already down. Click through its 3 states below.

CLOSED

Normal operation — every call goes straight through to the real service.

Click another state above to see what happens from there.

TLS and mutual TLS (mTLS)

TLS is what secures a normal HTTPS connection — the server proves its identity with a certificate, so you know you're really talking to who you think you are. Mutual TLS adds the same requirement in the other direction: the client has to prove its identity too.

TLS (one-way)

Mutual TLS

Observability / distributed tracing

Observability means being able to see what actually happened inside a system after the fact, instead of guessing. Distributed tracing records one request's full journey across every service it touched, so you can see exactly which hop was slow or where it broke.

Click a point on the timeline to see what's happening there. Total span shown: 0.3s.

Overview

Three services model a tiny flight-booking system, talking to each other over gRPC. An Orchestrator receives a booking request and coordinates two others: a User service that checks whether the user is allowed to book, and a Search service that returns flight options.

The actual point of the project isn't the booking logic — it's everything wrapped around those calls: retries with backoff and a circuit breaker for when Search starts failing, mutual TLS securing the inter-service hops, and OpenTelemetry traces plus Prometheus/Grafana metrics so there's an actual trail to look at when something goes wrong, not just silence. One thing worth saying plainly: mTLS genuinely secures two of the three connections in this system, not all of them — covered honestly in the decisions below rather than smoothed over.

Architecture

Click a component below to see what it does.

Click a component in the diagram above to see what it does.

Request flow, step by step

Step 1 of 6

A booking request arrives

The client calls the Orchestrator's BookFlight method over an unencrypted connection — the orchestrator's own inbound port has no TLS.

Key decisions

A hand-rolled circuit breaker, not a library — and it only wraps the Search call

The circuit breaker is plain Python, about 40 lines, tracking its own state machine. It's only applied to the call to Search, not to User validation or the streaming price feed.

Alternative considered: Use an existing circuit-breaker library, and/or apply it uniformly to every outbound call.
Tradeoff: Hand-rolling makes the exact behavior easy to see and reason about, at the cost of missing edge cases a mature library would handle. Only wrapping Search reflects that Search is the service deliberately made flaky for testing.
If reversed: A library would likely handle more edge cases correctly, but with less visibility into exactly what's happening and when.

Without a circuit breaker

With a circuit breaker

Retry wraps the circuit breaker, not the other way around

Each of the retry loop's up-to-3 attempts calls through the circuit breaker fresh — meaning the breaker's state is re-checked on every single retry attempt, not just once before the whole sequence begins.

Alternative considered: Check the circuit breaker once, and only enter the retry loop at all if it's currently closed.
Tradeoff: The way it's built means a request can genuinely be the one that trips the breaker mid-retry, which is realistic. The cost: once open, a request still burns through all 3 retry attempts and their waits even though each is destined to fail instantly.
If reversed: Checking the breaker once up front would fail faster once it's known to be open, but a request could no longer be the one whose own retries trip it.

Check once, up front

What was built: recheck every attempt

mTLS secures the two inter-service hops — not the client-facing edge

Both the Orchestrator→User and Orchestrator→Search connections require both sides to present a valid certificate. The connection into the Orchestrator itself has none of that.

Alternative considered: Also put the Orchestrator's own inbound port behind mTLS (or at minimum TLS).
Tradeoff: As built, this project explored backend-to-backend trust specifically. In a real deployment the client-facing edge would typically sit behind its own protection (a load balancer or API gateway), which this project doesn't include.
If reversed: Securing every hop, including the client edge, would close this gap entirely — it just wasn't what got built here.

User validation is a stub, on purpose — the point was the resilience patterns, not auth

"Is this user valid" is answered by comparing the user_id string to a single hardcoded value. There's no database, no real user store, no actual authentication behind it.

Alternative considered: Wire up a real user lookup, even a minimal one.
Tradeoff: Keeping User validation trivial kept the project's actual focus — circuit breakers, retries, mTLS, tracing, metrics — from getting diluted by an unrelated auth system.
If reversed: A real lookup would make the demo more realistic but wouldn't teach anything new about the resilience patterns this project set out to explore.

Watching the circuit breaker actually trip

The Search service has a commented-out block that, when manually enabled, makes it fail randomly about half the time — built specifically so the circuit breaker's behavior could be observed directly.

Step 1 of 8

1. The failure switch gets flipped on

A block at the top of Search's method gets manually uncommented: on every call, there's now roughly a 50% chance it throws an exception. There's no environment variable — it's a code comment toggled by hand.

Future work

Secure the orchestrator's own inbound port (client-facing edge) with TLS, not just its outbound calls to User and Search
Apply the circuit breaker to the User-service call path too, not only Search
Add centralized configuration through environment variables or a typed config file, instead of hardcoded values scattered across each service
Add health checks and readiness probes so each service could be monitored and restarted safely if this were containerized
Add request deadlines and graceful shutdown handling for more predictable behavior during failures and deploys
Containerize the 3 gRPC services themselves — currently only the observability stack runs under Docker Compose; the services run as local Python processes

Code excerpts

The circuit breaker's core check

The heart of the state machine — once open, every call is rejected instantly until the cooldown passes, at which point exactly one call gets through to test the waters.

if self.state == "OPEN":
    if time.time() - self.last_failure_time > self.recovery_time:
        self.state = "HALF_OPEN"
    else:
        raise Exception("Circuit is OPEN. Skipping call.")

What's actually a stub vs. what's real

Two honest limitations worth seeing in the actual code — the orchestrator secures its outbound calls but not its own inbound port, and "user validation" is a placeholder, not real logic.

# orchestrator_server.py — the orchestrator's own inbound port has no TLS:
server.add_insecure_port('[::]:50053')

# user_server.py — "validation" is a single hardcoded comparison:
if user_id == "123":

View source on GitHub