Real-Time Event-Driven Payment Processing Backend

FastAPI
Apache Kafka
PostgreSQL
Python

Payment systems fail constantly, by design — networks drop, processes crash mid-step, the same message gets delivered twice. This project explores what it actually takes to make a payment pipeline survive that instead of quietly losing money or double-charging someone, and it doesn't pretend every rough edge along the way is already solved.

Foundations

Everything below builds up, one idea at a time, to what this project actually is. Click around — most of it is meant to be played with, not just read.

API request / response

The client sends a request describing a payment, and the API sends back a response — in this case, an immediate acknowledgement, not a confirmation that the payment is fully processed yet.

A database row, and its "state"

Each payment is one row in a table, and that row has a status that changes over time — it starts "pending," then moves to either "processed" or "failed" once the real work happens.

pending

The row FastAPI writes immediately, before any real processing has happened.

Click another state above to see what happens from there.

Asynchronous processing, and why decouple it from the request

Synchronous means the client waits until all the work is done. Asynchronous means the server does the minimum to acknowledge the request, then finishes the real work later — so a slow downstream step doesn't make every client wait for it.

Synchronous

Asynchronous (what was built)

Message queue

A message queue (this project uses Apache Kafka) is like a mail room: the sender drops a message and moves on, and a separate reader picks it up whenever it's ready — the two sides never have to be available at the same moment.

"At least once" delivery, and why duplicates happen

Kafka guarantees a message won't be lost — but that guarantee comes with a catch: if a consumer crashes before confirming it finished, Kafka will deliver that same message again, on the assumption it's safer to risk a duplicate than to risk losing it.

Click a point on the timeline to see what's happening there. Total span shown: 6s.

Idempotency

Idempotency means handling the same message twice causes no extra effect — like a light switch already on staying on if you flip "on" again. It's what makes redelivery, above, safe instead of dangerous.

event_id	insert result
evt_482	inserted

A new event ID — the insert succeeds, and processing continues normally.

Dead-letter queue (DLQ)

When processing a message throws an error, a dead-letter queue is a separate holding area it gets moved to instead of crashing the consumer or vanishing silently — so one bad message doesn't block every payment behind it.

Overview

When a payment request comes in, the API does the bare minimum to respond fast: write a `pending` row to Postgres and drop a message describing the payment onto a Kafka topic, then return immediately. A separate consumer process picks that message up whenever it's ready, updates the real balance, and marks the payment `processed` — or `failed`, with the message rerouted to a dead-letter queue instead of vanishing.

The interesting problem isn't the happy path — it's what happens when Kafka redelivers a message the consumer already handled, or when the database write and the Kafka publish don't happen as one atomic unit, which turns out to be a real, still-open gap in the current code rather than something already solved (covered honestly in the failure scenario below).

Architecture

Click a component below to see what it does.

Click a component in the diagram above to see what it does.

Request flow, step by step

Step 1 of 7

A payment request comes in

POST /payments arrives with a user ID and an amount. The user ID has to start with "user_" or the request is rejected outright with a 400 error.

Key decisions

Manual offset commits, not Kafka's auto-commit

The consumer explicitly turns off Kafka's automatic "mark this as read" behavior, and instead tells Kafka "I'm done with this message" by hand, only after the work is actually finished.

Alternative considered: Leave Kafka's default auto-commit on, which marks a message as read on a timer, regardless of whether your code actually finished processing it.
Tradeoff: Auto-commit can mark a message "done" before your code finished with it — a crash in that window loses the message forever. Manual commits mean a bit more code for a real guarantee.
If reversed: A crash between "Kafka marked this read" and "the code actually finished" would silently lose that payment — never reprocessed, nothing flagging it happened.

Auto-commit (Kafka default)

Manual commit (what was built)

Real, database-enforced idempotency — not just an in-memory check

Duplicate messages are a certainty in a system like this, not an edge case. Before doing anything else, the consumer tries to insert the event's ID into a table where each ID can only exist once, and treats a rejected insert as "already done."

Alternative considered: Check an in-memory set or cache of "already processed" IDs instead of a real database constraint.
Tradeoff: An in-memory check is faster, but forgets everything on restart and breaks with more than one consumer process — the database check survives both.
If reversed: Without a real, persistent uniqueness check, a redelivered message would run the balance update a second time — a silent double-charge.

The database write and the Kafka publish are two separate steps, not one atomic unit

This is the most honest thing to say about this design: the payment gets written and committed *before* the event is published to Kafka, as two separate steps with nothing tying them together.

Alternative considered: An "outbox" pattern — write the event to the same database, in the same transaction as the payment row, then relay it to Kafka separately.
Tradeoff: The outbox pattern is more robust but meaningfully more machinery. What's shipped is simpler, and honest about the tradeoff: a Kafka publish failure after the DB commit leaves an orphaned "pending" payment with no automatic recovery.
If reversed: This isn't really reversible without adding the outbox machinery — the risk described is the direct, current consequence of not having it yet.

What's shipped

Outbox pattern (not built)

A dead-letter queue that separates failures cleanly — but doesn't yet close the loop on them

When something goes wrong processing a payment, the event is moved to its own channel with the error attached, instead of crashing the consumer or getting silently dropped.

Alternative considered: Build the full second half too: give the DLQ topic explicit partitions, and have its consumer actually store failed events somewhere reviewable and reprocessable.
Tradeoff: Shipping "quarantine failures separately" first protects the main stream immediately. The reprocessing half is real additional work that hasn't been built yet.
If reversed: Without even the current DLQ, a processing failure would either crash the whole consumer or silently disappear with no record at all.

Bare processes, not containers or a CI/CD pipeline

Unlike some of my other projects, the API and both consumers here run as plain Python processes, not inside Docker, and there's no automated deploy pipeline for this repo at all.

Alternative considered: Containerize the API and consumers and wire up a CI/CD workflow, the way the RAG and gRPC projects do.
Tradeoff: This project's focus was the messaging and consistency patterns themselves, not deployment tooling — a real, current limitation, listed honestly rather than implied otherwise.
If reversed: N/A — this reflects what exists today, not a design choice being defended as ideal.

When the database commit succeeds but the Kafka publish doesn't

Not every failure here is a crash mid-consumer — this one is quieter and happens entirely inside the very first request, before a consumer is even involved.

Step 1 of 7

1. A payment request arrives

A client sends POST /payments. FastAPI validates the user_id format and opens a database connection.

Two views of the same consumer

Happy path

Failure path

Either the whole transaction commits and the offset is marked done, or none of it does and the event is safely quarantined instead — there's no in-between state.

Future work

Close the database/Kafka consistency gap — either an outbox pattern or a reconciliation job that finds "pending" payments with no matching Kafka event and retries or flags them
Explicitly provision the DLQ topic with a real partition count instead of relying on broker auto-create
Give the DLQ consumer an actual reprocessing path — persist failed events somewhere reviewable, instead of only logging them
Add configurable retry with backoff before an event is routed to the DLQ at all, instead of failing straight to it on the first error
Add real observability — structured logging, metrics, tracing — since right now the only visibility into consumer health is log lines
Add a real payment-gateway simulator to validate balances against, instead of balance-as-bookkeeping with no floor at zero

Code excerpts

The idempotent insert

A genuinely atomic, database-enforced duplicate check — not a check-then-insert race, since the conflict handling is built into the single INSERT itself.

cursor.execute("""
    INSERT INTO processed_events (event_id) VALUES (%s)
    ON CONFLICT (event_id) DO NOTHING
""", (event_id,))
if cursor.rowcount == 0:
    return  # already handled this exact event

Where the DB commit and Kafka publish diverge

The exact ordering behind the failure scenario above — two independent steps with nothing tying them together.

conn.commit()          # payment row now permanently exists
produce_event(event)   # separate step — can still fail on its own

View source on GitHub