13 min
Real-Time Event-Driven Payment Processing Backend
- FastAPI
- Apache Kafka
- PostgreSQL
- Python
The happy path is simple: accept a payment, publish an event, process it. The harder question is what happens when only half of that sequence succeeds.
I built this system to explore event-driven payment processing with Kafka, then started deliberately testing the assumptions underneath it: duplicate delivery, consumer crashes, offset handling, failed events, and the gap between PostgreSQL and Kafka.
The original goal was straightforward: build an event-driven payment-processing backend that accepts a request over HTTP, hands the real work off asynchronously, and confirms the request without waiting for that work to finish. Separating synchronous ingestion from asynchronous processing with FastAPI and Kafka wasn't the hard part; wiring a producer and a consumer together is a well-worn pattern.
Once the basic pipeline worked, I stopped adding features for a while and started asking a different question: where could the state of a payment become inconsistent? Not "does it work" but "what happens when it doesn't": a crash mid-consumer, a redelivered message, a gap between two systems that both think they're the source of truth.
That question ended up being more interesting than the original feature list. The rest of this write-up follows that thread: what the system protects today, where it still doesn't, and why.
Assumed knowledge: HTTP APIs, relational databases, transactions, and basic asynchronous processing. Kafka-specific concepts (partitions, consumer groups, offsets, delivery semantics, idempotency, dead-letter queues) are explained where they affect this system.
What must remain true?
- The same event ID must not change state twice.Protected
- Kafka should not acknowledge unfinished work.Protected
- Failed events should remain discoverable.Protected
- A payment committed to PostgreSQL should eventually reach downstream processing.Known gap
What the first working version looked like
Trace a request step by step, or reset to see the whole picture.
The full pipeline: step through a request below, or click a step directly.
- The client sends POST /payments with a user ID and an amount.
- FastAPI validates the request, writes the payment row to Postgres as pending, and commits; that row now exists permanently, regardless of anything that happens next.
- Only after that commit does FastAPI separately produce an event to Kafka, then return a response to the client, without waiting for the payment to actually be processed.
- Sometime later, at its own pace, the consumer polls that event off Kafka.
- Before touching any balance, it tries to register the event's ID in a table that only accepts each ID once; if that fails, it already knows this is a redelivery and stops right there.
- Otherwise, it updates the balance and marks the payment processed, inside one database transaction.
- Only once that transaction commits does the consumer tell Kafka it's done with the message and commit the offset.
- If processing throws instead, the event goes to the dead-letter queue rather than crashing the consumer or disappearing silently.
Now break the system
The easiest way to understand the design is to break it at specific points and follow what happens next.
Consumer crashes before committing the offset
Kafka delivers evt_482
The consumer's poll loop receives the event and begins processing it.
Click a point on the timeline to see what's happening there. Total span shown: 8s.
The system does not depend on the event being delivered only once. It makes redelivery safe.
Kafka durably retains events according to its replication, acknowledgment, and retention configuration, while consumers track progress using offsets. A crash before the offset commits can cause a redelivery, and the consumer is built to tolerate that rather than assume it won't happen, enforced by the event_id uniqueness check in Postgres.
The idempotent insert
The duplicate check is database-enforced in one atomic statement, not a separate check-then-insert sequence that could race.
cursor.execute("""
INSERT INTO processed_events (event_id) VALUES (%s)
ON CONFLICT (event_id) DO NOTHING
""", (event_id,))
if cursor.rowcount == 0:
return # already handled this exact eventReliability, mechanism by mechanism
Three small design choices do most of the work described above.
Making redelivery safe
The consumer needs to recognize "I've already handled this event" before it touches any balance, and that recognition has to be atomic, or it doesn't actually protect anything.
A pattern that can race
if event_id not in processed_events:
insert(event_id)What's actually running
INSERT INTO processed_events (event_id) VALUES (...) ON CONFLICT (event_id) DO NOTHING;
Deciding when work is actually complete
Kafka's default behavior is to advance a consumer's on a timer, regardless of whether your code actually finished with that message. This project turns that off and commits the offset by hand, only once the database work has committed.
AUTO-COMMIT
offset may advance
before application work is complete
MANUAL COMMIT
database work succeeds
↓
offset committedWhy it matters: If the advances past an event before the application-side work finishes, a crash can leave the group with no normal reason to redeliver that event, since Kafka has no way to know the work never actually happened. Manual commits keep "Kafka thinks this is done" aligned with "the database agrees it's done."
Isolating failed events
When processing throws partway through, the event is moved to a dedicated dead-letter topic instead of crashing the consumer or retrying forever inline.
Solves
visibility, isolation, continued processing of everything behind it
Does not solve
root-cause analysis, repair, safe replay, a reprocessing strategy
Those are still manual today; the DLQ consumer currently just logs what arrives.
How much parallelism do four partitions actually give me?
The Failure Lab above is about correctness under crashes; this is about throughput under load, and it runs into a different kind of limit.
This is a conceptual assignment visualization: the exact algorithm Kafka uses to assign partitions may differ, but one partition is owned by at most one consumer within a consumer group at a time.
| Consumer | P0 | P1 | P2 | P3 |
|---|---|---|---|---|
| C1 |
Within one consumer group, useful parallelism is bounded by partition count; a 5th or 6th consumer in the same group just sits idle.
1,000 simulated events, 4-partition topic
This is a learning benchmark, not a production throughput claim.
The failure my current design does not solve
PostgreSQL succeeds. Kafka doesn't.
FastAPI ├── PostgreSQL commit ✓ └── Kafka publish ✕
Postgres and Kafka are two separate systems with no shared transaction between them. FastAPI commits the payment row first, then, as a second and independent step, tries to publish the corresponding event to Kafka.
A normal Postgres transaction can't atomically include a Kafka publish; there's no two-phase commit tying the two together. So the success of the first operation says nothing about whether the second will succeed.
If the broker is unreachable, or the producer's buffer is full, that publish can fail outright, or hang, since the flush call here has no timeout. Either way, Postgres now has a payment that Kafka has never heard of, and no consumer will ever see it.
The obvious-looking fix, retrying the publish, doesn't fully solve it either. A retry after an ambiguous failure (a timeout, a connection reset) can't always tell whether the first attempt actually landed, so blind retries can create a second, duplicate event for the same payment.
That's exactly why consumer-side , covered above, still matters even if the producer side gets more reliable: at-least-once delivery is what a producer retry buys you, and duplicates are the cost of that guarantee.
One possible next step: transactional outbox
Not yet implementedONE PostgreSQL transaction
├── payment row
└── outbox event row
outbox publisher
↓
KafkaWriting the payment row and an "" event row in the same Postgres transaction means either both exist or neither does, with no window where one system knows about a payment the other doesn't. A separate publisher process then relays outbox rows to Kafka at its own pace.
Tradeoff: This closes the gap described above, but adds a new moving part, the publisher, with its own failure mode: it can crash after sending to Kafka but before marking a row relayed, producing a duplicate publish. Consumers still need idempotency either way.
A simpler, complementary option is reconciliation: identify payments that remain pending beyond an expected processing window, and reconcile them against downstream processing records; no new write path, just a periodic check over the existing one.
Reliability patterns do not eliminate failure. They move the failure boundary to somewhere easier to recover from.
A few more decisions worth explaining
Why manual offset commits?
I wanted the offset commit to mean something specific: that the database work for this message actually finished, not that a timer elapsed.
Alternative considered: Leave Kafka's default auto-commit on, which advances the offset on a timer independent of whether the application finished processing.
Tradeoff: Manual offset control increases implementation responsibility (every code path, including failures routed to the , has to explicitly decide when it's safe to commit), but it lets processing completion align more closely with persisted application state instead of a timer.
Why database-enforced event idempotency, not an in-memory check?
I wanted the "have I processed this event?" decision to be atomic. A database uniqueness constraint avoids a separate check-then-insert race and makes duplicate detection part of the transaction boundary.
Alternative considered: Track "already processed" IDs in an in-memory set or cache instead of a database constraint.
Tradeoff: An in-memory check is faster, but forgets everything on restart and doesn't hold up once there's more than one consumer process; the database constraint survives both, at the cost of one extra write per event.
Why key events by user_id?
Kafka events for this project are produced with the user ID as the key, so every event for a given user lands in the same partition and is processed in the order it was produced.
Alternative considered: Random or unkeyed distribution, letting the producer spread events across partitions however it likes.
Tradeoff: Preserving per-user ordering constrains where those events can be processed: every event for one user is bound to a single partition, and therefore to whichever single consumer owns it, rather than being freely spreadable across the whole group. It can also create uneven partition load if some users generate substantially more traffic than others, since their events all queue behind the same partition regardless of how idle the rest of the topic is.
Why a DLQ that isolates failures but doesn't yet close the loop on them?
I wanted a failed event to stay visible and out of the way of everything behind it, without having to solve reprocessing on day one.
Alternative considered: Build the full second half too: give the DLQ topic explicit partitions, and have its consumer store failed events somewhere reviewable and reprocessable.
Tradeoff: Shipping "quarantine failures separately" first protects the main stream immediately. It also creates an operational responsibility this project hasn't built yet: investigating and eventually reprocessing or resolving those events safely.
Code that proves the claim
The idempotent insert
The duplicate check is database-enforced in one atomic statement, not a separate check-then-insert sequence that could race.
INSERT INTO processed_events (event_id)
VALUES (%s)
ON CONFLICT (event_id) DO NOTHING;Where the DB commit and Kafka publish diverge
These operations succeed or fail independently; see "The failure my current design does not solve" above for what that means when the second one doesn't.
conn.commit() # payment row now permanently exists
produce_event(event) # separate step, can still fail on its ownOpen problems I want to test next
PostgreSQL → Kafka consistency
Explore the transactional outbox pattern above, or a reconciliation job, to close the gap described in that section.
DLQ recovery
Build safe replay tooling; right now a failed event can be seen, but not reprocessed, without manual intervention.
Retry strategy
Add backoff and retry classification before an event is routed to the DLQ at all, instead of failing straight to it on the first error.
Observability
Track processing latency, consumer lag, failure counts, and DLQ volume; right now the only visibility into consumer health is log lines.
External processor simulation
Simulate a real payment processor's failure modes (timeouts, ambiguous outcomes, retries) and test reconciliation against them.
What this project taught me
Kafka doesn't remove application-level correctness problems; it relocates them. At-least-once delivery means duplicates are a normal, expected outcome, not an edge case, and that pushes the responsibility for correctness into how the consumer is written, not into the broker's guarantees.
Database constraints turned out to be one of the more useful tools here, not just for data integrity but as a concurrency primitive: a unique constraint with ON CONFLICT DO NOTHING answers "have I seen this before?" atomically, in a way an in-memory check can't.
"Processing succeeded" only means something once you've defined which state transitions actually need to be durable; the offset commit exists specifically to encode that definition in a way Kafka can act on.
And the hardest failures in this project weren't inside any single component; Postgres and the consumer are both individually reliable. They showed up at the seam between two systems that don't share a transaction, which is exactly where the gap above still lives.