16 min
Agentic RAG System
- Python
- MCP
- Qdrant
- FastAPI
- Docker
- Nginx
- Next.js
- TypeScript
- GitHub Actions
Getting an LLM to produce an answer was not the part I found most interesting. The harder problem was deciding what evidence it was allowed to use, showing that evidence to the user, and making the application refuse an answer when retrieval was too weak.
This is a Q&A system over Anthropic's API documentation. An agent retrieves through a custom MCP server backed by a Qdrant vector store, and FastAPI wraps that agent with a second, independent retrieval call and a deterministic abstention guard — so the model's confidence is never the only thing standing between a user and a wrong answer.
This is a retrieval-augmented generation () system: a question comes in through a FastAPI backend, which hands it to an OpenClaw . The agent's only path to real data is a set of four tools exposed by a custom server, which the query and searches a collection of ingested Anthropic documentation. The agent its answer in whatever comes back and is instructed to cite it.
The frontend doesn't hide any of this. Alongside the answer, it renders a — the actual chunks retrieved, their , and their source URLs — so a reader can see the evidence the system is claiming to rely on, not just trust the prose.
FastAPI adds one more layer: it calls the same retrieval tool a second time, independently of the agent, and uses that independent result to decide whether the evidence was strong enough to answer at all. If it wasn't, the application overrides the model's output — not because the model asked to be overridden, but because deterministic code, not the model, gets the final say. And the whole thing is deployed on a real VPS, not left running only on localhost.
Assumed knowledge: basic APIs, Python services, and the general idea of an LLM. Embeddings, vector retrieval, RAG, MCP tool use, grounding, and abstention are explained where they affect this system.
What must remain true?
- The agent retrieves documentation before answering.Behavioral rule
- The agent has no direct access to Qdrant.Architecturally constrained
- Weak retrieval does not result in a confident answer.Protected by FastAPI guard
- The evidence shown to the user corresponds to the evidence the answer was actually grounded in.Known residual assumption
What the system actually looks like
There are two retrieval paths in this architecture, and that distinction matters later.
Follow one question through the system
Trace one request step by step, or reset to see the whole architecture.
The full pipeline: step through a request below, or click a step directly.
Before any question can be answered
The request architecture above doesn't show how the knowledge base gets built — that happens once, ahead of time, not on the request path.
ingestion/ingest.py downloads docs.anthropic.com/llms-full.txt, chunks it, embeds every chunk, and upserts into a freshly recreated Qdrant collection — a full run (including a first-time ~550MB model download) takes about 24 minutes on a Hetzner AX41 dedicated server, embedding- and CPU-bound.
Chunking is header-aware, not a blind fixed-size split. The script splits each page on its ##/### Markdown headings first, then accumulates adjacent sections into a buffer until it lands in a 300–400 token target range (50-token overlap between chunks) — only a section that's individually oversized on its own gets pushed through a token-bounded fallback splitter. This keeps section_heading and parent_heading meaningful: they're the labels a reader sees in the retrieval inspector and the citations the agent produces, not an arbitrary character offset.
Every chunk carries its source_url and heading metadata alongside its text, all the way through to Qdrant's payload — that's what lets a retrieved chunk be cited back to a real page, not just quoted anonymously.
Retrieval itself is a vector search over nomic-embed-text embeddings, not a keyword or full-text match. Lexical search emphasizes matching terms; vector similarity can rank passages by semantic proximity even when the phrasing differs — a question about "rate limits" can still surface a passage about "request throttling." The corpus lands at roughly 3,500 chunks after this process, verified by a vector-count check (verify.py) in the 2,500–3,500 range.
The architectural boundary
"Only use the approved tool" is an instruction. It's not the same kind of guarantee as "the credentials to do anything else don't exist." This project is built around that distinction.
Instruction-only architecture
The path the prompt asks the agent to take.
A path the prompt asks it not to take — but that it's technically still able to.
A working client and real credentials sit right next to the model. Only a system prompt stands between the agent and using them directly.
This project
The only path that exists.
There's nothing here to call, even if the model wanted to.
The agent is only ever given the four MCP tools. It has no Qdrant client object and no Qdrant credentials in its environment — there's no direct route to bypass, not just one it's told not to take.
The model is still probabilistic — it can still misuse a tool, retrieve the wrong thing, or answer poorly. But which interfaces are even reachable from the agent's position is not a matter of the model's judgment; it's provisioning. This application exposes four retrieval tools through MCP, and the agent is not given another path to Qdrant.
The four tools
Rather than an unrestricted Qdrant client, the agent gets four narrow, purpose-built tools.
| Tool | Purpose |
|---|---|
| search_kb | Semantic retrieval for a question |
| get_source | Fetch the exact stored text for one chunk by ID |
| list_sections | List every topic the corpus covers |
| get_related | Find chunks similar to a given chunk |
Narrow, named tools do two things a generic database client wouldn't: they make every retrieval operation exposed to the agent enumerable ahead of time, and they let the server enforce shape and behavior (embedding, ranking, payload fields) on every call, rather than trusting the agent to construct a correct query.
The transparency feature has its own trust problem
Showing retrieved evidence looks like transparency. But that's only useful if the evidence shown is actually the evidence that grounded the answer — and this system doesn't currently prove that directly.
FastAPI's retrieved_chunks field doesn't come from capturing the agent's own internal . It comes from a second, independent search_kb call FastAPI makes itself, using the same question text and the same top_k the agent is expected to use. OpenClaw's deployed gateway doesn't currently expose the agent's own bundled MCP tool-call result through its /tools/invoke route, so there was no way to literally read back what the agent retrieved — this independent call is the substitute.
The expected case: both searches use the same question
Agent path
Inspector path
The residual risk: the agent doesn't search verbatim
If the agent reformulates the question before searching
POSSIBLE DIVERGENCE — inspector no longer matches
Retrieval is deterministic given identical inputs — same query, same top_k, same index state, same model. That's exactly why agent/SOUL.md's rule 1 explicitly instructs the agent to pass the question to search_kb exactly as the user wrote it, without paraphrasing, rewriting, expanding, or correcting it first. If the agent reformulates the query before searching, its retrieval can diverge from FastAPI's independent one, and the inspector panel would show chunks that weren't actually what grounded the answer.
So the honest claim isn't "the retrieval inspector proves exactly what the model saw." It's narrower: the inspector is authoritative only if the agent actually searched using the exact same query and retrieval parameters FastAPI used — a rule the instructions enforce, but that nothing currently verifies after the fact. A feature built to make the system transparent still carries its own unverified assumption.
Grounding is also knowing when not to answer
FastAPI computes a low-confidence signal from the independently retrieved chunks: if there are none, or if the best cosine-similarity score among them falls below 0.6, the evidence is treated as too weak to answer from.
| Example | Best similarity score | Result |
|---|---|---|
| Strong retrieval | 0.82 | Answer |
| Borderline | 0.64 | Answer |
| Weak retrieval | 0.51 | Abstain |
Illustrative, not measured production examples. A similarity score is a ranking signal, not a calibrated probability — 0.82 does not mean "82% confidence."
0.6 is an explicitly disclosed interim heuristic, not a validated threshold — the project's own SOUL.md says so directly. It came from a single 10-question manual retrieval test (8 in-corpus, 2 off-corpus), where in-corpus top scores landed between 0.64 and 0.90 and off-corpus top scores landed between 0.43 and 0.56. 0.6 separates that one small sample better than an earlier 0.4 guess did — nothing more. It's planned to be replaced with a relative-margin or learned-threshold approach instead of a fixed cutoff.
Code gets the final say
Two independent signals are OR'd together: the agent's own exact refusal sentence, and FastAPI's own check of the real retrieved_chunks scores. In testing, the agent didn't always reproduce its mandated refusal sentence verbatim even when scores were clearly low — on one off-corpus question it improvised a differently-worded refusal instead of quoting the required sentence, and on another it answered from general knowledge instead of abstaining at all. Trusting the model's own text alone wasn't reliable enough on its own.
So when either signal says abstain, FastAPI overwrites the answer with the fixed sentence itself, regardless of what the model actually returned. The LLM doesn't get final authority over whether an answer is shown to the user — the application code does.
The abstention override
main.py's /ask handler — the returned answer is guaranteed to be the refusal sentence whenever abstained is true, whether or not the model actually complied.
low_confidence = (
not retrieved_chunks
or max(c["score"] for c in retrieved_chunks) < ABSTENTION_SCORE_THRESHOLD
)
abstained = (ABSTENTION_SENTENCE in answer) or low_confidence
if abstained:
answer = ABSTENTION_SENTENCEMaking the system reachable
The backend works fully over plain HTTP — curl http://<vps-ip>:8000/ask is enough to exercise the whole system. Making it reachable from a browser-hosted, HTTPS-deployed frontend introduced two separate problems that don't show up when everything's local: mixed-content blocking, and CORS.
FastAPI and the MCP server run as systemd services directly on the VPS (not containerized); only Qdrant runs via Docker Compose. A self-hosted GitHub Actions runner on the same VPS restarts the FastAPI and MCP server systemd services and ensures the separately managed OpenClaw gateway container is running on every deploy — it never re-runs ingestion, which stays a deliberate manual step.
Nginx + Certbot exists only to solve mixed-content blocking: a browser won't let an HTTPS page (the Vercel-hosted frontend) call a plain-HTTP API, so a TLS-terminating reverse proxy sits in front of FastAPI's port 8000. It's explicitly optional in the project's own deployment docs — the backend is fully testable without it.
CORS is a separate, always-on restriction: FastAPI's CORSMiddleware only allows requests from an explicit ALLOWED_ORIGINS allowlist read from the environment, not a wildcard — /ask has no auth of its own, so any origin would mean any website's JavaScript could call it on a visitor's behalf.
Both FastAPI and the MCP server run as systemd services (Restart=always / Restart=on-failure) so they survive a disconnected SSH session or a reboot without anyone manually restarting them — the same reasoning that made systemd the right call for the payment backend's consumer process. Only Qdrant runs in Docker; the OpenClaw agent gateway runs as its own separately-managed container, registered once during onboarding rather than redeployed by this repo's own workflow.
A few decisions worth explaining
Why make MCP the retrieval boundary?
The agent needed a way to reach Qdrant, and the easiest version of that would have been handing it a database client directly, gated by a prompt instruction not to misuse it. MCP made it possible to instead provision the agent with only four narrow tools and nothing else — turning "please only use the approved tool" from a convention into the only thing that's actually reachable.
Alternative considered: A direct Qdrant client passed into the agent's context, with the system prompt instructing it to only call an approved wrapper function around it.
Tradeoff: MCP adds a server process and a protocol hop that a direct client wouldn't need. What it buys back is that the restriction lives in what's provisioned, not in what the model is told — a distinction that matters precisely when the model doesn't behave as instructed.
Why perform a second retrieval in FastAPI?
The retrieval inspector, the abstention check, and citation assembly all need real chunk data with real scores — and OpenClaw's gateway doesn't expose the agent's own internal tool-call result through its /tools/invoke endpoint in the deployed version. An independent search_kb call was the practical way to get that data at all.
Alternative considered: Wait on an upstream gateway change that surfaces the agent's actual tool-call result, and block the retrieval inspector feature until it exists.
Tradeoff: This ships the feature now, at the cost of the fidelity assumption described above — the independent call is expected to match the agent's real retrieval, not proven to.
Why let application code overrule the model?
Testing surfaced real cases where the agent didn't reliably abstain even when it clearly should have — sometimes improvising a different refusal, sometimes answering anyway. An instruction the model can silently skip isn't a safeguard by itself.
Alternative considered: Trust the model's SOUL.md instructions alone, and treat the abstention sentence as the only signal.
Tradeoff: The override can occasionally suppress an answer the model actually got right, if the independent retrieval score happens to land under 0.6 on a real in-corpus question. Consistency was chosen over that risk.
Why ship a provisional threshold?
A cutoff derived from a 10-question manual test is still better than no abstention behavior at all, and shipping it honestly labeled as provisional was preferable to blocking the feature on a proper calibration effort.
Alternative considered: Hold off on any abstention guard until a larger, labeled retrieval-quality dataset existed to calibrate a real threshold.
Tradeoff: 0.6 will misclassify some real questions near the boundary in both directions. The project's own docs already flag this and name the intended fix (a relative-margin or learned threshold) rather than presenting 0.6 as settled.
Why deploy the system instead of stopping at localhost?
A RAG pipeline that only runs in a notebook doesn't have to survive CORS, TLS, process restarts, or a real reverse proxy — problems that are trivial to wave away locally and very real the moment a browser on a different origin needs to call it.
Alternative considered: Demo the pipeline through a local script or notebook, with screenshots of a request/response pair.
Tradeoff: Deployment adds real operational surface — systemd units, an nginx config, a CORS allowlist, a CI workflow — that a local demo never has to get right. It's also the only way any of the transparency or abstention behavior described above is something a real visitor can actually exercise.
Code that proves the claim
Answer generation and evidence retrieval run concurrently
main.py's /ask handler — the agent's answer and FastAPI's independent retrieval are two separate coroutines, not a wait-then-wait sequence.
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
retrieved_chunks, answer = await asyncio.gather(
_search_kb_direct(question),
_ask_agent(client, question),
)search_kb, the MCP tool the agent actually calls
mcp_server/server.py — embeds the query, runs a cosine-similarity vector search, and returns ranked chunks with scores and source metadata.
@mcp.tool()
def search_kb(query: str, top_k: int = 4) -> list[dict[str, Any]]:
vector = embed_query(query)
hits = _client.query_points(
collection_name=QDRANT_COLLECTION,
query=vector,
limit=top_k,
with_payload=True,
).points
return [_point_to_result(hit.payload or {}, score=hit.score) for hit in hits]The CORS allowlist
main.py — an explicit origin allowlist from an environment variable, not a wildcard, since /ask has no auth of its own.
ALLOWED_ORIGINS = [o.strip() for o in os.environ.get("ALLOWED_ORIGINS", "").split(",") if o.strip()]
app.add_middleware(
CORSMiddleware,
allow_origins=ALLOWED_ORIGINS,
allow_methods=["POST", "OPTIONS"],
allow_headers=["Content-Type"],
)Where the guarantees stop
Inspector fidelity isn't structurally guaranteed
FastAPI's retrieved_chunks comes from an independent search_kb call, not a captured copy of what the agent itself retrieved. It's expected to match because retrieval is deterministic given identical input and the agent is instructed to search verbatim — but nothing currently verifies that the agent actually did.
0.6 is provisional
The abstention threshold came from a 10-question manual test, not a calibrated evaluation. It's disclosed as an interim heuristic in the project's own instructions, not presented as settled.
Grounding still depends partly on instructions
The architecture removes direct Qdrant access, which is a real constraint. But whether the agent calls search_kb first, uses the exact question text, cites correctly, and answers only from retrieved context is still governed by SOUL.md's instructions, not enforced by the runtime. This system doesn't make hallucination impossible — it makes certain kinds of it harder and easier to catch.
Testing isn't a mature automated evaluation suite
mcp_server/test_client.py is a manual verification script — it exercises all four tools and runs a fixed 10-question retrieval-quality check, printed for a human to read, with no assertions and no CI integration.
A stale number in the MCP tool's own docstring
search_kb's docstring still says "if every returned score is below 0.4, the corpus likely doesn't contain a reliable answer" — a leftover from before the threshold was raised to 0.6. The actual enforced cutoff, in both SOUL.md and main.py, is 0.6. This is a documentation inconsistency worth cleaning up, not the active behavior.
Open problems I want to validate next
- Capture the agent's actual retrieval result directly, removing the independent-search fidelity assumption entirely.
- Build a labeled query/evidence set and run a real retrieval evaluation, instead of a 10-question manual spot check.
- Replace the fixed 0.6 cutoff with a relative-margin or learned-threshold approach, as already planned in the project's own docs.
- Turn the manual test scripts into an automated test suite with real assertions, wired into CI.
- Test and document behavior when Qdrant, the MCP server, or the OpenClaw gateway is unavailable or times out mid-request.
- Add real observability: retrieval latency, score distributions, abstention rate, MCP error rate, and end-to-end request latency.
What this project changed in how I think about AI systems
Reliable grounding turned out to be a system property, not just a prompt instruction. The strongest guarantee in this project — no direct Qdrant access — comes from what's provisioned to the agent, not from what it's told. The weakest one — verbatim search, correct citation, honoring abstention — is still just an instruction, and testing showed instructions get skipped.
Tool access and model instructions are different kinds of controls, and it's worth being precise about which one is actually doing the work in a given claim. "The agent can only use these four tools" is an architectural fact. "The agent will always cite correctly" is a hope backed by a system prompt.
Transparency features need their own scrutiny. A retrieval inspector that shows the wrong chunks is arguably worse than no inspector at all, because it looks trustworthy while being subtly wrong. Building the feature honestly meant naming its own residual assumption, not just shipping the panel.
A similarity score is a ranking signal, not a probability, and treating it as one would have made the abstention guard look more rigorous than it is. Its actual justification — a small manual test, explicitly labeled provisional — is more honest and, I think, more useful to a reader than a confident-sounding number would have been.
And deployment turned an AI demo into a software system with ordinary engineering problems: networking, CORS, process supervision, TLS, and a redeploy workflow that has to survive a clean checkout wiping a gitignored .env file. None of that is specific to RAG or agents — it's just what happens once something has to actually be reachable.