Agentic RAG System
- Python
- MCP
- Qdrant
- FastAPI
- Docker
- Nginx
- Next.js
- TypeScript
- GitHub Actions
An AI agent is only as trustworthy as the boundary around what it's allowed to touch. This project explores a specific version of that problem: what happens when an agent's only path to real data is through tools it doesn't control — and how do you actually prove, after the fact, that it stayed inside that boundary?
Foundations
Everything below builds up, one idea at a time, to what this project actually is. Click around — most of it is meant to be played with, not just read.
API
An API is just a way for two programs to talk to each other — one side sends a request, the other sends back a response. This project's frontend sends a question to a `/ask` endpoint and gets an answer back.
The one public entry point this whole system exposes.
Database vs. vector database
A normal database finds rows by exact match — search for "rate limit" and you only get rows containing those exact words. A vector database searches by meaning instead, so a question about rate limits can still find a passage about "request throttling" even though they don't share a word.
Normal database — exact match
| chunk | text |
|---|---|
| #12 | "...avoid getting rate limited..." |
| #47 | "...request throttling helps avoid 429s..." |
Only the row with the literal words "rate limit" matches — the throttling row is just as relevant, but never gets found.
Vector database — meaning match
Click a point to see what it's close to in meaning.
Embedding
An embedding is what a vector database actually stores: each piece of text gets converted into a list of numbers that captures what it means. Text with similar meaning ends up with numbers that are mathematically close together — so "close in meaning" becomes "close on a map."
Click a point to see what it's close to in meaning.
Click any point — it's grouped with the other points closest to it in meaning, not by shared letters or words.
Retrieval / semantic search
Semantic search means turning a question into that same kind of number-list, then finding the document chunks whose numbers are closest to it. This is how the system finds a passage about "throttling" from a question that never uses that word.
Click a point to see what it's close to in meaning.
Click the diamond (your question) to see which stored passage it lands closest to.
LLM, and its core limitation
An LLM (large language model) is the AI doing the actual writing. Left alone, it can only answer from what it memorized during training — and it can guess confidently even when it's wrong, with no visible difference between a fact and a fluent-sounding mistake.
Without real documents
With real documents (RAG)
RAG (Retrieval-Augmented Generation)
RAG just means combining the two ideas above: retrieve the real passages that match the question by meaning, then have the model read those before answering, instead of relying on memory alone.
Agent / tool use
An "agent" is a language model that's been given tools (functions) it can call, and instructions on how to use them — instead of just being asked a question and writing text back, it can decide to go fetch information first.
MCP (Model Context Protocol)
MCP is a standard way to hand an agent a fixed set of tools and nothing else — like giving someone a remote control with exactly four buttons instead of the keys to the whole building. This project's agent can search, but it has no other way to reach the document database.
The top path doesn't exist for the agent — the only way through is the 4 tools MCP exposes.
Overview
This is a documentation Q&A system for Anthropic's API docs — ask something like "how does prompt caching work?" and it searches the real docs, reads the matching passages, and answers only from those, citing exactly where each claim came from.
The interesting engineering problem is the boundary around that search: the agent has zero database credentials and can only reach the documents through the 4-tool MCP server above — not a rule it's asked to follow, but a wall it's architecturally unable to get around. The whole stack (FastAPI, the MCP server, and a self-hosted Qdrant vector database) runs on a real VPS under systemd, with a Next.js frontend showing a "retrieval inspector" panel — exactly which passages were retrieved and how confident the match was, so the answer isn't a black box.
Architecture
Click a component below to see what it does.
Click a component in the diagram above to see what it does.
Request flow, step by step
The question arrives
A question hits POST /ask. If it's empty or just whitespace, FastAPI rejects it immediately with a 400 error.
Key decisions
MCP as the only door in, not a convention the agent is asked to follow
The agent has zero database credentials. It can't query Qdrant directly even if it wanted to — its entire universe of possible actions is the 4 functions the MCP server hands it.
- Alternative considered
- Give the agent a database connection string or a direct query tool and just instruct it (in its prompt) to only use approved queries.
- Tradeoff
- An instruction is a suggestion, not a guarantee — a confused or manipulated model could go around it. MCP costs more upfront wiring, but the boundary is architectural, not requested.
- If reversed
- If the agent had direct DB access, "the agent can only retrieve through MCP" would become a claim about behavior, not a fact about what's possible.
Asked nicely (instruction only)
MCP (what was built)
A second, independent search call — to make the transparency panel honest
The retrieval-inspector panel can't just ask the agent "what did you look up?" — so FastAPI reruns essentially the same search itself and shows that.
- Alternative considered
- Try to capture the agent's own internal tool-call result directly from the platform running it.
- Tradeoff
- This only works if the agent searches using the user's exact question text — a rule written into its instructions, not something the code can force. That's a real, disclosed residual risk.
- If reversed
- Without the independent call, the retrieval-inspector panel would have no real data to show — the platform doesn't currently expose the agent's internal tool calls.
Letting code overrule the model's own words
When the system decides an answer isn't well-grounded, it doesn't hope the model phrases its uncertainty well — it deletes whatever the model said and substitutes one fixed, pre-written sentence.
- Alternative considered
- Trust the language model's own refusal wording when it decides to decline.
- Tradeoff
- Trusting the model is more flexible, but it was directly observed, in testing, not doing that reliably — on at least one occasion the model answered from general knowledge instead of admitting weak results.
- If reversed
- Without the override, users would occasionally get a confidently-worded answer that was never actually backed by the retrieved documents.
Trust the model's wording
Code overrules if unsure
Shipping an honest, unvalidated confidence threshold instead of blocking on a perfect one
The cutoff for "this search result isn't good enough to answer from" is a single number, set from a small manual test — and the project says so, out loud, in its own instructions to the agent.
- Alternative considered
- Hold off shipping any abstention behavior until a properly calibrated, statistically validated threshold exists.
- Tradeoff
- A calibrated threshold is the better long-term answer, but needs far more labeled test data than a small project starts with. Shipping a documented best-guess threshold means the system behaves sensibly now, while being explicit the number is provisional.
- If reversed
- With no threshold check at all, a weak, barely-related search result would still get handed to the model as if it were solid grounding.
Deploying to a real, always-on server — not just a docker-compose demo
The backend runs on an actual VPS under systemd, the same process-supervision tool real Linux servers use, so it survives crashes and reboots without someone babysitting a terminal.
- Alternative considered
- Run everything in the foreground during a demo, or rely on a bare `docker compose up` with no restart policy.
- Tradeoff
- Real process supervision and CI/CD take more setup than a local demo, but the payoff is a system that's actually reachable and self-healing.
- If reversed
- A foreground process dies the instant its SSH session disconnects, with nothing bringing it back.
The retrieval-inspector's honesty problem
This isn't a crash or an outage — it's a subtler kind of failure: a transparency feature whose correctness depends on an instruction being followed, not on something the code can enforce.
1. A question comes in
A user types a question. FastAPI kicks off the agent's answer and its own direct verification search at the same time.
Future work
- Replace the fixed 0.6 abstention score cutoff with a relative-margin or learned-threshold approach — the project's own instructions already flag the current number as an interim heuristic from a 10-question manual test
- Fix the stale 0.4 value still referenced in the MCP search_kb tool's docstring, left over from before the real threshold was raised to 0.6
- Close the actual gap behind the retrieval-inspector workaround: capture the agent's own internal tool-call result directly once the gateway's tool-invoke endpoint supports it
- Add real automated test coverage — the existing verification scripts are genuinely useful but are manual/print-based smoke tests with no assert statements, not an automated pass/fail suite
Code excerpts
Code overruling the model's own words
Two independent signals — the model's own refusal text, or the code's own confidence check — can force the exact same honest sentence back to the user, discarding whatever the model actually said.
low_confidence = not chunks or max(c["score"] for c in chunks) < THRESHOLD
abstained = (ABSTENTION_SENTENCE in answer) or low_confidence
if abstained:
answer = ABSTENTION_SENTENCEOne of the agent's exactly 4 tools
This is the entire surface area the agent has for reaching the document database — one of 4 functions, and nothing else.
@mcp.tool()
def search_kb(query: str, top_k: int = 4) -> list[dict]:
vector = embed_query(query)
return qdrant.query_points(query=vector, limit=top_k).points