CertCoach — a RAG assistant that earns its answers

Studying for a cloud cert means living in three tabs at once: the exam guide, the service FAQs, and whatever half-remembered blog post the search engine served up. The answers exist — they’re just scattered, un-cited, and easy to get subtly wrong. CertCoach is my attempt to fix that: ask a question, get a concise answer grounded in the official corpus, with citations you can actually follow back to the source.

It’s a portfolio project, but I built it the way I’d build the real thing — because the interesting part of RAG isn’t wiring an LLM to a vector store. It’s the four stages between a raw document and a trustworthy answer, and knowing — with numbers — which parts actually pull their weight.

CertCoach is multi-tenant across three certs — AWS Solutions Architect Associate, GCP Associate Cloud Engineer, and HashiCorp Terraform Associate — sharing one codebase and one database, isolated by a tenant column that gets filtered before any similarity work happens. Everything runs on FastAPI, Postgres with pgvector, and models served through a managed LLM gateway, with Arize Phoenix tracing the whole request path.

Here’s how it comes together, stage by stage.

01 — Ingestion: turning docs into a searchable corpus

Ingestion is the offline pipeline that runs once and rebuilds whenever the corpus changes. A sources.yaml manifest lists every official doc — exam guides, service FAQs, well-architected frameworks — and the pipeline walks it end to end:

• Fetch each URL and check it’s worth keeping before saving it: the download actually succeeded, a file claiming to be a PDF really is one (not an error page in disguise), and an HTML page has enough real text to be a document — not an empty shell that only fills in once JavaScript runs. Every fetch is logged with its URL, a timestamp, and a content fingerprint (SHA-256), so I always know exactly what was ingested and when.
• Clean the raw HTML — strip <script>, <nav>, <footer> noise — then normalize everything to Markdown via markdownify (PDFs go through pdfminer.six). One format downstream, whatever the input.
• Chunk with a sliding window over tiktoken tokens: a 650-token target with 100-token overlap, but every cut snaps back to the nearest paragraph boundary so I never split a sentence in half. Overlap keeps a concept from being orphaned across two chunks.
• Embed in batches of 32 with text-embedding-3-large, truncated to 1536 dimensions via Matryoshka representation learning. The model natively outputs 3072 dims, but pgvector’s ANN index tops out at 2000 — and Matryoshka is built precisely so you can halve the dimensions for a small (~2–3%) quality cost. A cheap trade for keeping full index support.
• Store with an idempotent upsert: delete-then-insert per source, backfill prev/next chunk links for later context expansion, then rebuild the IVFFlat index so its centroids reflect the new vectors.

The payoff for all this bookkeeping shows up in the next stage: every chunk carries its citation, its neighbors, and its tenant, so retrieval can be both fast and honest.

02 — Retrieval: dense and sparse, in one database

Dense and sparse retrieval live in the same Postgres table — a common pattern, and the simplest thing that works at this scale. No separate search cluster to run or keep in sync. pgvector handles the embeddings (IVFFlat, cosine), a generated tsvector column handles keyword search (GIN index, BM25-style ts_rank_cd), and both are just columns on the chunks table.

A query fans out both ways at once:

1. Dense search — embed the query, pull the top 20 by vector distance. Great at meaning (“how do I make my app survive an AZ outage”).
2. Sparse search — BM25 over the full-text index, top 20. Great at exact terms (an IAM action like s3:GetObject, an EBS volume type like gp3, an instance type like t3.micro) that embeddings blur together.
3. Reciprocal Rank Fusion merges the two rankings by position, not score: 1/(60 + rank) summed across both lists. No training data, no tuning — it just rewards chunks that both methods agree on.
4. Context expansion — for the top 5, fetch each chunk’s stored prev and next neighbors in one batch query, so the LLM sees a coherent passage instead of a 650-token fragment cut from the middle of an explanation.

Hybrid matters because the two failure modes are opposite: dense retrieval misses rare literal tokens, sparse retrieval misses paraphrase. Fusing them covers both — and it’s the same insight that set up my most useful finding, below.

03 — Generation: answers that cite their work

Retrieval hands generation a set of expanded, cited passages. Generation’s job is to answer only from them.

The system prompt is deliberately strict: answer in 3–5 sentences, put citations in parentheses inline, and — a small rule that fixed a real annoyance — never open with a throat-clearing prefix like “Based on the provided context.” The very first word has to be a content word. Citations are deduplicated before they reach the response, so the same source doesn’t show up four times.

Beyond Q&A, the same generation layer powers quiz generation (JSON multiple-choice questions on a topic) and answer grading (an LLM scores a learner’s free-text answer against a reference, 0–1, with feedback). All three go through one model-agnostic interface defaulting to claude-sonnet-4-6.

One thing I want to be honest about: I tried a more elaborate prompt (v2) that forced the model to enumerate every relevant item and follow a fixed answer template. It regressed on both tenants — 0.804 vs 0.834 on AWS, and more sharply on GCP (0.523 vs 0.745). More prompt scaffolding isn’t automatically better, and I only knew it hurt because I measured it.

Here are the two prompts the eval compared. v1 (the concise default):

CRITICAL RULE: Your very first word must be a content word. NEVER start with
"Based on", "According to", "From the context", or any variant.

You are CertCoach, an expert tutor for cloud certification exams.
Answer the learner's question using ONLY the context provided.
Cite each claim with the source name in parentheses, e.g. (Amazon EC2 FAQs).
If the context is insufficient, say so clearly — do not fabricate.
Be concise: aim for 3-5 sentences unless the question demands more.

v2 (the elaborate one that lost):

You are CertCoach, an expert tutor for cloud certification exams.
Answer the learner's question using ONLY the context provided.

Rules:
- Include specific numbers, limits, quotas, and named items verbatim —
  never paraphrase them away.
- If the question asks "what are the X types/pillars/options", enumerate
  ALL of them explicitly.
- Cite each claim with the source name in parentheses, e.g. (Amazon EC2 FAQs).
- Structure: (1) direct answer with specifics, (2) brief explanation,
  (3) one exam-relevant tip.
- If the context is insufficient, say so clearly — do not fabricate.
- Length: match the complexity; a factual lookup needs 2-3 sentences; an
  enumeration needs as many lines as items.

The extra structure sounds helpful, but it pushed answers away from the terse, specifics-first shape the judge rewarded — which is exactly the kind of thing you can only catch by scoring it. Which brings me to the stage that decides everything.

04 — Eval: the stage that makes the rest trustworthy

Every claim above — the chunk size, the fusion, the prompt version — is only as credible as the harness that tests it. CertCoach has two gold sets I built by hand: 25 retrieval questions (each mapped to the source that should answer it) and 30 QA pairs (each with a reference answer), spread across all three tenants.

Retrieval is scored deterministically — recall@5, hit@1, MRR@10, NDCG@5 — the standard IR metrics, run in parallel across the gold set.

Answer accuracy uses an LLM-as-judge, with one deliberate twist: the judge is GPT-5 scoring Claude’s answers. Using a model to grade its own family invites self-preference bias; going cross-family keeps the scoring honest.

And the whole thing runs as an A/B sweep — four retrieval configurations in one command, printing a comparison table and writing the results to disk. That sweep is what turned an assumption into a decision.

The finding: eval killed my reranker

Conventional RAG wisdom says: retrieve broadly, then add a reranker to reorder the candidates with a stronger cross-encoder. I wired up Cohere’s reranker exactly as the playbooks recommend, fully expecting it to win.

The A/B eval said otherwise.

Configuration	Accuracy (judge)	P50 latency
dense only	baseline	~10 ms
hybrid, no rerank	0.761	~17 ms
hybrid + Cohere rerank	0.664	~3500 ms

The reranker made things worse on accuracy and ~200× slower. Not a wash — a clear loss on both axes.

The reason isn’t that rerankers are bad; it’s that mine had nothing to fix. At a ~300-chunk corpus, hybrid RRF already produces a strong top-5 — there just isn’t a long tail of near-misses for a cross-encoder to rescue. The reranker was solving a problem this corpus doesn’t have yet, and charging 2.4 seconds of round-trip latency to do it.

So hybrid_no_rerank became the production default. I kept the reranker behind a flag, because the calculus flips as the corpus grows — but I’m not paying for it until the eval tells me it’s worth it.

That’s the whole point of building the eval harness first: it let me delete a component I’d already built on the strength of numbers instead of vibes. The most valuable thing RAG evaluation gave me wasn’t a higher score — it was the confidence to make something simpler.

What I’d do differently

• Scale the corpus and re-run the sweep. The reranker finding is corpus-size-dependent by definition; I’d love to watch the exact point where it flips back to being worth the latency.
• Add retrieval-conditioned abstention to the eval. The system already abstains below a confidence threshold; I haven’t yet measured how well it abstains, which is its own metric worth tracking.
• Expand the gold sets. 25 and 30 items catch big regressions reliably but are noisy at the margins — a v2 prompt losing by 0.03 is suggestive, not conclusive, at this sample size.

If there’s one idea I want this project to carry, it’s that RAG is an empirical discipline. The architecture is just hypotheses until the eval harness turns them into decisions — and sometimes the best decision it hands you is to take something out.