Debugging RAG Pipelines: Embeddings, ChromaDB, Qdrant, and Retrieval Evidence

A Bad RAG Answer Has Several Possible Owners

When a RAG answer is wrong, changing the prompt first is usually guesswork. The failure may be missing source data, destructive chunking, a different embedding model at query time, a vector collection with the wrong dimension, weak retrieval, lost metadata, or generation that ignores good context.

Debug the pipeline as five contracts: ingest, embed, store, retrieve, and generate.

---

1. Prove Ingestion

Record a stable document ID, source URI, checksum, chunk index, and ingestion version. Then answer basic questions:

• How many documents and chunks should exist?
• Can a known sentence be found in the stored text?
• Were deleted documents removed from the collection?
• Did a parser turn a table, heading, or code block into noise?

Chunking should preserve meaning, not merely hit a character count. Keep enough overlap for references that cross boundaries, but avoid duplicating so much text that near-identical chunks crowd out alternatives.

2. Treat the Embedding Model as a Schema

An embedding vector has a dimension and a semantic model identity. Ingesting with one model and querying with another can fail loudly through a dimension mismatch or quietly through meaningless similarity.

Store metadata such as:

{
  "embedding_model": "all-MiniLM-L6-v2",
  "dimension": 384,
  "normalized": true,
  "chunker_version": "policy-v3"
}

Validate it before every write and query. A collection is not a bag that accepts any vector; it is a versioned retrieval index.

3. Inspect Retrieval Before Generation

Run a fixed evaluation set directly against ChromaDB or Qdrant local mode. Capture returned IDs, distances or scores, metadata filters, and source text. Test exact facts, paraphrases, ambiguous queries, and a question that should retrieve nothing.

If the correct chunk is absent from the top results, the language model cannot repair retrieval reliably. Tune chunking, filters, embedding choice, and top-k at this layer. If the chunk is present but the answer contradicts it, inspect prompt assembly and faithfulness.

4. Make Generation Cite Its Evidence

Give context clear boundaries and instruct the model to say when the supplied evidence is insufficient. Return source IDs alongside the answer. Then score retrieval relevance separately from answer faithfulness. One combined score hides which component regressed.

A release gate can require known-answer retrieval recall, a minimum faithfulness score, and zero citations to nonexistent source IDs. Keep a small human-reviewed set because automated judges can share the same biases as the system they judge.

Use Repair the RAG Dimension Contract and Build a Faithfulness Gate to practice this evidence-first workflow.