Ingest and Query

A safe ingestion loop

Ingestion is the overall job of getting source material into the vector store in the first place, and it's worth walking through as a repeatable sequence rather than a one-off script: read a source document, normalize it into a consistent form (strip odd whitespace, fix encoding issues), split it into chunks (the next module covers chunking strategy in depth), compute an embedding for each chunk using the model from the previous module, and then upsert each chunk's ID, vector, and metadata. "Upsert" is a useful word worth knowing - it means update the record if that ID already exists, or insert a new one if it doesn't, so re-running ingestion never creates duplicates as long as the IDs stay stable. It's also worth recording the source's checksum and the exact embedding-model contract used, so a later audit can confirm nothing silently drifted.

for chunk in chunks:
    vector = embed_model.encode(chunk.text)
    collection.upsert(
        ids=[chunk.id], embeddings=[vector],
        documents=[chunk.text], metadatas=[chunk.metadata],
    )

Querying mirrors ingestion: embed the incoming question using that same embedding model (never a different one - see the previous module's warning), then ask the store for the top-k nearest matches. Once results come back, don't just trust their order blindly - inspect the returned distance score, the actual document text, and the metadata for each result. Ordering alone isn't proof that a top result is genuinely relevant to the question; it only means it was the closest of whatever was available.

Scenario: A deploy script reruns ingestion after a routine restart and, because it used add() with a freshly generated random UUID for every chunk instead of a stable ID, ends up silently doubling every single document in the collection. Switching that call to upsert() with stable, content-derived IDs turns the exact same operation into a safe one that can be rerun any number of times with no side effects.