Vector stores and completing the RAG loop

In the previous post ranked chunks by looping over an in-memory array. That is fine for a demo and hopeless for a real corpus; you can’t reload and re-embed a million chunks on every question. You need a vector store: a database that persists embeddings and answers nearest-neighbour queries fast. Then you complete RAG by feeding the retrieved text to the chat model.

Microsoft.Extensions.VectorData.Abstractions provides the storage abstraction, mirroring the design of IChatClient: one interface, many backends (in-memory, SQLite-vec, Azure AI Search, Qdrant, Postgres/pgvector, Cosmos DB…).

Model your data as a record

A vector store holds typed records. You describe one with attributes that tell any connector how to build its schema. From PdfChunkRecord.cs:

public class PdfChunkRecord
{
    [VectorStoreKey]
    public string Key { get; set; } = Guid.NewGuid().ToString();

    [VectorStoreData]
    public string Text { get; set; } = string.Empty;          // returned with a hit -> goes in the prompt

    [VectorStoreData(IsIndexed = true)]
    public string SourceFile { get; set; } = string.Empty;    // provenance, for citing / deleting

    [VectorStoreVector(dimensions: 768, DistanceFunction = DistanceFunction.CosineDistance)]
    public ReadOnlyMemory<float> Vector { get; set; }
}

Two attributes carry the weight:

• Dimensions must equal your embedding model’s output exactly. 768 for nomic-embed-text, 1536 for text-embedding-3-small, 3072 for text-embedding-3-large. A mismatch is rejected on upsert. This is the number article 5 told you to write down.
• DistanceFunction decides what Score means at query time — and getting it backwards is a genuinely nasty bug. See below.

The ingestion service

PdfIngestionService.cs depends only on abstractions — IEmbeddingGenerator and VectorStoreCollection<string, PdfChunkRecord> — so it is fully testable and swappable. Ingest is extract → chunk → embed (batched) → upsert:

await _vectorCollection.EnsureCollectionExistsAsync(token);

var chunks = ChunkText(ExtractTextFromPdf(pdfStream));
var embeddings = await _embeddingGenerator.GenerateAsync(chunks, cancellationToken: token);

var records = chunks.Select((chunk, i) => new PdfChunkRecord
{
    SourceFile = fileName,
    Text = chunk,
    Vector = embeddings[i].Vector
}).ToList();

await _vectorCollection.UpsertAsync(records, token);

Search — and the distance-direction trap

const float _maxDistance = 0.8f;

var queryEmbedding = await _embeddingGenerator.GenerateAsync(query, cancellationToken: token);

await foreach (var result in _vectorCollection.SearchAsync(queryEmbedding.Vector, top, cancellationToken: token))
{
    // Record declares CosineDistance, so Score is a DISTANCE:
    //   0 = identical, 2 = opposite. LOWER IS BETTER -> the comparison is `<`.
    if (result.Score < _maxDistance)
        matches.Add(result.Record);
}

Read that comment twice. Because the record uses CosineDistance, Score is a distance and smaller means more relevant, so the filter is Score < threshold. If you switch the record to CosineSimilarity, the meaning inverts — larger is better — and this comparison must flip to >. Get it backwards and your RAG confidently retrieves the least relevant chunk in the corpus while looking like it works. Whenever you set a threshold, confirm which direction your distance function runs.

null is a real answer

SearchAsync returns null when nothing clears the threshold. That is information, not an error — it tells the caller to ask the raw question instead of fabricating grounding:

public static string BuildPrompt(string question, string? context)
    => context is null
        ? question
        : $"""
           Use the following context from the ingested documents to answer the question.
           If the context does not contain the answer, say so rather than guessing.

           Context:
           {context}

           Question: {question}
           """;

The “say so rather than guessing” line matters: it curbs the model from answering out-of-corpus questions with confident nonsense.

Registration: in-memory vs SQLite

The collection is the unit of registration, not the store. From Program.cs:

builder.Services.AddSingleton<VectorStoreCollection<string, PdfChunkRecord>>(_ =>
{
    if (usePersistentStore)
        return new SqliteVectorStore($"Data Source={dbPath}")
            .GetCollection<string, PdfChunkRecord>("pdf-chunks");

    return new InMemoryVectorStore()
        .GetCollection<string, PdfChunkRecord>("pdf-chunks");
});

InMemoryVectorStore is perfect for tests and demos. SqliteVectorStore persists to a file so ingestion survives a restart. The PdfIngestionService does not change between them — that is the abstraction paying off again. On mobile you would use the in-memory store; on desktop, SQLite.

The complete loop

Retrieve, then generate, in one pass:

var context = await ingestion.SearchAsync(question);
var prompt  = PdfIngestionService.BuildPrompt(question, context);

await foreach (var update in chatClient.GetStreamingResponseAsync(prompt))
    Console.Write(update.Text);

Run it

ollama pull qwen3:1.7b
ollama pull nomic-embed-text
dotnet run --project 06.RagVectorStore            # in-memory
dotnet run --project 06.RagVectorStore -- --sqlite # persist to disk

Ingested 4 chunk(s).

you> Which interface do I use to stream chat responses?
     (retrieved 512 chars of context)
ai > Use IChatClient.GetStreamingResponseAsync, which returns an
     IAsyncEnumerable<ChatResponseUpdate>.

you> What is the capital of Portugal?
     (no relevant context found -- answering ungrounded)
ai > That isn't covered by the ingested documents, but the capital of Portugal
     is Lisbon.

The first answer is grounded in the ingested PDF; the third is flagged as out-of-corpus and answered honestly. That distinction — grounded vs ungrounded — is exactly what a production RAG system must get right.

A packaging note

The SQLite path pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11, which NuGet flags (NU1903) as vulnerable. 2.1.11 is the newest published version at the time of writing, so there is no upgrade yet; the in-memory default does not touch SQLite. Track the advisory and bump when a fix ships.

What you learned

• Microsoft.Extensions.VectorData abstracts the vector database the way IChatClient abstracts the model.
• A record’s Dimensions must match your embedding model; its DistanceFunction defines the meaning and direction of Score.
• null from search is a signal to answer ungrounded, and the prompt should admit when it can’t answer.
• Swapping in-memory for SQLite is a registration change, not a code change.

Next: Generating images with IImageGenerator — the same abstraction pattern, applied to pixels.