RAG Explained for .NET Developers: Build a Document Q&A System in C#

Ask an LLM about your data — your product docs, your internal policies, last month's incident reports — and you'll get a confident answer invented from thin air. The model has never seen your data. Retrieval-Augmented Generation (RAG) is the standard fix, and despite the intimidating name, the core of it is something every .NET developer already understands: search, then string interpolation.

Let's demystify it by building a document Q&A system in C#.

The idea in one paragraph

Instead of asking the model a question cold, you first retrieve the most relevant snippets from your own documents, then paste them into the prompt: "Using only the following context, answer the question." The model stops being an oracle and becomes what it's actually good at — a reading-comprehension engine over text you supplied. Answers become grounded, current, and auditable ("which snippets produced this answer?").

The pipeline: ingest → chunk → embed → store; then per question: embed → search → prompt.

Step 1: Chunking your documents

Models have context limits and retrieval works best on focused passages, so documents get split into chunks. The simplest workable strategy — split by paragraphs, cap the size, overlap slightly so sentences at boundaries aren't orphaned:

csharp
public static List<string> ChunkText(string text, int maxChars = 1500, int overlap = 200)
{
    var chunks = new List<string>();
    var paragraphs = text.Split("\n\n", StringSplitOptions.RemoveEmptyEntries);
    var current = new StringBuilder();

    foreach (var para in paragraphs)
    {
        if (current.Length + para.Length > maxChars && current.Length > 0)
        {
            chunks.Add(current.ToString());
            var tail = current.ToString();
            current.Clear();
            current.Append(tail[Math.Max(0, tail.Length - overlap)..]);
        }
        current.AppendLine(para);
    }

    if (current.Length > 0) chunks.Add(current.ToString());
    return chunks;
}

Chunking quality caps your whole system's quality — if the answer to a question is split across two chunks that never get retrieved together, no model can save you. Start with paragraph-based chunks around 1,000–2,000 characters and tune with real questions.

Step 2: Embeddings — text as coordinates

An embedding model turns text into a vector (an array of floats) where semantic similarity becomes geometric closeness. "How do I get a refund?" and "returns policy" land near each other despite sharing no words — which is exactly why this beats keyword search for Q&A.

csharp
using Microsoft.Extensions.AI;

IEmbeddingGenerator<string, Embedding<float>> embedder =
    new OpenAIClient(apiKey)
        .GetEmbeddingClient("text-embedding-3-small")
        .AsIEmbeddingGenerator();

var chunkEmbeddings = await embedder.GenerateAsync(chunks);

Step 3: Store and search

Production systems use a vector database (Azure AI Search, Qdrant, pgvector on the Postgres you already run). But for learning — and honestly, for many internal tools with a few thousand chunks — in-memory cosine similarity is completely fine:

csharp
public record DocChunk(string Text, string Source, float[] Vector);

public static float CosineSimilarity(float[] a, float[] b)
{
    float dot = 0, magA = 0, magB = 0;
    for (int i = 0; i < a.Length; i++)
    {
        dot += a[i] * b[i];
        magA += a[i] * a[i];
        magB += b[i] * b[i];
    }
    return dot / (MathF.Sqrt(magA) * MathF.Sqrt(magB));
}

public static List<DocChunk> Search(
    float[] queryVector, List<DocChunk> chunks, int top = 4)
    => chunks
        .OrderByDescending(c => CosineSimilarity(queryVector, c.Vector))
        .Take(top)
        .ToList();

No magic — it's OrderByDescending over a similarity score. When your corpus or query volume outgrows a linear scan, swap this method's internals for a vector database call; the shape of the system doesn't change.

Step 4: The grounded prompt

csharp
var queryEmbedding = (await embedder.GenerateAsync([question]))[0].Vector.ToArray();
var relevant = Search(queryEmbedding, allChunks);

var context = string.Join("\n---\n",
    relevant.Select(c => $"[Source: {c.Source}]\n{c.Text}"));

var prompt = $"""
    Answer the question using ONLY the context below.
    If the context doesn't contain the answer, say
    "I couldn't find that in the documentation" — do not guess.
    Cite the source of the information you use.

    Context:
    {context}

    Question: {question}
    """;

ChatResponse answer = await chatClient.GetResponseAsync(prompt);

The two instructions in that prompt are your hallucination defenses: only the context and say when you don't know. Including sources lets your UI show citations — which is what turns "the AI said so" into an answer people can trust and verify.

Why RAG answers go wrong (and where to look)

When quality disappoints, resist the urge to blame the model — in my experience the failure is almost always in retrieval:

  • The right chunk wasn't retrieved. Test retrieval separately: for 20 real questions, look at the top-4 chunks yourself. If the answer isn't in them, fix chunking, retrieve more chunks, or add keyword search alongside vector search (hybrid search) — before touching anything else.
  • The chunk was retrieved but truncated or context-free. Revisit chunk boundaries and overlap; consider prepending each chunk with its document title and section heading during ingestion (cheap and surprisingly effective).
  • The question needs synthesis across many documents. ("Summarize all incidents this quarter") — plain top-k RAG isn't built for that; you need different pipelines (summarization over the full set).

Also budget for ingestion being a real pipeline: documents change, so you need re-indexing on update — a background job that detects changed sources and re-embeds them, not a one-off console run. That part is ordinary .NET engineering, which is good news: it's the part you're already great at.

The takeaway

RAG isn't an AI research project — it's a search feature with an LLM at the end. Chunk sensibly, embed, search, and write a prompt that forbids guessing. Every piece is inspectable, testable C#, and the "AI magic" turns out to be one API call at the very end of an ordinary data pipeline.

Where to go next