Running Local LLMs with Ollama and C#: A Step-by-Step Guide

There are three reasons .NET developers keep asking me about local LLMs: cost (no per-token bill during development), privacy (data that can't leave the building — a hard requirement in finance and healthcare), and learning (nothing demystifies AI like running the model on your own machine). Ollama makes all three accessible: it's the Docker of local models — pull, run, call an API.

Here's the full path from zero to calling a local model from C#.

Step 1: Install Ollama and pull a model

Install from ollama.com (Windows, macOS, Linux — on Windows it runs as a background service). Then:

bash
ollama pull llama3.1        # Meta's 8B general model, ~4.7 GB
ollama run llama3.1         # chat with it in the terminal right now

That's a full LLM running on your machine, no account, no API key, no internet after the download.

Model picking, briefly: the constraint is RAM/VRAM. An 8B model like llama3.1 runs comfortably on a 16 GB machine and handles chat, summarization, and extraction well. phi4-mini and gemma3:4b are strong small options for modest hardware; qwen2.5-coder is a solid local coding model. Bigger models are better but slower — start small, upgrade when quality demands it.

Step 2: Call it from C#

Ollama serves an HTTP API on localhost:11434. The cleanest .NET client is OllamaSharp, which also implements the standard IChatClient abstraction:

bash
dotnet add package OllamaSharp
csharp
using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(
    new Uri("http://localhost:11434"), "llama3.1");

ChatResponse response = await client.GetResponseAsync(
[
    new ChatMessage(ChatRole.System, "You are a concise assistant."),
    new ChatMessage(ChatRole.User, "Explain async/await in two sentences."),
]);

Console.WriteLine(response.Text);

Streaming works exactly as with hosted providers:

csharp
await foreach (ChatResponseUpdate update in
    client.GetStreamingResponseAsync("Write a haiku about C#"))
{
    Console.Write(update.Text);
}

Because this is IChatClient, everything you've built against Microsoft.Extensions.AI runs unchanged against the local model. That's the workflow I recommend to every team: local model in development (fast iteration, zero cost, works offline), hosted model in production, switched by configuration:

code
IChatClient client = config["AI:Provider"] switch
{
    "ollama" => new OllamaApiClient(new Uri(config["AI:OllamaUrl"]!), config["AI:Model"]!),
    _ => new OpenAIClient(config["AI:ApiKey"]!)
            .GetChatClient(config["AI:Model"]!).AsIChatClient(),
};

Alternative: the OpenAI-compatible endpoint

Ollama also exposes an OpenAI-compatible API at http://localhost:11434/v1, so the official OpenAI SDK works with a redirected endpoint — handy for code you can't refactor onto IChatClient:

csharp
using OpenAI;
using OpenAI.Chat;

var client = new ChatClient(
    "llama3.1",
    new ApiKeyCredential("ollama"), // any non-empty string
    new OpenAIClientOptions { Endpoint = new Uri("http://localhost:11434/v1") });

Local embeddings too

The other half of the AI stack — embeddings for semantic search and RAG — also runs locally, and small embedding models are fast on CPU:

bash
ollama pull nomic-embed-text
csharp
IEmbeddingGenerator<string, Embedding<float>> embedder =
    new OllamaApiClient(new Uri("http://localhost:11434"), "nomic-embed-text");

var vectors = await embedder.GenerateAsync(
    ["reset password", "billing question"]);

A fully local RAG pipeline — your documents never leave the machine — is just this embedder plus the search-and-prompt pattern, with the chat model swapped to llama3.1. For sensitive internal documents, this is often the difference between "the compliance team said no" and "shipped."

Setting expectations honestly

A local 8B model is not GPT-4-class, and pretending otherwise leads to disappointed stakeholders. From experience:

  • Great locally: summarization, classification, extraction with clear instructions, dev/test of AI features, semantic search (embeddings especially — near-parity with hosted for many uses).
  • Noticeably weaker: complex multi-step reasoning, subtle instruction-following, long-context tasks, tool-calling reliability (it works, but expect more malformed calls — validate hard).
  • Speed depends brutally on hardware. With a decent GPU, an 8B model streams comfortably; CPU-only is usable for background jobs but sluggish for interactive chat.

The pattern that works: prototype and develop locally, measure on your actual task, and promote to a hosted model only where the quality gap matters. Often it doesn't — a well-prompted local model classifying support tickets can be indistinguishable from the expensive option, at zero marginal cost.

Production notes, if you self-host

Running Ollama beyond your laptop (an internal server, a container with GPU passthrough) is legitimate architecture for privacy-sensitive workloads:

  • Set OLLAMA_HOST=0.0.0.0 to listen beyond localhost — and put auth in front of it (a reverse proxy with API keys); Ollama itself has none.
  • Configure keep-alive (OLLAMA_KEEP_ALIVE) so the model stays loaded between requests; cold loads take seconds.
  • Watch concurrency: one GPU model serves requests sequentially by default. For multi-user load you'll want OLLAMA_NUM_PARALLEL tuning or a queue in front.

The takeaway

Local LLMs turn AI development from a metered API into an unlimited sandbox — and with IChatClient as the seam, choosing local vs hosted becomes a config value instead of an architecture decision. Pull a model tonight; the C# integration will take you fifteen minutes.

Where to go next