Local LLM Deployment with Python: Ollama, VRAM, and a Real Endpoint

Read the tag before you pull

The suffix on a model name carries the quantization. q4_K_M means 4-bit weights with K-grouped scales on the sensitive tensors: about 4.5 bits per parameter in practice. Fewer bits means less VRAM and a little less accuracy, so start at Q4_K_M and move up only when your own evaluation asks for it.

params_b = 7
bits_per_weight = 4.5          # Q4_K_M, not exactly 4
weights_gb = params_b * 1e9 * bits_per_weight / 8 / 1024**3
print(round(weights_gb, 2))

Weights first, opinions later

Budget the KV cache as well

Weights are only half the bill. Every token in the context keeps a key and a value vector in every layer, and at 8k context that is often another 2 to 4 GB. When weights plus cache do not fit, the model spills to CPU and throughput collapses by an order of magnitude.

def kv_cache_gb(layers, hidden, ctx, bytes_per=2):
    # two tensors, K and V, fp16 by default
    return 2 * layers * hidden * ctx * bytes_per / 1024**3

print(round(kv_cache_gb(32, 4096, 8192), 2))

Context is not free

Serve it, then call it like any API

ollama serve exposes an HTTP surface on port 11434. A blocking POST to /api/generate with stream false returns one JSON object. Set a timeout longer than the slowest generation you expect; the default of a few seconds cuts real answers in half.

import requests

payload = {
    "model": "qwen2.5:7b-instruct-q4_K_M",
    "prompt": "Say hi in five words.",
    "stream": False,
}
resp = requests.post("http://127.0.0.1:11434/api/generate", json=payload, timeout=120)
resp.raise_for_status()
print(resp.json()["response"])

One endpoint, one timeout

Stream when a human is waiting

With stream true the body is newline-delimited JSON: each line carries a fragment in response, and done flips true on the last one. Parse line by line, skip empty lines, print as you go. The user sees words in under a second instead of staring at a spinner.

import json, requests

with requests.post(url, json=payload, stream=True, timeout=120) as resp:
    for line in resp.iter_lines():
        if not line:
            continue
        chunk = json.loads(line)
        print(chunk.get("response", ""), end="", flush=True)
        if chunk.get("done"):
            break

NDJSON, one fragment per line

Degrade to a smaller model, not to a 500

Out-of-memory kills and daemon 500s are normal on a laptop. Keep an ordered list of models and fall through it, then report which one answered so the logs explain the quality drop later. A slower answer beats a stack trace.

MODELS = ["qwen2.5:7b-instruct-q4_K_M", "qwen2.5:3b-instruct-q4_K_M"]

def generate(prompt):
    for name in MODELS:
        try:
            return name, call(name, prompt)
        except (TimeoutError, RuntimeError):
            continue
    raise RuntimeError("all local models failed")

A fallback chain you can log

Watch latency, throughput, and failures

First-token latency is what users feel; tokens per second is what the hardware delivers. Track both plus the failure rate per model, and alarm on the median rather than the mean, because one 30-second outlier should not page anyone at night.

import statistics

first_token_ms = [820, 910, 780, 2400, 860]
print(round(statistics.median(first_token_ms)), max(first_token_ms))

Medians, not means

Your practice checklist

• Model tag and quantization are written down before pulling.
• Weights plus KV cache fit in VRAM with headroom.
• The endpoint has an explicit timeout longer than the slowest answer.
• Streaming output is parsed line by line, tolerating empty lines.
• A smaller model answers when the big one runs out of memory.
• First-token latency, tokens per second, and failure rate are logged.

Common questions

Keep reading:

• Python model monitoring and drift checks
• Python Model Evaluation Exercises: Beyond Accuracy
• Python tokens and text vectors
• Python Prompt Validation Exercises: JSON, Schemas, Safety