---
name: ollama-local-inference
description: Local LLM inference via Ollama on resource-constrained GPUs (4 GB VRAM class). Covers model selection by VRAM budget, Modelfile creation (num_ctx, num_gpu), KV-cache sizing, and the hard limits that force trade-offs between context length and speed.
version: 1.0.0
author: Hermes
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [ollama, local-llm, gguf, vram, low-vram, modelfile, kv-cache, num_ctx, quant]
---

# Ollama local inference on constrained GPUs

Use this skill when the user runs Ollama locally and is hitting performance problems (slow responses, OOM, hangs) — especially on 4 GB VRAM cards like GTX 1650 / GTX 970.

## When to use

- User reports "model hangs for minutes" / "no response" with Ollama
- User wants more context length (num_ctx) but model OOMs
- Need to pick a model that fits a given VRAM budget
- Need to create a custom Ollama model with specific parameters (Modelfile)
- Need to clean up unused models and free disk

## Hard limits you must know

### KV-cache scales with context

For a transformer, the KV-cache memory is roughly proportional to `num_ctx * num_layers * hidden_dim * 2 (K+V) * bytes_per_element`.

Practical rule for 3B-class models on 4 GB VRAM:
- 8K context  → ~250-500 MB KV cache → fits
- 16K context → ~500 MB-1 GB → marginal
- 32K context → ~1 GB+ → OOM risk on 4 GB cards
- 64K context → 2 GB+ → impossible on 4 GB

**If the user wants more than 8K context on a 4 GB card, the model + cache will not fit. The options are:** smaller model, smaller quant, or accept CPU offload (which kills speed).

### Model + cache must both fit

Total VRAM ≈ model file size + KV cache + activations + overhead (~200-400 MB).

Examples on a 4 GB card:
- 2 GB model (3B Q4) + 1 GB KV cache for 64K → **OOM** (3 GB + overhead > 4 GB)
- 2 GB model + 500 MB cache for 8K → fits with headroom
- 4.7 GB model (7B Q4) + any cache → marginal, model alone nearly fills the card

## OOM diagnosis

If Ollama returns:
```
llama-server reported out-of-memory during startup: alloc_tensor_range: failed to allocate Vulkan0 buffer of size ...
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
```
The KV cache (driven by num_ctx) is the culprit. Reduce num_ctx.

## Modelfile creation

A Modelfile lets you derive a tagged model with custom parameters from a base. Create it in a path that doesn't have spaces or weird characters — Ollama's `-f` flag is finicky.

Windows-safe pattern (write Modelfile to a simple path, then `cd` to that directory before running `ollama create`):

```bash
# 1. Write Modelfile to C:/Users/<user>/AppData/Local/Temp/Modelfile
# Content:
#   FROM <base-model>
#   PARAMETER num_ctx <value>
#   PARAMETER num_gpu 999

# 2. Create the model
cd "C:/Users/<user>/AppData/Local/Temp"
ollama create <new-tag> -f Modelfile
```

**Pitfall**: running `ollama create -f /path/to/Modelfile` with a Windows absolute path often fails with "no Modelfile found". The `cd` first trick reliably works. POSIX-style /c/... paths from MSYS bash may not resolve.

## Recommended model sizes for common VRAM budgets

| VRAM  | Max model       | Max context (3-4B) | Realistic config                          |
|-------|-----------------|--------------------|-------------------------------------------|
| 4 GB  | 3-4B Q4 (2 GB)  | 8K                 | llama3.2:3b @ 8K, qwen3:4b @ 8K          |
| 6 GB  | 7B Q4 (4-5 GB)  | 16K                | qwen2.5:7b @ 8-16K                        |
| 8 GB  | 7B Q4           | 32K                | qwen2.5:7b @ 16-32K                       |
| 12 GB | 13B Q4          | 32K                | qwen2.5:14b @ 16-32K                      |
| 24 GB | 30B+            | 64K+               | any modern model                           |

**For 64K+ context, you need 12+ GB VRAM** or accept CPU offload (10× speed penalty).

## Speed diagnosis

If a model is "installed and runs but takes 10 minutes per response":
1. Check `nvidia-smi` while a request is in flight — if GPU utilization is 0%, model is on CPU.
2. Common cause: model is larger than VRAM. Ollama silently falls back to CPU/RAM.
3. Fix: pick a smaller model. Don't try to compensate with smaller quant — the gap between 14B CPU and 7B GPU is larger than 7B Q4 vs 7B Q8.

## Model hygiene

`ollama list` shows all pulled models. Clean up unused ones to free disk:
```bash
ollama rm <tag>            # single model
# "Zugriff verrichtet" / "access denied" = file locked, stop the model first:
ollama stop <tag>
ollama rm <tag>
```

## When to give up and use cloud

For a 4 GB VRAM card, "fast + 64K context" is impossible locally. Be honest with the user upfront. The options are:
- Fast + 8K: 3-4B model in VRAM
- Slow + 64K: 14B model on CPU (10 min/response)
- Fast + Cloud: pay for API or use Ollama's cloud models (e.g. `minimax-m3:cloud`)

## CUDA toolchain mismatch on older GPUs (GTX 970, GTX 780, etc.)

A specific failure mode on Maxwell / Kepler / early Pascal cards (compute
capability < 7.0, anything before ~GTX 1080):

```
Error: 500 Internal Server Error: llama-server process has terminated:
exit status 0xc0000409: ... stack-based buffer overrun ...
CUDA error: the provided PTX was compiled with an unsupported toolchain.
```

Cause: Newer Ollama releases (0.32.x+) ship CUDA kernels built with a
toolchain that drops support for older compute capabilities. The model
file is fine — the runtime can't load the compiled kernel onto the GPU.

**Fix attempts (in order):**
1. **Ollama downgrade** — install an older Ollama version (e.g. 0.5.7 or
   0.3.12) that still includes the relevant SASS for your CC. The Windows
   installer from `https://github.com/ollama/ollama/releases/download/v0.5.7/OllamaSetup.exe`
   works for this. Run `/S` for silent install. After installing, restart
   the `ollama` process.
2. **Force CPU mode** — set `OLLAMA_NUM_GPU=0` in the user's environment
   BEFORE starting `ollama serve`. The env var must be set in the parent
   process that launches the Ollama service; setting it after-the-fact in
   your shell doesn't propagate to the already-running Ollama daemon. On
   Windows: stop the running ollama process first, then start it with the
   env var present in the parent shell.
3. **Try `OLLAMA_LLM_LIBRARY=cpu`** — alternative force-CPU env var some
   versions respect.
4. **Accept slowness + use a 3-4B model** — on CPU, anything 7B+ is
   unusable for chat. Stick to `qwen3:4b` or `llama3.2:3b`.

**Don't waste time on:**
- `winget` / `choco` / `scoop` for Ollama (none solve this — the bug is
  in the released runtime binary, not the installer)
- `ollama stop` / `ollama rm` (the installed models are fine, the runtime
  is broken)
- Setting `OLLAMA_HOST`, `OLLAMA_ORIGINS`, `OLLAMA_KEEP_ALIVE` (none
  affect kernel loading)

**Verify GPU compute capability** with
`nvidia-smi --query-gpu=compute_cap --format=csv` — if it's `5.x` or
lower, you have this problem on recent Ollama builds.

**Working around the limit when downgrade isn't an option:**

If you can't downgrade (e.g., the user has other models that need the
newer Ollama), proxy through OpenRouter (one free API key covers many
models) or Ollama's own cloud router. See `references/openrouter-fallback.md`
for a turnkey proxy that auto-falls-back to local Ollama when the cloud
rate-limit hits.

## Hermes config wiring

`~/.hermes/config.yaml`:
```yaml
provider: ollama
model: <tag-from-ollama-list>
ollama:
  host: http://localhost:11434
```

Restart Hermes after changing the model — config is read at startup, not hot-reloaded.

## Workflow

1. **Ask about VRAM** if not already known. `nvidia-smi --query-gpu=memory.total --format=csv`.
2. **Ask what model they have** with `ollama list`. Don't guess.
3. **Compute the fit**: model size vs. VRAM, then subtract room for KV cache.
4. **If model doesn't fit, recommend the smaller alternative** with the same family if possible (qwen3:4b not 14b, llama3.2:3b not 8b).
5. **Create Modelfile** with num_ctx sized to leave headroom (8K is safe default on 4 GB).
6. **Test** with a short prompt + num_predict=15. First-load latency is high; second request is the real number.
7. **Update** `~/.hermes/config.yaml` to point at the new tag.
8. **Clean up** the old, broken models with `ollama rm`.

## Output format

When recommending a model, give the user the trade-off in one line:
- "X fits, Y speed, Z context. 64K is impossible on your card — alternatives: ..."

Don't bury the limitation in a list of options. Lead with the constraint.