A local LLM lesson: GPU offload and MoE models

2026-10-11

Ollama is not installed on my machine right now. The binary, the package, and the service are gone, and I do not know why. This note describes the setup from 2026-09-15, when it ran on an RTX 3060 with 6 GB of VRAM.

The setting

num_gpu tells Ollama how many layers to put on the GPU. A value of 99 means all of them. You pass it per request in options.

Dense models: it helps

Ollama’s memory estimator under-filled my card by about 1 GB. It stopped near 4.7 GiB of 5.7 GiB and reported a spill for models that fit. With num_gpu=99, qwen2.5-coder:7b went from an apparent context ceiling of 10k to a real 64k. A model that does not fit returns a load error. So the override is safe to try on a small dense model.

MoE models: it hurts

gpt-oss:20b is a mixture-of-experts model. Without help, Ollama keeps attention and the KV cache on the GPU and leaves the routed experts in system RAM. The model ran in 4306 MiB of VRAM at about 27 tokens a second.

With num_gpu=99, Ollama tried to allocate 12.6 GB on a 6 GB card. It died with cudaMalloc failed: out of memory.

My rule: num_gpu=99 for small dense models only. Let Ollama place MoE models.

Two smaller facts

• Setting LLAMA_ARG_* environment variables did nothing on Ollama 0.34. It does not pass them to its llama-server child. So --n-cpu-moe was unreachable. It was also unneeded, because the expert offload is automatic.
• Thinking models return an empty string with done_reason: length when the num_predict cap is too low. I scored those as wrong answers and got three false results. Check for empty output before you score.

What is still open

I explain the MoE crash only as far as I measured it. I did not read the Ollama source, so I do not know why the forced offload ignores the expert split. I will check my notes before I reinstall.