OllamaLab — Local LLM hardware and inference speed.

Start here: sizing and speed for local models

OllamaLab covers the hardware side of running local models with Ollama: how much GPU memory a model needs, how fast a card can read those weights back on every generated token, and what to change when a model runs slower than it should. Two numbers decide almost everything - the VRAM a model and its quantisation occupy, and the memory bandwidth that sets generation speed once it fits. Every guide below works outward from those two.

Work out what fits

• How much VRAM you need to run local LLMs with Ollama - the sizing method: weights, KV cache, and context length, and which model fits 8, 12, 16, or 24 GB.
• The VRAM calculator - enter a model size, quantisation level, and context window and get the memory total plus an estimated generation rate for a given card.
• Which models fit in 8 GB of VRAM - the most common card tier, turned into concrete parameter classes and the context each one holds.
• The best GPU for Ollama by tier - capacity and memory bandwidth compared side by side, from 8 GB up to 48 GB across two cards.

Make it fast, and fix it when it is slow

• Ollama tokens per second: what sets your speed - why generation is bound by memory bandwidth, and how to read your own rate with --verbose.
• Why Ollama is not using your GPU - how to tell whether the GPU is being used at all, and the diagnostic order from detection to capacity to context length.

Specific setups

• Ollama on Apple Silicon - how unified memory changes the capacity-and-bandwidth trade, tabulated per chip.
• Ollama without a GPU - what CPU-only inference can and cannot do, and why memory-channel count sets the ceiling.
• Running Ollama in Docker - the container setup, including the GPU passthrough that keeps inference off the CPU.