Lab notes · September 25, 2026
GPU bench notes, September 2026
The raw numbers our articles cite. Read the method first; repeat it before you trust us.
Bench under test
A desktop board with two NVIDIA cards installed together: a GeForce RTX 3060 Ti 8 GB (PCIe x4 slot) and a GeForce RTX 2060 6 GB (PCIe x16 slot). Both cards run under a software power cap set with nvidia-smi -pl: 150 W on the 3060 Ti and 125 W on the 2060, the 2060's minimum. Roughly 3 GB of the 2060's memory is held by a speech-to-text model that is always resident, so Ollama had about 3 GB of the 2060 and all 8 GB of the 3060 Ti to work with. Ollama 0.34.2 in Docker, models from the Ollama library, Q4_K_M quantisation. Driver telemetry sampled with nvidia-smi --query-gpu during generation. Idle GPU draw is the driver's reported board power with no model loaded, not wall power.
Method
One request to /api/generate with stream:false and num_predict:320, a 40-word prompt, after one warm-up run of the same model. Tokens per second is Ollama's own eval_count / eval_duration. Prompt processing is prompt_eval_count / prompt_eval_duration. Power and temperature were sampled once mid-generation. Single runs, so treat ±5% as noise.
Results
| Model (Q4_K_M) | Weights on disk | Where it ran | Generation | Prompt | GPU power during run | GPU temp |
|---|---|---|---|---|---|---|
| qwen2.5:7b | 4.4 GB | RTX 3060 Ti, fully in VRAM (4.6 GB used) | 76.3 tok/s | 3,847 tok/s | 149.6 W (at the 150 W cap) | 51 °C |
| llama3.1:8b | 4.6 GB | RTX 3060 Ti, fully in VRAM (5.0 GB used) | 73.9 tok/s | 515 tok/s (cold prompt cache) | 149.9 W | 51 °C |
| qwen2.5:14b | 8.4 GB | Split: 6.7 GB on the 3060 Ti + 1.7 GB on the 2060; ~0.5 GB spilled to CPU | 24.9 tok/s | 1,293 tok/s | 117.7 W + 39.4 W | 49 °C / 43 °C |
Idle board power with no model loaded: RTX 3060 Ti 16.8 W, RTX 2060 6.6 W (driver-reported). Idle temperatures 40 °C and 37 °C in a case with poor airflow. Earlier in the week, with the 2060 idle and both cards available, the same 14B model ran fully in VRAM across the two cards at about 38 tok/s; the 25 tok/s above is what a partial CPU spill costs.
What these numbers say
- A 7B or 8B model at Q4 fits an 8 GB card with room to spare and runs at 70+ tok/s, far faster than anyone reads.
- A 14B model at Q4 needs about 9 GB in VRAM to stay fully on GPU. On 8 GB it spills and loses more than a third of its speed; on 6 GB it is not a comfortable fit.
- A 150 W cap costs nothing visible here: the 3060 Ti sat at the cap and still produced 76 tok/s.
Measured September 25, 2026. We re-run these when the hardware or Ollama version changes.