Bench notes, September 2026

Bench under test

A desktop board with two NVIDIA cards installed together: a GeForce RTX 3060 Ti 8 GB (PCIe x4 slot) and a GeForce RTX 2060 6 GB (PCIe x16 slot). Both cards run under a software power cap set with nvidia-smi -pl: 150 W on the 3060 Ti and 125 W on the 2060, the 2060's minimum. For these first runs roughly 3 GB of the 2060's memory was already taken by another workload, so Ollama had about 3 GB of the 2060 and all 8 GB of the 3060 Ti to work with. (The card-vs-card runs further down had both cards empty.) Ollama 0.34.2 in Docker, models from the Ollama library, Q4_K_M quantisation. Driver telemetry sampled with nvidia-smi --query-gpu during generation. Idle GPU draw is the driver's reported board power with no model loaded, not wall power.

Method

One request to /api/generate with stream:false and num_predict:320, a 40-word prompt, after one warm-up run of the same model. Tokens per second is Ollama's own eval_count / eval_duration. Prompt processing is prompt_eval_count / prompt_eval_duration. Power and temperature were sampled once mid-generation. Single runs, so treat ±5% as noise.

Results

Model (Q4_K_M)	Weights on disk	Where it ran	Generation	Prompt	GPU power during run	GPU temp
qwen2.5:7b	4.4 GB	RTX 3060 Ti, fully in VRAM (4.6 GB used)	76.3 tok/s	3,847 tok/s	149.6 W (at the 150 W cap)	51 °C
llama3.1:8b	4.6 GB	RTX 3060 Ti, fully in VRAM (5.0 GB used)	73.9 tok/s	515 tok/s (cold prompt cache)	149.9 W	51 °C
qwen2.5:14b	8.4 GB	Split: 6.7 GB on the 3060 Ti + 1.7 GB on the 2060; ~0.5 GB spilled to CPU	24.9 tok/s	1,293 tok/s	117.7 W + 39.4 W	49 °C / 43 °C

Idle board power with no model loaded: RTX 3060 Ti 16.8 W, RTX 2060 6.6 W (driver-reported). Idle temperatures 40 °C and 37 °C in a case with poor airflow. Earlier in the week, with the 2060 idle and both cards available, the same 14B model ran fully in VRAM across the two cards at about 38 tok/s; the 25 tok/s above is what a partial CPU spill costs.

Power-limit sweep: RTX 3060 Ti, qwen2.5:7b

Same card, same model, five software power caps set with nvidia-smi -i <GPU> -pl <W>. At each cap: three generation runs of 512 tokens (seed 1, temperature 0) after one warm-up; driver telemetry sampled every 250 ms during each run. Tokens per second is the mean of the three runs; draw is the mean of samples above 40 W across runs two and three (run one includes ramp-up); temperature is the peak across all three runs. The runs went from lowest to highest cap back to back, so the later steps started warmer. The card went back to its 150 W cap afterwards.

Cap	Generation	Avg draw under load	Tokens per watt	Avg core clock	Peak temp	vs stock 200 W
100 W (card minimum)	58.8 tok/s	99.6 W	0.59	1,092 MHz	50 °C	−24% speed, −49% power
125 W	72.3 tok/s	122.5 W	0.59	1,516 MHz	53 °C	−7% speed, −37% power
150 W	75.5 tok/s	145.4 W	0.52	1,675 MHz	57 °C	−3% speed, −25% power
175 W	76.9 tok/s	172.0 W	0.45	1,789 MHz	61 °C	−1% speed, −11% power
200 W (stock)	77.8 tok/s	193.4 W	0.40	1,855 MHz	66 °C	baseline

For interpreting driver clock-event flags, see the SW Power Cap diagnostic guide. Its counter-duration example is hypothetical and separate from the measured sweep here.

Memory clock stayed at 6,801 MHz at every cap. The cap only pulled the core clock down, and token generation on a 7B model is mostly limited by memory bandwidth, which is why the first 75 W of cuts cost so little speed.

Card vs card: RTX 3060 Ti and RTX 2060, each on its own

Measured September 25, 2026, late evening. Each card got its own Ollama 0.34 instance pinned to that card alone, with the same settings as our everyday setup (flash attention on, q8_0 KV cache), and nothing else loaded on either GPU. Same prompt and method as the sweep above: three 512-token runs per row (seed 1, temperature 0) after a warm-up; speed is the mean of the three runs, draw is the mean of runs two and three, temperature is the peak. "On GPU" is Ollama's own report of how much of the model sat in the card's memory; the rest ran on the CPU. The RTX 2060's software power range is 125–170 W (170 W stock); the 3060 Ti's is 100–220 W (200 W stock).

Model (Q4_K_M)	Card · cap	Generation	On GPU	GPU draw	Tokens per watt	Peak temp
qwen2.5:7b	RTX 3060 Ti · 150 W	75.8 tok/s	100%	144.2 W	0.53	56 °C
	RTX 3060 Ti · 200 W	77.9 tok/s	100%	191.9 W	0.41	62 °C
	RTX 2060 · 125 W	57.7 tok/s	100%	121.5 W	0.47	64 °C
	RTX 2060 · 170 W	59.1 tok/s	100%	165.6 W	0.36	80 °C
llama3.1:8b	RTX 3060 Ti · 150 W	73.6 tok/s	100%	148.0 W	0.50	51 °C
	RTX 3060 Ti · 200 W	75.3 tok/s	100%	191.4 W	0.39	59 °C
	RTX 2060 · 125 W	44.8 tok/s	86%	54.7 W	n/a (CPU shares the work)	54 °C
	RTX 2060 · 170 W	45.0 tok/s	86%	51.1 W	n/a	54 °C
qwen2.5:14b	RTX 3060 Ti · 150 W	14.5 tok/s	71%	68.7 W	n/a	48 °C
	RTX 3060 Ti · 200 W	14.5 tok/s	71%	68.8 W	n/a	47 °C
	RTX 2060 · 125 W	8.6 tok/s	48%	39.5 W	n/a	52 °C
	RTX 2060 · 170 W	8.6 tok/s	48%	40.6 W	n/a	54 °C

Idle board power with no model loaded, same night: RTX 3060 Ti 16.5 W, RTX 2060 6.9 W. Tokens per watt is only shown where the whole model sat on the card; once part of it runs on the CPU, GPU watts no longer describe the work.

These generation results describe language models, not speech synthesis. The ElevenLabs versus Kokoro and Piper guide separates documented speech-output costs and requirements from latency measurements that would need a separate test.

Memory, firmware and unattended boot (the same bench, late September 2026)

Read from the machine itself with dmidecode, ethtool, nmcli, systemd-analyze and the kernel log on September 26, 2026. Nothing here is a vendor number.

What	Reading
Board / firmware	Gigabyte B450M DS3H V2, firmware F62b (dated 2021-09-06)
CPU	AMD Ryzen 5 5600G (12 threads)
Memory installed	Four DIMMs, two different kits: 2 × 8 GB Corsair CMK16GX4M2B3200C16 (single-rank) + 2 × 16 GB Samsung-labelled 16GXMP3200CL16 (dual-rank) = 48 GB
Memory speed	All four reported at 3200 MT/s configured speed (XMP enabled in firmware); firmware reports 1.2 V configured voltage
Stability	3 days 13 hours continuous uptime under GPU and LLM load with zero machine-check or memory-error entries in the kernel log (only the EDAC/MCE driver start-up lines)
Boot to fully up	24.6 s total: firmware 8.2 s, loader 1.0 s, kernel 2.7 s, initrd 2.1 s, userspace 10.6 s
Wake-on-LAN	NIC supports p/u/m/b/g; set to `g` (magic packet) and persisted with NetworkManager `wake-on-lan magic`
Power loss	Firmware set to power the machine back on when AC power returns (the board has no front-panel button connected)
Self-recovery	systemd hardware watchdog RuntimeWatchdogSec = 1 min; `kernel.panic = 10` (reboot 10 s after a panic); sleep and suspend targets masked

Boot recovery does not establish a secure remote-access path. For a generic bench, the Tailscale versus Cloudflare Tunnel guide explains access models, authentication and provider visibility separately from the recovery settings above.

What these numbers say

A 7B or 8B model at Q4 fits an 8 GB card with room to spare and runs at 70+ tok/s, far faster than anyone reads.
A 14B model at Q4 needs about 9 GB in VRAM to stay fully on GPU. On 8 GB it spills and loses more than a third of its speed; on 6 GB it is not a comfortable fit.
Capping the 3060 Ti from 200 W to 150 W cost 3% of generation speed and cut draw under load by a quarter. Below 125 W the speed drops off quickly.
On its own, the RTX 2060 runs a 7B model at about three quarters of the 3060 Ti's speed, but an 8B model no longer fits entirely in its 6 GB and drops to 45 tok/s. A 14B model is slow on either card alone; it needs both cards together (about 38 tok/s) or a 12 GB-plus card.

Measured September 25, 2026. We re-run these when the hardware or Ollama version changes.

GPU bench notes, September 2026