Ollama tokens per second vs response time: read the API timings

Output tokens per second describes generation speed, not the whole wait for a useful answer. Read Ollama’s loading, prompt and generation counters separately. If the question is what the reader sees, record timing at the client as well.

This is a research-based guide for a generic GPU test bench, using official documentation checked on October 19, 2026. No new cold-start, warm-start, prompt-cache or first-content measurements are recorded here. Every numeric example below is chosen arithmetic, not a real request or a performance promise.

Choose the timing question first

Decide what the comparison needs to establish before collecting a number:

• Output-generation rate: how many output tokens were generated per generation second?
• Server duration: what interval does the server report for that response?
• Client completion: how long did the client wait until it received the complete result?
• First useful content: when did the chosen client first receive or display content that meets your stated criterion?

Those are different labels. Preserve them even when the numbers happen to look similar. A comparison of the first interval cannot settle the last one.

Our NVIDIA power-limit guide reports generation-rate results under its recorded workload. Use this timing framework before carrying those results into a claim about an interactive application’s response time.

Read counts and durations with their units

The current generate API reference documents:

• total_duration: time spent generating the response, in nanoseconds.
• load_duration: model-loading time, in nanoseconds.
• prompt_eval_count: input-token count.
• prompt_eval_cached_count: prompt tokens read from cache.
• prompt_eval_duration: time evaluating uncached prompt tokens, in nanoseconds.
• eval_count: generated output-token count.
• eval_duration: output-generation time, in nanoseconds.

Keep the units and field names with the saved record. Check which fields the installed runtime actually returns. Missing is not zero, and a missing or zero duration is not a valid denominator.

The uncached-prompt definition matters: do not divide all input tokens by that duration and silently label the result a cache-neutral prompt-processing rate. Keep available cache counters and the request conditions beside it. Nor should you assume that the named durations form an exact, exhaustive sum of the server total.

Calculate generation rate without relabelling it

Convert nanoseconds to seconds, then divide output tokens by the output-generation duration:

seconds = duration_ns / 1,000,000,000
generation_tokens_per_second = eval_count / (eval_duration / 1,000,000,000)

These selected inputs illustrate why the denominator needs a label. Both examples generate the same output count in the same generation duration, but have different chosen loading and total durations.

Metric	Illustration A	Illustration B
Output tokens	100	100
Model loading (s)	2.00	0.10
Uncached prompt evaluation (s)	1.00	1.00
Output generation (s)	2.00	2.00
Server total (s)	5.20	3.30
Generation rate (tokens/s)	50.00	50.00
Output / server total (tokens/s)	19.23	30.30

The table is recomputed in code. Dividing output count by server total produces an output-per-server-second ratio, not isolated generation speed and not client elapsed time. The identical generation rates coexist with different totals in this illustration; no hardware speedup has been measured.

Do not identify an unexplained difference between total and component durations as network time. These fields alone do not establish that attribution.

Separate model residency from prompt caching

Model residency and prompt reuse need separate labels. A model already being loaded does not by itself establish which prompt tokens were cached. Look at the relevant returned fields rather than treating a repeated request as an otherwise unchanged trial.

The Ollama FAQ documents keep_alive as the model’s retention policy. A value of 0 requests unloading after the response, and the API parameter overrides the server’s OLLAMA_KEEP_ALIVE setting. This controls residency; it is not a guarantee about the next request’s response time or a command to clear every cache.

For a proposed comparison, distinguish a request after model unloading from one made while the model is resident. Record what was requested and what was observed. Preserve prompt-cache information independently instead of calling both conditions simply “cold” or “warm.”

Measure first useful content at the client

Ollama’s streaming documentation describes newline-delimited JSON responses and the stream:false option for a single JSON response. A complete-response record simplifies collecting aggregate fields; it does not preserve the arrival time of streamed content.

For a proposed client test, use a monotonic clock to record request submission, first useful content and completion. Define whether “first useful” means received content or displayed content, and whether intermediate thinking output counts. State that definition with the result.

Do not derive first-token latency from aggregate duration counters, or assume the first JSON line, first output token and first useful visible answer are interchangeable. Instrument the event you want to compare.

Keep a reproducible comparison record

Record the runtime version, model tag or digest, quantization, input, configured context, parallel settings and runtime placement. Include model-retention policy, available prompt-cache counters, output limits and thinking mode where applicable. The Ollama context and concurrency guide explains why those memory settings belong beside a workload result. For a worked example, use the 3060 Ti vs RTX 2060 comparison to keep resident and offloaded runs separate.

Repeat the proposed workload under stated conditions and preserve actual token counts and the spread of results. No universal run count or performance threshold is prescribed here. Keep server records and client timestamps separate, then report only the comparison those records support.