Building a Local LLM Benchmark Harness with TypeScript, Ollama & SQLite

Local LLMs are easier than ever to run, but they're surprisingly hard to benchmark fairly.

The first number most people look at is tokens per second. In practice, that number is influenced by everything from cold starts and output length to GPU thermals and generation settings. Two models can appear twice as fast simply because one talks twice as much.

I wanted a way to measure local models consistently on my own hardware—not just once, but repeatedly, with historical results I could compare over time. That led me to build a small, reusable benchmark harness around Ollama, TypeScript, and SQLite.

This article walks through how the harness evolved, the engineering decisions behind it, and what I learned after collecting more than 500 lines of raw benchmark data across multiple local LLMs.

Project

📦 Repository: https://github.com/Jaival-Suthar/inference-lab

🚀 Latest Release: https://github.com/Jaival-Suthar/inference-lab/releases/latest

All benchmarks in this article were collected on an NVIDIA GeForce RTX 5050 Laptop GPU (8 GB VRAM) using Ollama. Unless otherwise noted, every model was evaluated using identical prompts and benchmark settings.

Before looking at the benchmark results, it helps to understand how the harness is structured and why each component has a single responsibility.

Here's the shape of the harness:

Every benchmark session flows through the runner once, and each component records what it's responsible for. The runner doesn't know about SQLite, and SQLite doesn't know about GPUs. That separation is what let me add --max-tokens and GPU telemetry later without touching the storage layer.

The first version

The first version was intentionally simple: send a prompt, time the response, print tokens/sec. It answered "does this work" but not "can I trust this number."

Two problems showed up almost immediately:

• Cold starts lie. The first call to any model pays a one-time cost because it loads weights into VRAM, JIT warmup in the runtime, page cache misses. If you benchmark on run one, you're measuring disk I/O and model load time, not inference speed.

• A single run is a sample of one. Latency on a local GPU varies run to run because of thermal throttling, background processes, even how full the KV cache context is. One number tells you almost nothing about the real distribution.

So the harness grew warmup runs and multi-run benchmarking as first-class concepts, not afterthoughts.

Why warmup runs aren't stored

Every model gets one untimed warmup call before the benchmark runs start. That run is discarded entirely. It isn't averaged in, stored in SQLite, or touched by the stats layer.

This matters because a warmup run and a benchmark run are answering different questions. Warmup answers "is the model loaded and ready." Benchmark runs answer "how fast is this model once it's hot." Mixing them into the same average quietly inflates your latency numbers and makes every comparison across models unfair, since some models pay a bigger cold-start penalty than others (DeepSeek R1's warmup took 42s in my runs; Phi-4-mini's took under 8s. That gap alone would have swamped every other signal in the data).

Why benchmark summaries are computed in-memory

Each benchmark session computes its average, median, min, max, TTFT stats, and token counts in-memory immediately after the runs finish, rather than being derived later from whatever rows happen to be in SQLite at query time.

The reason is correctness under change. If I later add a filter, change how I define "outlier," or rerun stats after deleting a bad row, I don't want the summary for a specific session silently drifting. The session's summary is a fact about that session at the moment it ran. It gets computed once, stored once, and never recomputed implicitly. Historical comparisons across sessions are a separate, deliberate query, not a side effect of storage.

Why historical stats come from SQLite

Flat files or JSON logs would have worked for a single run. They fall apart the moment the actual question becomes "how does Qwen3 8B's TTFT this week compare to three weeks ago," or "give me every session where VRAM usage exceeded 5GB."

SQLite gives me that for free: indexed queries, joins across sessions and models, and aggregate functions, without standing up a database server for what is still a single-machine CLI tool. It's the smallest piece of infrastructure that makes the historical questions answerable instead of requiring me to grep through log files.

Why --max-tokens matters for fair comparisons

This was the fix that mattered most, so it's worth showing rather than just describing. I used the same prompt across every model, "explain KV cache," with no cap on generation length:

The tokens/sec column looks like it's telling you Qwen3 is the fastest model. It isn't. It's telling you Qwen3 generated 4x more tokens than Phi-4 Mini for the identical prompt. Latency and TTFT are really measuring output length here, not model speed.

await benchmark({
  warmup: 1,
  runs: 5,
  maxTokens: 512,
});

Capping generation to 512 tokens for every model turns that into an actual comparison:

Same output budget, same prompt, and the ranking changes completely. Gemma 3 4B, a 4B model, outran every 8B model on tokens/sec. Qwen3 8B and DeepSeek R1 8B, which looked 2-4x apart in the uncapped table, converge to within 3% of each other once the amount of work is held constant. That's the actual finding: throughput differences in the first table were mostly an artifact of verbosity, not architecture.

It's worth being honest about the tradeoff, though. For a reasoning model like DeepSeek R1, capping tokens doesn't just normalize output length. It can cut the reasoning trace off mid-thought, before the model reaches an answer. The benchmark becomes fair on throughput and unfair on task completion. That's not a flaw in the methodology so much as a reminder that "tokens/sec" and "did the model actually finish the job" are two different questions, and a speed benchmark can only honestly answer the first one.

Why TTFT is more important than raw throughput for user experience

Tokens/sec measures how fast text streams out once it starts. Time-to-first-token measures how long a person stares at a blank screen before anything happens. For a chat interface, TTFT is the number that actually predicts whether the product feels responsive. A model that streams at 150 tok/s but takes 20 seconds to start feels slower than one that streams at 55 tok/s but starts in under a second.

The capped-run data made this concrete. Before capping max tokens, Qwen3 8B's average TTFT was around 19.6s; DeepSeek R1's was around 19s, with a max over 47s. After capping, both dropped to under a second. That's too large a swing to be purely about smaller output. It suggests TTFT, as measured here, is more coupled to total generation than a strict "time to first streamed token" should be. I'm treating that as an open question rather than a settled result: worth instrumenting more precisely before I lean on the number in anything more consequential than this write-up. Benchmarking is only useful if the metrics mean what we think they mean.

Why GPU telemetry helps evaluate local models

Speed numbers alone don't tell you whether a model is a reasonable choice for a given machine. Two models can hit similar tokens/sec while one sits at 65°C and 60W and the other sits at 78°C and 95%+ utilization for the whole run. On a laptop GPU, that difference is the difference between "usable for long sessions" and "thermal throttling after ten minutes."

Capturing VRAM usage, utilization, temperature, and power draw per benchmark run turns "which model should I run locally" into a question you can actually answer with data specific to your hardware, rather than trusting a leaderboard benchmarked on a very different GPU.

Here's what that telemetry actually looked like, averaged across the capped 512-token runs on an RTX 5050 Laptop GPU:

Power draw is nearly flat across every model, at 60-73W regardless of size. That tells me this GPU isn't power-limited at these model sizes. It's thermally and VRAM-bound instead. The clearest signal is temperature: Phi-4 Mini and Gemma stay in the mid-60s°C, while both 8B models pushing near-100% utilization climb into the mid-70s°C. On a laptop chassis, that's the number that predicts whether a model is fine for a quick query versus something that'll throttle during a long session.

What the benchmarks revealed

• Gemma 3 4B punched well above its weight. At 512 tokens capped, it hit 83.4 tok/s, faster than every 8B model tested, while using barely more than half their VRAM. Parameter count alone didn't predict throughput here.

• Phi-4 Mini was the most hardware-efficient model in the set. Lowest VRAM (3.0 GB), lowest average temperature, competitive throughput. If the constraint is "fit comfortably on modest hardware," it's the clear pick from this data.

• Qwen3 8B and DeepSeek R1 8B converge once output length is controlled. They looked 2x apart on raw tokens/sec in the uncapped runs; capped, they're within a few percent of each other. The uncapped gap was mostly verbosity, not architecture.

• Reasoning models need a different benchmark methodology, not just a token cap. DeepSeek R1's uncapped runs show latency and TTFT numbers that are really measuring "how long the model thinks before it starts answering," which isn't the same quantity as TTFT for an instruct model. Comparing them on the same axis without noting that is misleading even when the numbers are accurate.

• Power draw stayed flat while temperature didn't. All five models sat in a narrow 60-73W band regardless of size, but temperature spread by nearly 10°C between the coolest and hottest. On this GPU, thermal headroom, not power, looks like the real constraint for sustained local inference.

What the harness actually became

Not a benchmark of five models, but a harness that can benchmark any model, on any machine, repeatably: warmup isolation, multi-run statistics, historical SQLite storage, comparison and export commands, Markdown reports, token-capped fair comparisons, and per-run GPU telemetry.

The single biggest lesson wasn't about any one model. It was that benchmark methodology matters more than any individual benchmark result. A number without a controlled comparison (same prompt, same output length, same warmup discipline) isn't wrong, exactly. It's just answering a different question than the one you think you're asking.

The benchmark harness answers how a model behaves locally. The next step is answering what useful systems can be built on top of that foundation. That starts with retrieval, where this inference runtime begins powering real AI applications rather than benchmark reports.