hearth-bench

“Computer, how do I build a voice assistant that doesn't send my data to the cloud?”

We asked a computer on a desk, out loud. This page is what came back, how long each part took, and what it ran on.

The answers

Each model got the question and the same instructions, with a fresh context. Shown whole, unedited.

Qwen3.6 35B-A3B · 4-bit

You can build a local voice assistant by using open-source software like Rhasspy or Mozilla DeepSpeech. These tools run entirely on your computer or a small device like a Raspberry Pi, processing audio directly without sending it to the internet. This keeps your voice data private and secure within your own home network.

124 ms to first word0.75 s in all83.0 tokens/s

Gemma 4 26B-A4B · 6-bit

To build a private assistant, you should look into running large language models locally on your own hardware. You can use open source tools like Ollama or LocalAI to handle the processing without an internet connection. You will also need a local speech to text engine to process your voice commands.

160 ms to first word1.16 s in all50.2 tokens/s

Qwen3.8 27B · 6-bit

You can run open source speech recognition and text to speech models locally on your own machine. Tools like Whisper for speech to text and Piper for text to speech work offline. Pair them with a local large language model like Llama or Mistral for the assistant brain. This keeps all your audio and text on your hardware.

187 ms to first word3.29 s in all19.8 tokens/s

Phi-4 mini · 4-bit

To build a voice assistant that doesn't send your data to the cloud, you can use local processing. This means running the assistant on your device, like a smartphone or a computer, without sending data to external servers. You'll need to develop or use existing software that can process voice commands locally. Keep in mind that this approach may limit some features and may not be as secure as cloud-based solutions. Always ensure you're following best practices for data privacy and security.

247 ms to first word6.33 s in all14.5 tokens/s

The instructions every model got

You are a voice assistant. Everything you write is read aloud by a text-to-speech engine to a person who is listening, not reading. Keep replies short and conversational: two to four sentences, in plain spoken English. No lists, headings, markdown, emoji or code. Spell out anything a voice cannot say. If the person wants more detail, they will ask.

The pictures

One prompt, one seed, every image model. The time is for one image from a model that is already loaded.

Z-Image

48.5s per image

1024 × 1024 pixels · model already loaded · first request took 63.2 s

Z-Image: its picture for the prompt below

FLUX.2 Klein 9B

5.4s per image

1024 × 1024 pixels · model already loaded · first request took 28.7 s

Image not shown: the model's licence terms do not allow it here.

OmniGen2

53.1s per image

1024 × 1024 pixels · model already loaded · first request took 92.3 s

The prompt

Tech-junkie poster, metal industrial animated style. Sharp, almost photo-realistic textures. A griffin, all gray, with head, beak and wings of an eagle with the powerful legs and tail of a lion. The wings extend out from his sides, showing flowing dark rows of feathers in 3 shades of gray. His eyes are glowing grey with cyan light. He is wearing a saddle, small compared to his size, held on with tactical utility straps with glowing cyan neon accents. He is sitting on a rain-slicked rooftop ledge, resting back on his hind legs, looking directly at the viewer with a challenging, questioning gaze. Gritty neo-Tokyo background, soft bokeh of neon lights, heavy rain, moody dramatic rim lighting, volumetric fog. No title. Only visible text is on street signs.

Hear it

Qwen3.8 27B's answer, read by Kokoro in 6 voices.

You can run open source speech recognition and text to speech models locally on your own machine. Tools like Whisper for speech to text and Piper for text to speech work offline. Pair them with a local large language model like Llama or Mistral for the assistant brain. This keeps all your audio and text on your hardware.

Heart · Nova · Emma · Michael · Daniel · Onyx

Locally Running Services

Everything on this tab came from these, running on the machine itself with no call to an outside API. Each one is open source: follow a link and run it.

LLM

OpenAI API via llama-swap ↗

llama.cpp servers behind a model-swapping proxy. A 35B-A3B model at 4-bit answers by default.

OpenAI API via OpenVINO Model Server ↗

Phi-4 mini at 4-bit.

TTS

Kokoro-FastAPI ↗

Kokoro, 82M parameters, with an OpenAI-style speech endpoint.

Image Gen

OpenAI API via llama-swap ↗

Z-Image, FLUX.2 Klein and OmniGen2, each behind the same proxy.

The command behind this tab

hearthbench showcase --config config/showcase.yaml --out-dir DIR

Where the time goes in one spoken turn

Asked aloud: “Computer, how do I build a voice assistant that doesn't send my data to the cloud?” The clock stops at the first audio of the reply.

Hobby

Developer

Side by side

Means over the timed turns.

metric	Hobby	Developer
Speech to text	600ms	522ms
Text to speech, to first audio	1,581ms	290ms
Machine at workspeech to text + agent's first words + text to speech	2,184ms	817ms
Turn closed → first audio	2,087ms	800ms
Question asked → first audio	3,892ms	2,606ms

Every stage, every setup

Full statistics and the settings each run used.

Hobby

stage	mean ms	p50	p95	min	max	n
Speech to text	600	601	615	578	619	12
End-of-turn silence (a setting)	1,804	1,805	1,805	1,803	1,805	12
Agent stand-in, first words	2.8	2.7	3.8	1.3	4.1	12
Agent stand-in, first sentence complete (scripted)	503	504	504	502	505	12
Text to speech, to first audio	1,581	1,526	1,815	1,509	1,868	12
Turn closed → first audio	2,087	2,030	2,322	2,016	2,373	12
Question asked → first audio	3,892	3,835	4,127	3,821	4,178	12

Turns timed: 12 of 12, after 1 thrown away
Question clip: synthesized by this setup's TTS, voice am_eric, 4.47 s
End-of-turn silence: 1.8 s
Agent: voiceloop-agent: a scripted stand-in, no LLM. It sends its reply in two halves 0.5 s apart, so the first sentence completes 0.5 s after it starts.
Clock: this process, at the moment each event is observed

Events of the charted turn

event of the charted turn	ms
chunk_received	0.0
user_start	587
user_text	588
user_stop	1,805
agent_start	1,806
agent_text	1,806
agent_text	2,308
agent_stop	2,309
tts_request	2,309
audio	3,833

Developer

stage	mean ms	p50	p95	min	max	n
Speech to text	522	523	528	512	528	12
End-of-turn silence (a setting)	1,806	1,806	1,807	1,805	1,807	12
Agent stand-in, first words	5.2	5.2	5.5	4.9	5.5	12
Agent stand-in, first sentence complete (scripted)	505	505	505	504	505	12
Text to speech, to first audio	290	264	408	258	523	12
Turn closed → first audio	800	774	918	767	1,033	12
Question asked → first audio	2,606	2,580	2,724	2,573	2,838	12

Turns timed: 12 of 12, after 1 thrown away
Question clip: synthesized by this setup's TTS, voice am_eric, 4.49 s
End-of-turn silence: 1.8 s
Agent: voiceloop-agent: a scripted stand-in, no LLM. It sends its reply in two halves 0.5 s apart, so the first sentence completes 0.5 s after it starts.
Clock: this process, at the moment each event is observed

Events of the charted turn

event of the charted turn	ms
chunk_received	0.0
user_start	526
user_text	528
user_stop	1,807
agent_start	1,810
agent_text	1,812
agent_text	2,315
tts_request	2,317
agent_stop	2,318
audio	2,580

What this does not say

The clock starts with the whole question in hand. Microphone capture and the voice detector's closing silence are not in it.
The agent is a scripted stand-in, not a language model. Its amber bar is a fixed pause; a real model's time to a first sentence would take its place.
The grey bar is a wait the voice loop is set to make, the same on every setup. Transcription runs inside it.
Each chart draws the turn whose total was the median.

Locally Running Services

Everything on this tab came from these, running on the machine itself with no call to an outside API. Each one is open source: follow a link and run it.

STT

Whisper-STT ↗

Whisper small.en at 8-bit, on OpenVINO GenAI.

TTS

Kokoro-FastAPI ↗

Kokoro, 82M parameters, with an OpenAI-style speech endpoint.

The command behind this tab

hearthbench --services SETUP.yaml e2e --iterations 12 --merge-into RESULT.json

Language model

Streamed chat completions. First-token time is from the request; decode speed counts tokens after the first.

metric	Hobby	Developer
Model	phi4-int4-ov	qwen3.6-35b-a3b:Q4_K_P
KV cachecontext the server holds	grows as needed (cache_size 0), up to 256 sequences	262,144 tokens, one cache shared by 8 slots
Decode speedhigher is better	15.4tok/s	99.7tok/s
First tokenlower is better	382ms	197ms
First word of the answerlower is better; after any reasoning	382ms	197ms

Full statistics and every run

Hobby

timing	mean ms	p50	p95	min	max	n
First token of any kind	382	383	395	358	397	12
First token of the answer	382	383	395	358	397	12

Model served as: Phi-4-mini-instruct-int4-ov
Tokens per run: 128
Runs that never reached an answer: 0

Every run (12)

run	first token ms	first answer ms	wall ms	tokens	tokens/s
1	358	358	4,078	58.0	15.6
2	385	385	4,428	63.0	15.6
3	382	382	8,650	125	15.1
4	381	381	4,092	58.0	15.6
5	372	372	4,452	63.0	15.4
6	385	385	8,660	125	15.1
7	397	397	4,123	58.0	15.6
8	381	381	4,445	63.0	15.5
9	383	383	8,749	125	14.9
10	390	390	4,159	58.0	15.4
11	393	393	4,451	63.0	15.5
12	376	376	8,641	125	15.1

Developer

timing	mean ms	p50	p95	min	max	n
First token of any kind	197	198	264	130	278	12
First token of the answer	197	198	264	130	278	12

Model served as: qwen3
Tokens per run: 128
Runs that never reached an answer: 0

Every run (12)

run	first token ms	first answer ms	wall ms	tokens	tokens/s
1	130	130	613	47.0	97.3
2	278	278	1,582	128	98.1
3	213	213	1,505	128	99.0
4	136	136	606	47.0	100.0
5	253	253	1,534	128	99.9
6	196	196	1,473	128	100
7	137	137	602	47.0	101
8	254	254	1,537	128	99.8
9	192	192	1,480	128	99.3
10	136	136	602	47.0	101
11	246	246	1,527	128	100.0
12	199	199	1,475	128	100

Speech to text

Fixed clips, each sent whole. Speed is audio length over wall time.

metric	Hobby	Developer
Speedhigher is better	7.9× real time	9.3× real time
One clip, request to transcriptlower is better	462ms	392ms
Word error ratelower is better; clean test clips	0.000	0.000

Full statistics and every run

Hobby

timing	mean ms	p50	p95	min	max	n
One clip, request to transcript	462	455	493	440	498	12

Every run (12)

run	wall ms	audio s	RTF
1	442	3.5	0.128
2	469	3.7	0.127
3	498	3.7	0.133
4	446	3.5	0.129
5	488	3.7	0.132
6	451	3.7	0.121
7	450	3.5	0.130
8	480	3.7	0.129
9	446	3.7	0.119
10	440	3.5	0.127
11	477	3.7	0.129
12	459	3.7	0.123

Developer

timing	mean ms	p50	p95	min	max	n
One clip, request to transcript	392	386	413	376	415	12

Every run (12)

run	wall ms	audio s	RTF
1	384	3.5	0.111
2	415	3.7	0.112
3	390	3.7	0.104
4	386	3.5	0.112
5	412	3.7	0.111
6	382	3.7	0.102
7	378	3.5	0.109
8	409	3.7	0.110
9	386	3.7	0.103
10	378	3.5	0.109
11	408	3.7	0.110
12	376	3.7	0.101

Text to speech

Fixed sentences. First byte is when audio starts arriving.

metric	Hobby	Developer
Speedhigher is better	2.4× real time	27.6× real time
First audio bytelower is better	1,495ms	132ms

Full statistics and every run

Hobby

timing	mean ms	p50	p95	min	max	n
Request to first audio byte	1,495	819	4,809	755	6,100	12

Voice: af_bella
Speaking rate: 1.1

Every run (12)

run	first byte ms	wall ms	audio s	characters	RTF
1	768	769	3.5	61.0	0.222
2	6,100	6,100	3.7	66.0	1.6
3	3,753	3,753	3.7	66.0	1.0
4	755	755	3.5	61.0	0.219
5	863	863	3.7	66.0	0.233
6	778	778	3.7	66.0	0.208
7	772	773	3.5	61.0	0.223
8	835	836	3.7	66.0	0.226
9	803	804	3.7	66.0	0.215
10	801	802	3.5	61.0	0.232
11	854	855	3.7	66.0	0.230
12	852	853	3.7	66.0	0.228

Developer

timing	mean ms	p50	p95	min	max	n
Request to first audio byte	132	116	198	113	270	12

Voice: af_bella
Speaking rate: 1.1

Every run (12)

run	first byte ms	wall ms	audio s	characters	RTF
1	270	271	3.5	61.0	0.078
2	139	141	3.7	66.0	0.038
3	133	133	3.8	66.0	0.035
4	113	113	3.5	61.0	0.033
5	117	117	3.7	66.0	0.032
6	117	118	3.8	66.0	0.031
7	115	116	3.5	61.0	0.033
8	115	115	3.7	66.0	0.031
9	115	115	3.8	66.0	0.031
10	118	118	3.5	61.0	0.034
11	115	115	3.7	66.0	0.031
12	114	114	3.8	66.0	0.030

Embeddings

One passage at a time, then a small batch.

metric	Hobby	Developer
One passagelower is better	–	44.0ms
Batchedhigher is better	–	55vectors/s

Full statistics and every run

Developer

timing	mean ms	p50	p95	min	max	n
One passage	44.0	32.2	90.6	21.7	90.9	12

Model served as: qwen3-embed
Vector size: 1024

The setups

The config each run was given, as the result file recorded it, with credentials removed.

Hobby

Measured: 2026-10-12
Architecture: x86_64
hearth-bench commit: not recorded
Result schema: 0.2.0

args:
  iterations: 12
  max_tokens: 128
llm:
  base_url: http://127.0.0.1:4444/v3
  model: Phi-4-mini-instruct-int4-ov
  verify_tls: true
embeddings:
  base_url: http://127.0.0.1:4443/v1
  model: nomic-embed-text:latest
  verify_tls: true
stt:
  url: http://127.0.0.1:8080
tts:
  url: http://127.0.0.1:8880
  voice: af_bella
  speed: 1.1
agent:
  url: http://127.0.0.1:4500
judge:
  base_url: http://127.0.0.1:4444/v3
  model: Phi-4-mini-instruct-int4-ov
  verify_tls: true

Developer

Measured: 2026-10-12
Architecture: x86_64
hearth-bench commit: not recorded
Result schema: 0.2.0

args:
  iterations: 12
  max_tokens: 128
llm:
  base_url: http://127.0.0.1:8000/v1
  model: qwen3
  verify_tls: true
  extra_body:
    chat_template_kwargs:
      enable_thinking: false
embeddings:
  base_url: http://127.0.0.1:8000/v1
  model: qwen3-embed
  verify_tls: true
stt:
  url: http://127.0.0.1:8080
tts:
  url: http://127.0.0.1:8880
  voice: af_bella
  speed: 1.1
agent:
  url: http://127.0.0.1:4500
judge:
  base_url: http://127.0.0.1:8000/v1
  model: qwen3
  verify_tls: true

What this does not say

Each column is one result file from one machine. Run-to-run spread is in the statistics; machine-to-machine spread is not.
Speech-to-text accuracy is scored on a handful of clean clips. A word error rate of zero says the model is wired correctly, not that it is perfect.
A setup is one chosen configuration. Nothing here says it is the best one for its hardware.
Hobby: embeddings: unavailable at discovery, component skipped
Hobby: scoring: not run (use `hearthbench score`, it's slow/judge-cost-bearing)
Developer: scoring: not run (use `hearthbench score`, it's slow/judge-cost-bearing)

Locally Running Services

Everything on this tab came from these, running on the machine itself with no call to an outside API. Each one is open source: follow a link and run it.

LLM

Hobby

OpenAI API via OpenVINO Model Server ↗

Phi-4 mini at 4-bit.

Developer

OpenAI API via llama-swap ↗

llama.cpp servers behind a model-swapping proxy. A 35B-A3B model at 4-bit answers by default.

STT

Whisper-STT ↗

Whisper small.en at 8-bit, on OpenVINO GenAI.

TTS

Kokoro-FastAPI ↗

Kokoro, 82M parameters, with an OpenAI-style speech endpoint.

Embeddings

OpenAI API via llama-swap ↗

A 0.6B embedding model at 8-bit, on the same proxy.

The command behind this tab

hearthbench --services SETUP.yaml run --iterations 12 -o RESULT.json