Seven engines, one RTX 5090

Decode speed, prefill speed and time to first token for Basalt, Infernix, capy-flash, Strata, ninfer-ext, ninfer and llama.cpp, on NVIDIA's NVFP4 Qwen3.8-Flash-Next. In an October 2026 comparison of seven engines on one RTX 5090 at 400 W, Basalt had the highest decode speed in all seven requests (180 tok/s on prose against 163 for the next engine, Infernix; 354 on structured output against 349), the highest prefill at 50K, 100K and 200K tokens (8,666 tok/s at 100K against 6,630), and the shortest time to first token on the structured request and at 50K, 100K and 200K tokens (23.79 s at 200K against 31.81 s); Infernix was ahead at 25K prefill (6,493 against 6,492 tok/s) and on prose time to first token (0.16 s against 0.17 s), and ninfer had the shortest conversation time to first token (0.19 s against 0.23 s).

How it was measured. Bold marks the best value in each column.

One engine at a time. Every test starts a fresh server, and the first request after load is the measurement: no warm-up.
The same messages for every engine, greedy (temperature 0), thinking off. Tests: structured output, conversation, prose, and prompts of about 25K, 51K, 102K and 205K tokens.
Decode tok/s = (output tokens - 1) / (last - first token time). Prefill tok/s = prompt tokens / time to first token.
One RTX 5090 at 400 W, Ryzen 9 9950X, 121 GB RAM. int8 KV cache where the engine offers it (fp8 for ninfer and ninfer-ext). Experts that do not fit in VRAM stream from system RAM.
Each engine converted NVIDIA's NVFP4 Qwen3.8-Flash-Next checkpoint with its own tool (llama.cpp ran Magnitude's GGUF of it). October 2026; Basalt at commit 5b0eaeb8.

Decode speed, tokens per second (higher is better).
Engine	Struct	Conv	Prose	25K	50K	100K	200K
Basalt	354	188	180	190	187	184	160
Infernix	349	170	163	163	174	176	158
capy-flash	292	165	152	152	143	152	135
Strata	204	146	123	130	123	121	114
ninfer-ext	140	101	95	92	86	79	66
ninfer	54	77	45	53	46	47	44
llama.cpp	39	39	39	37	36	35	32

Prefill speed, tokens per second, by prompt length (higher is better).
Engine	25K	50K	100K	200K
Basalt	6,492	7,662	8,666	8,609
Infernix	6,493	6,544	6,630	6,436
capy-flash	4,026	4,272	5,206	5,458
Strata	3,530	3,738	4,129	4,245
ninfer-ext	3,552	4,049	4,683	4,588
ninfer	2,080	2,236	2,308	2,248
llama.cpp	1,354	1,420	1,394	1,257

Time to first token, seconds (lower is better).
Engine	Struct	Conv	Prose	25K	50K	100K	200K
Basalt	0.35	0.23	0.17	3.94	6.65	11.81	23.79
Infernix	0.42	0.24	0.16	3.94	7.79	15.43	31.81
capy-flash	0.88	0.50	0.29	6.35	11.93	19.65	37.52
Strata	0.45	0.29	0.20	7.24	13.64	24.78	48.24
ninfer-ext	1.05	0.47	0.21	7.19	12.59	21.85	44.63
ninfer	1.78	0.19	0.17	12.28	22.80	44.32	91.07
llama.cpp	1.09	0.75	0.31	18.87	35.91	73.39	162.86

Engines: commit, and what is not NVIDIA's weights

Basalt. the Basalt repository, commit 5b0eaeb8b18a9302db492d98c0120247d019d3a2. Routed experts are NVIDIA's NVFP4, bit-exact. PLE FP8 is re-encoded to Q8_0, and dense projections BF16 to Q8_0. The MTP drafter, expert profile and vision sections are copied from Basalt's own base pack (a self-distilled MTP head, not NVIDIA's MTP head).
Infernix. https://github.com/Wallawalla47/Infernix, main, commit 7dbe99a3ef7cf334142ad0bb6da5a8aacb06af84. Recipe Dense8 (Infernix's own benchmark recipe): dense classes 8-bit, routed experts and the FP8 PLE table as NVIDIA stores them; router, attention q/k/v, embedding and norms stay BF16. The drafter is Infernix's own trained DFlash2 drafter, not NVIDIA's MTP.
capy-flash. https://github.com/ValerioDolci/capy-flash, main, commit 4e45286a518bee631fa27cf24a32066ba1758222. Single-GPU NVFP4 is not an upstream-documented path (the profile targets the 64 GB edition). The MTP head is capy-flash's own re-distilled head with a draft shortlist; indexer BF16, GDN Q8; experts are NVIDIA's NVFP4.
Strata. https://github.com/Niko1221/Strata, main, commit 011c9e600a93bbc9fd451c563501b5831c32bbca. Experts are NVIDIA's NVFP4, bit-exact. The PLE table is ISTA's IQ4_NL table from another source (not NVIDIA's FP8); dense, head and embedding Q8_0; the MTP head is Qwen's BF16 head re-packed with q2_0 experts (not NVIDIA's FP8 MTP).
ninfer-ext. https://github.com/giveen/ninfer-ext, master, commit bc1686614cce26ab06f5176fd2cd557d5b1425f6. The MTP bank is re-encoded from block-FP8 to NVFP4; the n-gram table is re-scaled to one row-scaled FP8 matrix (not NVIDIA's per-tensor FP8); projections Q8 and vocab Q8/Q6 (NVIDIA stores BF16).
ninfer. https://github.com/Neroued/ninfer, master, commit 64d10fbf9f8e6bfc79ab0570b94674c5b9997929, patched by the benchmark's author. Q8 main projections; the PLE table is re-quantized to NF4 (lossier than the other engines' FP8 table); aux projections BF16. A quick perplexity check ranked it behind the others (3.519 against about 3.29).
llama.cpp. https://github.com/ggml-org/llama.cpp, master, commit 36cbf415f1c72c199e956c00fed0418f22ca53f2. NVIDIA's experts bit-exact including input_scale; other projections and the PLE table are Q8_0 from Magnitude's patched converter. Expert placement is llama.cpp's own --fit.

A second GPU

Hardware	RTX 5090 at 400 W + RTX 5060 Ti at 180 W, Ryzen 9 9950X, 128 GB DDR5
Pack and settings	Qwen3.8-FN-NVFP4-DQ6_K, int8 KV, Basalt main at c531c21a
Decode, prose	306.5 tok/s
Decode, conversation	334.9 tok/s
Decode, structured output	601.2 tok/s
Prefill, 16,691-token prompt	8,367 tok/s
Prefill, 66,836-token prompt	10,320 tok/s

Back to Basalt