# Basalt: a fast local inference engine for Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on NVIDIA Blackwell (sm_120)

> Basalt is a local inference engine for mixture-of-experts language models, by Jesús Gulfo (https://jgulfo.com). It runs on an NVIDIA Blackwell GPU (compute capability 12.0, sm_120) on Linux and Windows x64, with the experts that do not fit in VRAM held in system RAM or on more GPUs. Its server speaks the OpenAI and Anthropic APIs. MIT licensed. Page: https://basalt.jgulfo.com/

## Summary

- Models: Qwen3.8-Flash-Next (11 packs, default Qwen3.8-FN-NVFP4-DQ6_K) and DeepSeek-V4.1-Flash (2 packs). Each model is one .basalt file. Both accept images.
- Context: default 8,192 tokens, set with --max-context, up to 262,144 on Flash-Next and 1,048,576 on DeepSeek.
- Expert tiers: main-card VRAM, extra GPUs (Flash-Next: up to three more; DeepSeek: one more), then system RAM, with an adaptive tier that moves experts as the conversation changes.
- Parallel slots: --parallel N, 1 to 8.
- Structured output: tool calls and JSON schemas are enforced by a grammar while decoding (response_format json_schema or json_object, Anthropic output_config.format). On by default; --no-grammar turns it off; strict: true on a schema or tool still holds it. Output cut off by max_tokens can still be incomplete, and the grammar does not make the model pick the right tool.
- Speed: In an October 2026 comparison of seven engines on one RTX 5090 at 400 W, Basalt had the highest decode speed in all seven requests (180 tok/s on prose against 163 for the next engine, Infernix; 354 on structured output against 349), the highest prefill at 50K, 100K and 200K tokens (8,666 tok/s at 100K against 6,630), and the shortest time to first token on the structured request and at 50K, 100K and 200K tokens (23.79 s at 200K against 31.81 s); Infernix was ahead at 25K prefill (6,493 against 6,492 tok/s) and on prose time to first token (0.16 s against 0.17 s), and ninfer had the shortest conversation time to first token (0.19 s against 0.23 s).
- Quality: on the same Q8_0 weights, KL divergence from the BF16 reference is 0.0200 for Basalt against 0.0325 for the reference engine, which is 38% closer to the full-precision model.

## Benchmark: seven engines on one RTX 5090

October 2026. One RTX 5090 at 400 W. Each engine converted NVIDIA's NVFP4 Qwen3.8-Flash-Next with its own tool. The same seven greedy requests (structured output, conversation, prose, and prompts of 25K, 50K, 100K and 200K tokens), one cold start per request. Bold marks the best value in each column.

### How it was measured

- One engine at a time. Every test starts a fresh server, and the first request after load is the measurement: no warm-up.
- The same messages for every engine, greedy (temperature 0), thinking off. Tests: structured output, conversation, prose, and prompts of about 25K, 51K, 102K and 205K tokens.
- Decode tok/s = (output tokens - 1) / (last - first token time). Prefill tok/s = prompt tokens / time to first token.
- One RTX 5090 at 400 W, Ryzen 9 9950X, 121 GB RAM. int8 KV cache where the engine offers it (fp8 for ninfer and ninfer-ext). Experts that do not fit in VRAM stream from system RAM.
- Each engine converted NVIDIA's NVFP4 Qwen3.8-Flash-Next checkpoint with its own tool (llama.cpp ran Magnitude's GGUF of it). October 2026; Basalt at commit 5b0eaeb8.

### Decode speed, tokens per second (higher is better)

| Engine | Struct | Conv | Prose | 25K | 50K | 100K | 200K |
|---|---:|---:|---:|---:|---:|---:|---:|
| Basalt | **354** | **188** | **180** | **190** | **187** | **184** | **160** |
| Infernix | 349 | 170 | 163 | 163 | 174 | 176 | 158 |
| capy-flash | 292 | 165 | 152 | 152 | 143 | 152 | 135 |
| Strata | 204 | 146 | 123 | 130 | 123 | 121 | 114 |
| ninfer-ext | 140 | 101 | 95 | 92 | 86 | 79 | 66 |
| ninfer | 54 | 77 | 45 | 53 | 46 | 47 | 44 |
| llama.cpp | 39 | 39 | 39 | 37 | 36 | 35 | 32 |

### Prefill speed, tokens per second, by prompt length (higher is better)

| Engine | 25K | 50K | 100K | 200K |
|---|---:|---:|---:|---:|
| Basalt | 6,492 | **7,662** | **8,666** | **8,609** |
| Infernix | **6,493** | 6,544 | 6,630 | 6,436 |
| capy-flash | 4,026 | 4,272 | 5,206 | 5,458 |
| Strata | 3,530 | 3,738 | 4,129 | 4,245 |
| ninfer-ext | 3,552 | 4,049 | 4,683 | 4,588 |
| ninfer | 2,080 | 2,236 | 2,308 | 2,248 |
| llama.cpp | 1,354 | 1,420 | 1,394 | 1,257 |

### Time to first token, seconds (lower is better)

| Engine | Struct | Conv | Prose | 25K | 50K | 100K | 200K |
|---|---:|---:|---:|---:|---:|---:|---:|
| Basalt | **0.35** | 0.23 | 0.17 | **3.94** | **6.65** | **11.81** | **23.79** |
| Infernix | 0.42 | 0.24 | **0.16** | **3.94** | 7.79 | 15.43 | 31.81 |
| capy-flash | 0.88 | 0.50 | 0.29 | 6.35 | 11.93 | 19.65 | 37.52 |
| Strata | 0.45 | 0.29 | 0.20 | 7.24 | 13.64 | 24.78 | 48.24 |
| ninfer-ext | 1.05 | 0.47 | 0.21 | 7.19 | 12.59 | 21.85 | 44.63 |
| ninfer | 1.78 | **0.19** | 0.17 | 12.28 | 22.80 | 44.32 | 91.07 |
| llama.cpp | 1.09 | 0.75 | 0.31 | 18.87 | 35.91 | 73.39 | 162.86 |

Ratios from these tables: prefill up to 2.10x the engine most like Basalt (Strata, 8,666 against 4,129 tok/s at 100K) and up to 6.8x llama.cpp (8,609 against 1,257 at 200K); structured-output decode 1.74x Strata (354 against 204 tok/s) and 9.1x llama.cpp (354 against 39); time to first token at 100K 2.10x faster than Strata (11.81 s against 24.78 s) and at 200K 6.8x faster than llama.cpp (23.79 s against 162.86 s). Against the next-fastest engine, Infernix: 1.31x the prefill at 100K, and 1.10x the decode on prose (180 against 163 tok/s).

## Engine provenance

NV is NVIDIA's checkpoint, nvidia/Qwen3.8-Flash-Next-NVFP4. Each entry lists what is not NVIDIA's weights.

### Basalt

- Source: the Basalt repository, commit 5b0eaeb8b18a9302db492d98c0120247d019d3a2
- Convert: python tools/convert/convert.py --sections-from <basalt base NVFP4 pack> --checkpoint NV --dense-q8-0 --out Qwen3.8-FN-NVIDIA-NVFP4-Q8dense.basalt --name Qwen3.8-FN-NVIDIA-NVFP4-Q8dense
- Serve: bin/basalt serve --config cfg.json, with kv int8, resident-experts, vram-reserve-mib 500 and the shipped defaults for the rest
- Not NVIDIA's weights: Routed experts are NVIDIA's NVFP4, bit-exact. PLE FP8 is re-encoded to Q8_0, and dense projections BF16 to Q8_0. The MTP drafter, expert profile and vision sections are copied from Basalt's own base pack (a self-distilled MTP head, not NVIDIA's MTP head).

### Infernix

- Source: https://github.com/Wallawalla47/Infernix, main, commit 7dbe99a3ef7cf334142ad0bb6da5a8aacb06af84
- Convert: python -m tools.convert --model NV --recipe qwen3_8_flash_next_nvfp4_dense8 --components text,vision,mtp --proposal --device cpu
- Serve: infernix-serve with --kv-dtype int8 --spec dflash2 --draft-model Qwen3.8-Flash-Next-DFlash2-Infernix.infernix --draft-tokens auto --lm-head-draft
- Not NVIDIA's weights: Recipe Dense8 (Infernix's own benchmark recipe): dense classes 8-bit, routed experts and the FP8 PLE table as NVIDIA stores them; router, attention q/k/v, embedding and norms stay BF16. The drafter is Infernix's own trained DFlash2 drafter, not NVIDIA's MTP.

### capy-flash

- Source: https://github.com/ValerioDolci/capy-flash, main, commit 4e45286a518bee631fa27cf24a32066ba1758222
- Convert: python tools/capy_flash/convert.py --model NV --out <dir>
- Serve: NINFER_DEVICES=0 ninfer-serve <art> --profile flash-next-1gpu --device 0
- Not NVIDIA's weights: Single-GPU NVFP4 is not an upstream-documented path (the profile targets the 64 GB edition). The MTP head is capy-flash's own re-distilled head with a draft shortlist; indexer BF16, GDN Q8; experts are NVIDIA's NVFP4.

### Strata

- Source: https://github.com/Niko1221/Strata, main, commit 011c9e600a93bbc9fd451c563501b5831c32bbca
- Convert: tools/nvfp4_convert.py --model NV --outfile model.gguf, then tools/iq_pack.py, the MTP packing tools and the PLE table from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
- Serve: python -m serve.server --engine strata with --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --kv int8
- Not NVIDIA's weights: Experts are NVIDIA's NVFP4, bit-exact. The PLE table is ISTA's IQ4_NL table from another source (not NVIDIA's FP8); dense, head and embedding Q8_0; the MTP head is Qwen's BF16 head re-packed with q2_0 experts (not NVIDIA's FP8 MTP).

### ninfer-ext

- Source: https://github.com/giveen/ninfer-ext, master, commit bc1686614cce26ab06f5176fd2cd557d5b1425f6
- Convert: python -m tools.convert --model NV --recipe qwen3_8_flash_next_nvfp4 --components text,vision,mtp
- Serve: ninfer-serve <art> --kv-capacity auto --kv-dtype fp8 --expert-cache auto --spec mtp --draft-tokens 3
- Not NVIDIA's weights: The MTP bank is re-encoded from block-FP8 to NVFP4; the n-gram table is re-scaled to one row-scaled FP8 matrix (not NVIDIA's per-tensor FP8); projections Q8 and vocab Q8/Q6 (NVIDIA stores BF16).

### ninfer

- Source: https://github.com/Neroued/ninfer, master, commit 64d10fbf9f8e6bfc79ab0570b94674c5b9997929, patched by the benchmark's author
- Convert: tools.convert with --recipe qwen3_8_flash_next --source nvidia=NV, after a 35-line patch to tools/convert/qwen4_exp.py so the converter reads NVIDIA's FP8 PLE table and NVFP4 experts directly
- Serve: ninfer-serve q.ninfer --prefill-chunk 4096 --kv-dtype fp8 --spec mtp --draft-tokens 3 --no-prefix-reuse
- Not NVIDIA's weights: Q8 main projections; the PLE table is re-quantized to NF4 (lossier than the other engines' FP8 table); aux projections BF16. A quick perplexity check ranked it behind the others (3.519 against about 3.29).

### llama.cpp

- Source: https://github.com/ggml-org/llama.cpp, master, commit 36cbf415f1c72c199e956c00fed0418f22ca53f2
- Convert: none: Magnitude's GGUF, magnitudedev/Qwen3.8-Flash-Next-NVFP4-GGUF, not converted by the benchmark's author
- Serve: llama-server -fit on -np 1 -fa on -b 4096 -ub 4096 --no-webui, MTP off
- Not NVIDIA's weights: NVIDIA's experts bit-exact including input_scale; other projections and the PLE table are Q8_0 from Magnitude's patched converter. Expert placement is llama.cpp's own --fit.

## Basalt with a second GPU

| | |
|---|---|
| Hardware | RTX 5090 at 400 W + RTX 5060 Ti at 180 W, Ryzen 9 9950X, 128 GB DDR5 |
| Pack and settings | Qwen3.8-FN-NVFP4-DQ6_K, int8 KV, Basalt main at c531c21a |
| Decode, prose | 306.5 tok/s |
| Decode, conversation | 334.9 tok/s |
| Decode, structured output | 601.2 tok/s |
| Prefill, 16,691-token prompt | 8,367 tok/s |
| Prefill, 66,836-token prompt | 10,320 tok/s |

## FAQ

### What is Basalt?

Basalt is a local inference engine for mixture-of-experts language models. It runs Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on an NVIDIA Blackwell GPU (compute capability 12.0, sm_120) on Linux and Windows x64, with the experts that do not fit in VRAM held in system RAM or on more GPUs. Its server speaks the OpenAI and Anthropic APIs. It is MIT licensed.

### Which GPUs does it support?

NVIDIA Blackwell GPUs with compute capability 12.0 (sm_120) and CUDA 13. Basalt is built for Blackwell only. Linux x86-64 with AVX2 is where it is tested and benchmarked, and Windows 10 1903+ x64 is supported with some limits. Community benchmark runs include the RTX 5090, RTX 5080, RTX 5070 Ti, RTX 5060 Ti and RTX PRO 4000 and 6000 Blackwell cards.

### Which models does it run?

Qwen3.8-Flash-Next, with 11 ready-made packs on Hugging Face (default Qwen3.8-FN-NVFP4-DQ6_K), and DeepSeek-V4.1-Flash, with 2 packs. Each model is one .basalt file. Context defaults to 8,192 tokens and --max-context sets it, up to 262,144 on Flash-Next and 1,048,576 on DeepSeek. Both models accept images.

### How fast is it on an RTX 5090?

In a comparison of seven engines on one RTX 5090 at 400 W (NVIDIA's NVFP4 Qwen3.8-Flash-Next, the same greedy requests), Basalt decoded prose at 180 tokens per second and structured output at 354 tokens per second. Prefill is a strength: 8,666 tokens per second at 100K tokens, 1.31 times the next-fastest engine, and up to 2.1 times the engine most like it. With a second GPU (an RTX 5060 Ti) it decodes prose at 306.5 tokens per second and prefills 10,320 tokens per second at 64K tokens.

### How does Basalt compare to other engines?

In an October 2026 comparison of seven engines on one RTX 5090 at 400 W, Basalt had the highest decode speed in all seven requests (180 tok/s on prose against 163 for the next engine, Infernix; 354 on structured output against 349), the highest prefill at 50K, 100K and 200K tokens (8,666 tok/s at 100K against 6,630), and the shortest time to first token on the structured request and at 50K, 100K and 200K tokens (23.79 s at 200K against 31.81 s); Infernix was ahead at 25K prefill (6,493 against 6,492 tok/s) and on prose time to first token (0.16 s against 0.17 s), and ninfer had the shortest conversation time to first token (0.19 s against 0.23 s).

### Can it use a second GPU or system RAM?

Yes. The main card's VRAM holds the most routed experts, extra GPUs hold experts nowhere else has, and system RAM takes the rest. An adaptive tier moves experts between them as the conversation changes. Qwen3.8-Flash-Next takes the main card plus up to three expert-tier cards, and DeepSeek-V4.1-Flash the main card plus one.

### Does it support tool calls and structured output?

Yes. Tool calls and JSON output (response_format json_schema or json_object, and Anthropic output_config.format) are held to their schema by a grammar applied while decoding. It is on by default, --no-grammar turns it off, and strict: true on a schema or tool still holds it. Output cut off by max_tokens can still be incomplete, and the grammar does not make the model pick the right tool.

### Is it open source, and how do I run it?

Basalt is MIT licensed, at github.com/jesdga95/basalt. Build it with CMake 3.24+ (GCC 13+ on Linux), CUDA 13 and Python 3.10+, download a pack from Hugging Face, then run bin/basalt serve --config basalt_config.json. The server listens on 127.0.0.1.

## Links

- Page: https://basalt.jgulfo.com/
- Benchmarks: https://basalt.jgulfo.com/benchmarks/
- Source: https://github.com/jesdga95/basalt
- Packs: https://huggingface.co/jesdga/Qwen3.8-Flash-Next-Basalt and https://huggingface.co/jesdga/DeepSeek-V4.1-Flash-1017-Basalt
- Community benchmarks: https://jesdga95.github.io/basalt/