Basalt

Big MoE models on one Blackwell GPU.

Basalt is a fast local inference engine for Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on NVIDIA Blackwell (sm_120). The experts that do not fit in VRAM go to a second GPU or to system RAM. The server speaks the OpenAI and Anthropic APIs.

Quick start GitHub Packs on Hugging Face
Expert tiers

Experts sit in layers, hottest on top.

Each token uses only a few experts. Basalt keeps the busiest in VRAM, puts more on one or more extra GPUs, and leaves the rest in RAM. As the conversation shifts, so do they.

Every kernel is written for Blackwell, FP4 included. Nothing else.

Tier 1Main GPUThe most routed experts, in VRAM. Tier 2Peer GPUsOne or more extra cards, optional. Tier 3System RAMEverything else.
Web UI

See what it is doing.

Speed, both GPUs, the expert tiers and the log, live on the same port as the API.

Basalt web UI: decode speed, prefill speed, two GPUs and the log while a long prompt streams

Up to eight requests at once.

Every slot shows its own speed, context and draft accept rate. OpenAI and Anthropic clients share the same slots.

Basalt web UI with four parallel slots busy: three decoding and one reading a long prompt, split between OpenAI and Anthropic requests
Models

Two models, one command.

Each model is a single .basalt file that holds everything the server needs.

Qwen3.8-Flash-Next

437.5tok/s decode

  • 256K context
  • 11 ready-made packs
  • Images supported

RTX 5090 at 400 W + RTX 5060 Ti at 180 W, NVFP4, Linux. 7,741 tok/s prefill at 16K.

DeepSeek-V4.1-Flash

149.1tok/s decode

  • 1M context
  • 2 ready-made packs
  • Images supported

RTX 5090 at 400 W + RTX 5060 Ti at 180 W, 121 GB RAM, IQ1_M pack, Linux. 2,133 tok/s prefill at 16K.

Structured output

No more broken tool calls.

Basalt blocks malformed JSON while it generates, so your agents get arguments they can parse, not retries.

Compare

Faster, and closer to the source.

Up to2x

Faster prefill than similar engines.

Up to 6.8x vs general-purpose engines

Up to1.7x

Faster decode than similar engines.

Up to 9x vs general-purpose engines

Up to2x

Faster time to first token than similar engines on long prompts.

Up to 6.8x vs general-purpose engines

38%

Closer to the full-precision model than the reference engine, on the same weights.

Measured on one RTX 5090 at 400 W, NVFP4 Qwen3.8-Flash-Next, same greedy requests.

All the numbers and how they were measured

Quick start

One file in, one endpoint out, on 127.0.0.1 with the OpenAI and Anthropic APIs.

1. Build and download a pack 
git clone https://github.com/jesdga95/basalt && cd basalt
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
pip install -r requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt \
  Qwen3.8-FN-NVFP4-DQ6_K.basalt --local-dir models
2. Configure and serve 
{
  "version": 2,
  "models": [{ "model": "models/Qwen3.8-FN-NVFP4-DQ6_K.basalt",
               "gpu": "0,1", "kv": "int8" }],
  "engine": { "resident-experts": true },
  "server": { "port": 8080 }
}

Needs Linux x86-64 or Windows 10 1903+ x64, an NVIDIA Blackwell GPU with CUDA 13, CMake 3.24+, Python 3.10+, and RAM for the experts that do not fit in VRAM.

Community

Runs from real machines.

Submit yours with basalt-bench --community.

HardwarePackPrefill 16KDecode
RTX 5090 at 400 W + 400 W (2 cards)NVFP48,233553.6
RTX 5090 at 400 W + RTX 5060 Ti at 180 WNVFP47,741437.5
RTX 5090 at 400 WNVFP47,733317.2
RTX 5080 at 360 WIQ3_S4,877186.8
RTX 5060 Ti at 180 WNVFP42,68791.9
RTX 5060 Ti at 180 W + 180 W (2 cards)IQ3_S2,370161.3

See every run.

What people say

“I'm getting 169 tok/s generation speed. … So Basalt is giving the best results so far. Thank you for the great job!”

@BuyukBang · Hugging Face

“Basalt's own sm120 optimizations and NVFP4 for Blackwell is a real deal”

@Bairaktarchik · Hugging Face

“Downloaded and working like a charm! … a single 5070ti + 64gb ddr5 + 5800x3d still gives me ~1900pp and ~80 tg for code analysis thinking, so it is still very comfortable to use.”

@matthewanthony · Hugging Face

“this is really amazingly fast on a machine like mine that is limited by RAM.”

@wright802 · GitHub

FAQ

What is Basalt?

A local inference engine for mixture-of-experts models on NVIDIA Blackwell GPUs. It runs Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash and speaks the OpenAI and Anthropic APIs.

Which GPUs does it support?

NVIDIA Blackwell (sm_120) with CUDA 13, on Linux and Windows x64. Community runs include the RTX 5090, 5080, 5070 Ti, 5060 Ti and the RTX PRO 4000 and 6000 Blackwell.

Which models does it run?

Qwen3.8-Flash-Next (11 packs) and DeepSeek-V4.1-Flash (2 packs), both with image input. Context goes up to 262,144 tokens on Flash-Next and 1,048,576 on DeepSeek.

How fast is it on an RTX 5090?

On one RTX 5090 at 400 W: 180 tok/s on prose, 354 on structured output, and 8,666 tok/s prefill at 100K tokens. With a second GPU, an RTX 5060 Ti, prose decodes at 306.5 tok/s and a 64K prompt prefills at 10,320 tok/s.

How does Basalt compare to other engines?

In a seven-engine comparison on one RTX 5090, Basalt had the highest decode speed in all seven requests and the highest prefill at 50K, 100K and 200K tokens, 1.31 times the next-fastest engine at 100K. The full tables are on the benchmarks page.

Can it use a second GPU or system RAM?

Yes. The main card holds the busiest experts, extra GPUs the next tier and system RAM the rest, and an adaptive tier moves them as the conversation changes.

Does it support tool calls and structured output?

Yes. Tool calls and JSON schemas are enforced by a grammar while decoding, on by default. Output cut off by max_tokens can still be incomplete.

Is it open source?

Yes, MIT licensed, at github.com/jesdga95/basalt. Build it, download a pack from Hugging Face and run bin/basalt serve --config basalt_config.json.

All the numbers

How it was measured. Bold marks the best value in each column.
  • One engine at a time. Every test starts a fresh server, and the first request after load is the measurement: no warm-up.
  • The same messages for every engine, greedy (temperature 0), thinking off. Tests: structured output, conversation, prose, and prompts of about 25K, 51K, 102K and 205K tokens.
  • Decode tok/s = (output tokens - 1) / (last - first token time). Prefill tok/s = prompt tokens / time to first token.
  • One RTX 5090 at 400 W, Ryzen 9 9950X, 121 GB RAM. int8 KV cache where the engine offers it (fp8 for ninfer and ninfer-ext). Experts that do not fit in VRAM stream from system RAM.
  • Each engine converted NVIDIA's NVFP4 Qwen3.8-Flash-Next checkpoint with its own tool (llama.cpp ran Magnitude's GGUF of it). October 2026; Basalt at commit 5b0eaeb8.
Decode speed, tokens per second (higher is better).
EngineStructConvProse25K50K100K200K
Basalt354188180190187184160
Infernix349170163163174176158
capy-flash292165152152143152135
Strata204146123130123121114
ninfer-ext1401019592867966
ninfer54774553464744
llama.cpp39393937363532
Prefill speed, tokens per second, by prompt length (higher is better).
Engine25K50K100K200K
Basalt6,4927,6628,6668,609
Infernix6,4936,5446,6306,436
capy-flash4,0264,2725,2065,458
Strata3,5303,7384,1294,245
ninfer-ext3,5524,0494,6834,588
ninfer2,0802,2362,3082,248
llama.cpp1,3541,4201,3941,257
Time to first token, seconds (lower is better).
EngineStructConvProse25K50K100K200K
Basalt0.350.230.173.946.6511.8123.79
Infernix0.420.240.163.947.7915.4331.81
capy-flash0.880.500.296.3511.9319.6537.52
Strata0.450.290.207.2413.6424.7848.24
ninfer-ext1.050.470.217.1912.5921.8544.63
ninfer1.780.190.1712.2822.8044.3291.07
llama.cpp1.090.750.3118.8735.9173.39162.86
Engines: commit, and what is not NVIDIA's weights
  • Basalt. the Basalt repository, commit 5b0eaeb8b18a9302db492d98c0120247d019d3a2. Routed experts are NVIDIA's NVFP4, bit-exact. PLE FP8 is re-encoded to Q8_0, and dense projections BF16 to Q8_0. The MTP drafter, expert profile and vision sections are copied from Basalt's own base pack (a self-distilled MTP head, not NVIDIA's MTP head).
  • Infernix. https://github.com/Wallawalla47/Infernix, main, commit 7dbe99a3ef7cf334142ad0bb6da5a8aacb06af84. Recipe Dense8 (Infernix's own benchmark recipe): dense classes 8-bit, routed experts and the FP8 PLE table as NVIDIA stores them; router, attention q/k/v, embedding and norms stay BF16. The drafter is Infernix's own trained DFlash2 drafter, not NVIDIA's MTP.
  • capy-flash. https://github.com/ValerioDolci/capy-flash, main, commit 4e45286a518bee631fa27cf24a32066ba1758222. Single-GPU NVFP4 is not an upstream-documented path (the profile targets the 64 GB edition). The MTP head is capy-flash's own re-distilled head with a draft shortlist; indexer BF16, GDN Q8; experts are NVIDIA's NVFP4.
  • Strata. https://github.com/Niko1221/Strata, main, commit 011c9e600a93bbc9fd451c563501b5831c32bbca. Experts are NVIDIA's NVFP4, bit-exact. The PLE table is ISTA's IQ4_NL table from another source (not NVIDIA's FP8); dense, head and embedding Q8_0; the MTP head is Qwen's BF16 head re-packed with q2_0 experts (not NVIDIA's FP8 MTP).
  • ninfer-ext. https://github.com/giveen/ninfer-ext, master, commit bc1686614cce26ab06f5176fd2cd557d5b1425f6. The MTP bank is re-encoded from block-FP8 to NVFP4; the n-gram table is re-scaled to one row-scaled FP8 matrix (not NVIDIA's per-tensor FP8); projections Q8 and vocab Q8/Q6 (NVIDIA stores BF16).
  • ninfer. https://github.com/Neroued/ninfer, master, commit 64d10fbf9f8e6bfc79ab0570b94674c5b9997929, patched by the benchmark's author. Q8 main projections; the PLE table is re-quantized to NF4 (lossier than the other engines' FP8 table); aux projections BF16. A quick perplexity check ranked it behind the others (3.519 against about 3.29).
  • llama.cpp. https://github.com/ggml-org/llama.cpp, master, commit 36cbf415f1c72c199e956c00fed0418f22ca53f2. NVIDIA's experts bit-exact including input_scale; other projections and the PLE table are Q8_0 from Magnitude's patched converter. Expert placement is llama.cpp's own --fit.

A second GPU

HardwareRTX 5090 at 400 W + RTX 5060 Ti at 180 W, Ryzen 9 9950X, 128 GB DDR5
Pack and settingsQwen3.8-FN-NVFP4-DQ6_K, int8 KV, Basalt main at c531c21a
Decode, prose306.5 tok/s
Decode, conversation334.9 tok/s
Decode, structured output601.2 tok/s
Prefill, 16,691-token prompt8,367 tok/s
Prefill, 66,836-token prompt10,320 tok/s
Permalink