# Basalt

> A fast local inference engine for Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on NVIDIA Blackwell (sm_120), by Jesús Gulfo (https://jgulfo.com). MIT licensed.

Basalt is a local inference engine for mixture-of-experts language models. It runs Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on an NVIDIA Blackwell GPU (compute capability 12.0, sm_120) on Linux and Windows x64. The experts that do not fit in VRAM live in system RAM, and optionally on more GPUs. The server speaks the OpenAI and Anthropic APIs and ships a web UI.

- Models: Qwen3.8-Flash-Next (11 packs, default NVFP4-DQ6_K, images, context up to 262,144 tokens) and DeepSeek-V4.1-Flash (2 packs, images, context up to 1,048,576 tokens). Each model is one .basalt file.
- Expert tiers: the main card's VRAM holds the most routed experts, extra GPUs hold experts nowhere else has, and system RAM takes the rest. An adaptive tier moves them as the conversation changes.
- Parallel slots: --parallel N (1 to 8) serves N requests at once. The context is shared between the slots.
- Structured output: tool calls and JSON (response_format json_schema or json_object, Anthropic output_config.format) are held to their schema by a grammar applied while decoding. On by default.
- Speed: In an October 2026 comparison of seven engines on one RTX 5090 at 400 W, Basalt had the highest decode speed in all seven requests (180 tok/s on prose against 163 for the next engine, Infernix; 354 on structured output against 349), the highest prefill at 50K, 100K and 200K tokens (8,666 tok/s at 100K against 6,630), and the shortest time to first token on the structured request and at 50K, 100K and 200K tokens (23.79 s at 200K against 31.81 s); Infernix was ahead at 25K prefill (6,493 against 6,492 tok/s) and on prose time to first token (0.16 s against 0.17 s), and ninfer had the shortest conversation time to first token (0.19 s against 0.23 s).
- Quality: scored against a BF16 reference by KL divergence. On the same Q8_0 weights, Basalt is 38% closer to the full-precision model than the reference engine (KL divergence 0.0200 against 0.0325).
- Requires: Linux x86-64 with AVX2 or Windows 10 1903+ x64, an NVIDIA Blackwell GPU with CUDA 13, CMake 3.24+, GCC 13+ (Linux), Python 3.10+, and enough RAM for the experts that do not fit in VRAM.

Page: https://basalt.jgulfo.com/
Benchmarks, with the method and every number: https://basalt.jgulfo.com/benchmarks/
Everything in plain text, including the FAQ: https://basalt.jgulfo.com/llms-full.txt
Source: https://github.com/jesdga95/basalt
Packs: https://huggingface.co/jesdga/Qwen3.8-Flash-Next-Basalt and https://huggingface.co/jesdga/DeepSeek-V4.1-Flash-1017-Basalt
Community benchmarks: https://jesdga95.github.io/basalt/