How to Run Nemotron 3.5 Lightning Locally With llama.cpp

Nemotron 3.5 Lightning activates 3 billion parameters per token. Its Q4 GGUF is still nearly 24 GiB. If you want to run Nemotron 3.5 Lightning locally, that distinction determines whether you get a useful server or a very sophisticated out-of-memory error.

This guide uses llama.cpp to turn the model into a local OpenAI-compatible API. The setup starts with the actual file size, chooses a quant with headroom, caps the initial context, and tests thinking and instruct modes separately. The aim is not merely to make the weights load. It is to leave enough memory for the work around them.

Choose a Nemotron 3.5 quant from the file size

NVIDIA released Nemotron 3.5 Lightning on August 11, 2026 under the OpenMDW 1.1 license. The official model card describes a 30B-total, 3B-active mixture-of-experts model that combines Mamba-2, MoE, and selected attention layers.

The router activates a fraction of the model for each token, which reduces compute. It does not turn the other weights into mist. All 30 billion parameters still have to live in memory or move between system RAM and the accelerator. The current Unsloth GGUF repository makes the hardware tradeoff concrete:

Quant	File size	Reasonable starting point
UD-IQ2_M	18.10 GiB	16 GB VRAM with CPU offload
UD-Q3_K_XL	19.78 GiB	24 GB VRAM
UD-Q4_K_S	22.79 GiB	32 GB unified memory
UD-Q4_K_XL	23.75 GiB	32 GB+ with more headroom
Q8_0	32.60 GiB	48 GB-class systems

Those are file sizes, not complete memory requirements. The context state, runtime buffers, operating system, and your application also need space. On a 24 GiB capacity, Q3 leaves about 4.2 GiB before any of that overhead; Q4_K_XL leaves roughly 0.25 GiB. Start with Q3 on a 24 GB-class GPU. A 32 GB unified-memory system has a more credible Q4 budget, while a 16 GB GPU needs a smaller quant and partial CPU offload.

Unsloth summarizes 4-bit operation as roughly 20 GB of RAM, but the exact Q4 files currently range from 22.79 to 23.75 GiB. Check the repository before downloading; quant names are more useful than rounded marketing numbers. This is the same memory-first rule used in the Muse Glimmer local guide, with a different model and a considerably larger Q4 bill.

Build a current llama.cpp server

Use the current llama.cpp source so the model support and server flags match the commands below. After installing Git, CMake, and a C++ compiler, build the server:

git clone https://github.com/ggml-org/llama.cpp

cmake -S llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF \
  -DLLAMA_CURL=ON

cmake --build llama.cpp/build --config Release -j \
  --target llama-server

Metal is enabled by default on macOS. For an NVIDIA GPU, add -DGGML_CUDA=ON to the CMake configuration line. The official llama.cpp build guide also covers Vulkan and other backends. Confirm the binary exists before spending the next half-hour downloading weights:

./llama.cpp/build/bin/llama-server --version

Download one GGUF deliberately

The model repository contains several files large enough to make an accidental “download all” memorable. Install the Hugging Face client and request one quant explicitly. This example uses the safer 24 GB starting point:

python3 -m pip install -U huggingface_hub

hf download \
  unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF \
  --local-dir nemotron-3.5 \
  --include "*UD-Q3_K_XL*"

For 32 GB unified memory, replace the pattern with *UD-Q4_K_S* or *UD-Q4_K_XL*. For a 16 GB GPU plus ample system RAM, begin with *UD-IQ2_M* and offload only as many layers as fit. Unsloth’s local-running documentation uses Q4_K_XL; the change here is intentional because a 24 GB card needs breathing room.

Start a local API with a sane context

NVIDIA advertises context up to 1 million tokens. That is a capability ceiling, not a sensible first allocation on consumer hardware. Start at 16K, prove the server, and increase it only while measuring memory use.

./llama.cpp/build/bin/llama-server \
  --model nemotron-3.5/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-UD-Q3_K_XL.gguf \
  --ctx-size 16384 \
  --n-gpu-layers 999 \
  --reasoning on \
  --temp 0.6 \
  --top-p 0.95 \
  --min-p 0.01 \
  --host 127.0.0.1 \
  --port 8080

The high GPU-layer value asks llama.cpp to offload every layer it can. If the model does not fit, lower --n-gpu-layers until the server loads; llama.cpp supports hybrid CPU and GPU inference. That rescue path trades speed for capacity. On a CPU-only system, omit the option.

The official ggml-org card also offers a short llama-server -hf command. The explicit file path above is less magical and keeps the quant decision visible. When the server is ready, test its OpenAI-compatible endpoint:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "List three checks before editing a repository."}
    ],
    "temperature": 0.6
  }'

Switch between thinking and instruct mode

Nemotron 3.5 Lightning supports configurable reasoning through its chat template. Current llama-server builds expose that choice as --reasoning on, off, or auto. Unsloth recommends temperature 0.6 and top-p 0.95 for thinking mode. Use that profile when the model must inspect constraints or recover from a failed tool call.

For routine formatting, classification, or schema-constrained execution, restart with --reasoning off --temp 0.2. Keeping the profiles separate makes latency easier to measure and prevents a simple JSON cleanup from turning into a memoir. If the surrounding agent already has a stronger planner, Lightning can handle repeated execution while the larger model handles difficult decisions.

Troubleshoot memory, context, and stale builds

Symptom	First thing to check	First fix
Unknown model architecture	llama.cpp build date	Pull and rebuild current source
Out of memory during load	GGUF size versus free memory	Use Q3/Q2 or lower GPU layers
Load succeeds, generation fails	Context and runtime headroom	Reduce --ctx-size
Output is much slower than expected	Heavy CPU offload	Use a smaller quant or more accelerator memory
Other devices can reach the API	Bind address	Restore --host 127.0.0.1

Weights and context are different memory problems. A smaller quant reduces the first; a smaller --ctx-size reduces the second. Three billion active parameters can make token computation efficient, but repeated transfers from system RAM can erase that advantage. Change one variable at a time: build, quant, GPU offload, then context.

Keep the server on localhost while testing. Local weights improve data control, but they do not make tool calls trustworthy. Before granting shell, browser, or repository access, secure the agent around the local model with scoped permissions and output validation.

Use Lightning for execution, then test your own tasks

NVIDIA’s launch announcement positions Lightning as an execution model for high-volume agent work. Build a 20-task set covering tool selection, argument formatting, result validation, short repository operations, and structured summaries. Record schema-valid outputs, correct tool choices, wall-clock latency, and peak memory. One impressive answer is a demo; twenty repeatable tasks begin to resemble evidence.

Published scores can identify candidates, but they cannot tell you how CPU offload, quantization, your tools, and your prompts interact. Use them to form a test plan, then test model claims on your own workflow. The best quant is the smallest one that preserves enough quality while meeting your latency and memory limits.

A fast local model still needs memory headroom

The useful question is not whether Nemotron fits once. It is whether the server can repeat your target task while context grows, tools return messy data, and the surrounding application takes its share of memory. That is the test future “small-active” MoE releases will face too: compute efficiency can improve faster than workstation capacity.

Nemotron 3.5 Lightning’s active-parameter count is genuinely useful: it explains why a 30B model can serve repeated work efficiently. It does not waive the storage bill. Three billion parameters may work on each token, but your machine still has to host the other twenty-seven billion.