Fine-Tuning 4-Bit Models: Adapting a Base That Only Ships Quantized

You reach for QLoRA to adapt an open model to your task, open the Hugging Face repo, and find that the only weights on offer are 4-bit. There is no BF16 checkpoint. The config says MXFP4 or NVFP4, the model card says it was trained that way on purpose, and your training script, which was built to quantize a full-precision base down to 4-bit for memory, now has nothing to quantize. This is the moment fine-tuning 4-bit models stops being a solved recipe and becomes a precision decision you have to make deliberately.

The confusion is understandable, because the word "4-bit" shows up on both sides of the QLoRA workflow and means two different things. In QLoRA, 4-bit is a memory trick you apply to a full-precision base so it fits in VRAM while you train. In a 4-bit-native release, 4-bit is the model, the reference precision, the thing the vendor evaluated and shipped. Adapting the first is the standard tutorial. Adapting the second is a different problem, and treating it like the first is how you either double-quantize your base into mush or spend a week producing an adapter you cannot deploy.

This piece is the precision companion to the fine-tuning cluster. Our LoRA and QLoRA tutorial covers the standard recipe end to end, and our piece on Inkling as a fine-tuning base covers how to grade a base before you adapt it. Here the question is narrower: when the checkpoint is already quantized, what precision do you train the adapter in, what do you merge back into, and what do you actually serve.

Why QLoRA assumes a BF16 base

QLoRA is worth restating precisely, because every step of it leans on an assumption that a 4-bit-native release removes. The workflow has three moving parts.

First, it takes a frozen BF16 base and quantizes it to NF4, a 4-bit normal-float format, purely to shrink the memory footprint during training. This is a post-hoc, lossy compression. The underlying model was full precision; NF4 is a temporary coat you apply so a 70B base fits on one 48GB card.

Second, it trains a small LoRA adapter in BF16. The frozen NF4 base is dequantized on the fly, layer by layer, so the forward and backward passes compute in BF16 while only the tiny adapter matrices carry gradients. The base never updates; NF4 is just its storage form for the duration.

Third, when training finishes you can merge the BF16 adapter delta back into a BF16 base and get a single merged BF16 model, which you then quantize however you like for deployment. Or you keep the adapter separate and serve it against the base.

The load-bearing assumption is in the first and third steps: there is a full-precision base underneath. NF4 is reversible enough for training because the real weights were never 4-bit to begin with, and the merge target in step three is a BF16 checkpoint that exists. QLoRA is a memory optimization layered over a full-precision model. Take the full-precision model away and none of the three steps mean what they used to.

What changes when the base is 4-bit-native

A 4-bit-native checkpoint is quantization-aware trained. The model was optimized with the 4-bit numerics in the loop, so the released weights are not a compressed copy of a better checkpoint. They are the checkpoint. Kimi K3 is the clean example: Moonshot trained it MXFP4 from supervised fine-tuning through reinforcement learning, and the roughly 1.56 TB MXFP4 release is the only published precision. A BF16 K3 would be about 5.6 TB and does not exist. As we put it in the self-hosting breakdown, 4-bit here is a floor, not a dial: there is no higher-precision checkpoint to fall back to.

Three things change, one per QLoRA step.

There is nothing to quantize, and quantizing anyway is harmful. Running load_in_4bit over an already-4-bit checkpoint means stacking NF4 on top of MXFP4 or NVFP4. You would be re-quantizing weights that were carefully calibrated to their native 4-bit grid, snapping them onto a second, coarser 4-bit grid that knows nothing about the QAT calibration. The base gets worse before you have trained a single step. The correct move is to load the native checkpoint as-is and pass no quantization config at all.

The adapter still trains in higher precision, but the base stays native. You keep the LoRA adapter in BF16 and let the frozen 4-bit base dequantize on the fly for the matmul, exactly as QLoRA does, except the base is already in its shipped format rather than a memory-saving copy of a BF16 original. This part actually ports cleanly. The adapter is small, it holds gradients, and BF16 compute over a dequantized native-4-bit base is a reasonable training loop.

There may be nothing to merge into. This is the real break. A LoRA merge adds the BF16 adapter delta into the base weights and writes out a single model. If the base is BF16, the sum stays BF16 and the result is clean. If the base is a 4-bit-native checkpoint, merging the delta forces you to requantize the sum back onto the 4-bit grid, which throws away most of the adapter's precision and disturbs the very QAT calibration that made the base good. So on a 4-bit-native base you generally do not merge. You serve the base in its native format with the adapter kept separate, which both vLLM and SGLang support through multi-LoRA serving.

The difference between the two load paths is small in code and large in consequence. QLoRA quantizes a full-precision base on the way in:

# QLoRA over a BF16 base: NF4 is a memory trick applied to a full-precision model.
# transformers 4.5x, peft 0.15+, bitsandbytes 0.4x
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,   # dequantize to BF16 per layer for the matmul
)
base = AutoModelForCausalLM.from_pretrained(
    "org/some-bf16-base",                    # a full-precision checkpoint
    quantization_config=bnb,
    device_map="auto",
)
model = get_peft_model(base, LoraConfig(r=32, lora_alpha=32, target_modules="all-linear"))
# ... train; then base is BF16 upstream, so model.merge_and_unload() has a clean target.

A 4-bit-native checkpoint is already quantized, so you pass no quantization config at all. Adding one would double-quantize the base:

# 4-bit-native base (e.g. an MXFP4 QAT release): load as-is, no BitsAndBytesConfig.
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained(
    "org/some-mxfp4-native",                 # quantization already declared in the config
    device_map="auto",                       # loader honors the native 4-bit format
)
model = get_peft_model(base, LoraConfig(r=32, lora_alpha=32, target_modules="all-linear"))
# ... train the BF16 adapter; do NOT merge_and_unload (no higher-precision target).
# Serve the native base with the adapter attached via the engine's multi-LoRA path.

The three cases

The decision reduces to one question asked at the top: does a higher-precision checkpoint exist? The answer sorts every open release into one of three cases.

Case 1: A BF16 base ships (Inkling)

This is the case that is not really a new problem, and it is worth naming precisely so you do not overthink it. Thinking Machines shipped Inkling, its 975B-parameter Apache 2.0 mixture-of-experts model, with a full BF16 checkpoint and a separate NVFP4 deployment checkpoint. The BF16 needs roughly 2 TB of aggregated VRAM; the NVFP4 needs about 600 GB.

When a BF16 base ships, you have your full-precision model. Run standard QLoRA or plain LoRA against the BF16 base, merge the adapter back into BF16, then quantize the merged model for deployment or simply deploy the vendor's NVFP4 checkpoint with your adapter applied. The 4-bit checkpoint here is a serving artifact, not a training base, and you never train against it. Inkling is deliberately built for this: it is the clean case where "tune in BF16, serve in 4-bit" holds exactly as the tutorial assumes.

Case 2: An NVFP4 deployment checkpoint is what you were handed

The trap in Case 1 is confusing the deployment checkpoint for the base. NVFP4 is NVIDIA's 4-bit float format, and it is a strong one for serving: a per-block FP8 scale over 16-element micro-blocks, finer than MXFP4's coarser 32-element MX scaling with an E8M0 scale, which reports lower quality loss on most transformer weight distributions. The damage 4-bit does is concentrated on the tail and the outlier tokens, not on aggregate perplexity, which is exactly why NVFP4's tighter blocks matter and exactly the point our quantization deep dive makes about where low-bit actually breaks.

None of that makes the NVFP4 checkpoint a training base. If someone points you at an NVFP4 file and asks you to fine-tune it, the first question is whether a BF16 checkpoint exists upstream. If it does, tune that and requantize the merged result to NVFP4 yourself using a ModelOpt-style export, which vLLM has supported through its NVFP4 and ModelOpt paths since around v0.26.0, including per-token online MoE quantization. If a BF16 checkpoint does not exist, you are not in Case 2 at all. You are in Case 3.

Case 3: Only a 4-bit-native checkpoint ships (Kimi K3)

This is the hard case, and it is the one the tutorials do not cover. Kimi K3 ships MXFP4 as its only published checkpoint, QAT from SFT through RL. There is no BF16 to fall back to, so all three of the changes above apply at once: nothing to quantize, an adapter you train in BF16 over the native base, and no clean merge target.

The workflow that actually holds up is narrower than QLoRA's. Load the MXFP4 checkpoint in its native format with no additional quantization. Train a BF16 LoRA adapter against the dequantized-on-the-fly base. Do not merge; keep the adapter as a separate artifact and serve it against the native base with multi-LoRA. If your serving stack cannot host an adapter separately and demands a single merged checkpoint, then merging forces a requantization back to MXFP4, and you have to measure how much of the adapter's gain survives that step rather than assuming it does.

Community requantizations exist, and they do not solve this. There is an nvidia/Kimi-K3-NVFP4 conversion and a W4AFP8 variant, but converting one 4-bit format to another does not manufacture a higher-precision base to tune. You cannot recover representational capacity that the QAT release already spent. A requantized copy is still a 4-bit-native base for the purposes of this decision.

The comparison

The table encodes the whole argument. The first two rows are the same underlying situation, a full-precision base with a 4-bit serving artifact, and they collapse into the standard recipe. The third row is the genuinely different problem, and its distinguishing feature is the empty merge-target cell.

A decision procedure

Run this in order the moment you open the repo, before you write a training config.

1. Check for a higher-precision checkpoint. Read the repo file list and the config. If a BF16 (or FP8) checkpoint exists, you are in Case 1 or 2: tune that, and treat any 4-bit file as a deployment artifact only. Stop here.
2. If only a 4-bit checkpoint exists, confirm it is native, not a community post-hoc quant. A QAT release states so in the model card and technical report. A third-party post-hoc quant of a base whose BF16 exists elsewhere means the real base is elsewhere; go back to step 1 and fetch it.
3. Load the native checkpoint as-is. Pass no BitsAndBytesConfig, no load_in_4bit. The quantization is already in the checkpoint config and the loader will honor it. Adding NF4 on top double-quantizes and degrades the base.
4. Train a BF16 LoRA adapter, vanilla first. Rank 16 to 32, alpha equal to rank, all linear layers targeted. Do not reach for DoRA or an exotic PEFT variant until vanilla LoRA has given you a baseline on identical settings. Method rarely beats data.
5. Decide the merge based on whether a full-precision target exists. BF16 base present: merge to BF16, then quantize for deployment. 4-bit-native only: do not merge. Serve the native base with the adapter attached through your engine's multi-LoRA path.
6. Validate on your own distribution, not perplexity. Score a held-out slice of your real task before and after, and probe the long-context and tail behavior specifically, because that is where the 4-bit floor and any forced requantization do their damage. A flat perplexity delta hides a degraded tail.

The headroom question QLoRA never had to answer

There is one more difference that has no step in the workflow but decides whether the whole exercise is worth it. QLoRA's NF4-over-BF16 premise takes representational headroom for granted, because the base weights were full precision and NF4 was only their storage form during training. The model's capacity to represent rare patterns, the outlier channels, the long tail of the distribution, all of that lived in the BF16 weights and the adapter had room to push against it.

A 4-bit-native base has already spent that budget. Quantization-aware training does not make 4-bit free; it makes 4-bit the best it can be given the format, which means the representational capacity that a BF16 base holds in reserve is simply not present. The tail is where 4-bit takes its damage, and the tail is often exactly what a specialization adapter is trying to move. So the honest expectation for fine-tuning 4-bit models in Case 3 is that the adapter has less to work with than the same adapter over a full-precision base would, and that the effect is invisible in aggregate metrics and visible in the tail. This is consistent with the standing line across our fine-tuning coverage: fine-tune to change behavior, not to inject knowledge, keep vanilla LoRA as the baseline, and let your own before-and-after on your own data settle the question. The precision case adds one clause to that line. When there is no higher-precision checkpoint underneath, measure the tail, because the headroom you are counting on may already be gone.

None of this is a reason to avoid a 4-bit-native base. K3-class models are 4-bit-native precisely because BF16 at that scale is unservable, and the format is what makes the open release exist at all. It is a reason to size your expectations to the base you actually have, and to stop treating a QAT 4-bit checkpoint as a BF16 model wearing a smaller coat.

Key Takeaways

1. QLoRA is a memory optimization over a full-precision base, not a general 4-bit recipe. It NF4-quantizes a BF16 base for training memory, trains a BF16 adapter on top, and merges back into BF16. A 4-bit-native base removes the full-precision model that all three steps depend on.
2. The deciding question is whether a higher-precision checkpoint exists. If a BF16 or FP8 checkpoint ships, tune that and treat any 4-bit file as a deployment artifact. Inkling ships both BF16 (about 2 TB) and NVFP4 (about 600 GB) exactly so you can tune high and serve low.
3. Never run QLoRA over a 4-bit-native checkpoint. Passing load_in_4bit stacks NF4 on top of a QAT MXFP4 or NVFP4 base, snapping calibrated weights onto a second coarser grid and degrading the base before training starts. Load the native checkpoint with no added quantization config.
4. On a 4-bit-native base, do not merge. Merging a BF16 adapter forces a requantization back to the 4-bit grid that throws away adapter precision and disturbs the QAT calibration. Keep the adapter separate and serve it with multi-LoRA, as Kimi K3's MXFP4-only release requires.
5. NVFP4 and MXFP4 are serving formats, not training bases. NVFP4's 16-element micro-blocks with an FP8 scale report lower quality loss than MXFP4's coarser MX scaling, but converting one 4-bit format to another never manufactures the higher-precision base you would rather tune.
6. A 4-bit-native base has less adapter headroom, and the damage is on the tail. QAT does not make 4-bit free; the representational capacity a BF16 base holds in reserve is not present. Validate on your own distribution and probe the tail and long context, because perplexity hides exactly the degradation a specialization adapter cares about.
7. The standing fine-tuning line still holds, plus one clause. Vanilla LoRA first, data beats method, do not fine-tune to inject knowledge that RAG or context should carry. The clause the precision case adds: when no higher-precision checkpoint exists, measure the tail before you trust the gain.