LLM Inference Interview Questions: The Serving Ladder Interviewers Actually Climb

Inference is where LLM interviews stop rewarding people who have only read papers. Anyone can say a transformer predicts the next token; the serving question asks what actually happens on the GPU when a request arrives, and it escalates as cleanly as attention does. "How would you serve this model" sounds like a systems warm-up, but the honest answer forces a distinction most candidates skip, and once you skip it every follow-up answer sits on sand. That is what the good version of these llm inference interview questions is really testing: not whether you can name a serving framework, but whether you understand where the time and the memory actually go.

This guide walks the ladder the way an interviewer climbs it. For each rung: the question as it gets asked, what it is really probing, the answer at the depth that lands, and the trap that ends the conversation early. It is the serving companion to attention mechanism interview questions, and it leans on the same habit: connect the mechanism to the machine.

Rung 1: "Walk me through what happens when the model generates a response"

What it probes: whether you know that inference has two phases with opposite performance characteristics, or you think of generation as one uniform loop.

The answer that lands names prefill and decode. When a request arrives, the model first processes the entire prompt in a single forward pass, computing attention over all input tokens at once. That is prefill, and it is compute-bound: you are doing large matrix multiplications with high arithmetic intensity, so the GPU's math units are the constraint. Then generation begins, one token at a time, each new token attending back over everything before it. That is decode, and it is memory-bandwidth-bound: each step moves the entire model's weights and the growing cache through the memory system to produce a single token, so the bottleneck is how fast you can read memory, not how fast you can multiply.

The trap: treating generation as one loop. The candidate who says "it runs a forward pass per token" is not wrong, but has flattened the single most important distinction in serving. Prefill saturates compute; decode starves on bandwidth. Almost every optimization in the rest of this interview exists because those two phases want different things, and batching, quantization, and speculative decoding all target the memory-bound decode phase specifically. If you do not separate the phases here, rung four onward will not have anywhere to stand.

Rung 2: "Are you optimizing for latency or throughput?"

What it probes: whether you know these are different, often opposed objectives, and can name the metrics for each.

The strong answer refuses the false choice and names the metrics. Latency splits into two numbers: time to first token (TTFT), dominated by prefill, and time per output token (TPOT), dominated by decode. Throughput is tokens per second across all concurrent requests, which is what determines your cost per token. The two objectives pull against each other: batching more requests together raises throughput because you amortize each weight load across many sequences, but it can raise per-request latency because everyone waits for the slowest member of the batch. A chat product optimizes TTFT and TPOT for one user; a batch summarization pipeline optimizes throughput and does not care about tail latency. Knowing which one you are serving decides every later tradeoff.

The trap: quoting a single "speed" number. "It does 100 tokens per second" is meaningless without asking: per request or aggregate, at what batch size, at what context length. An interviewer who hears an unqualified throughput figure will immediately ask "at what concurrency," and the candidate who has to backpedal has revealed they think of performance as one scalar. The senior move is to ask which objective matters before proposing anything, the same reflex that separates real capacity planning from benchmark quoting.

Rung 3: "What is the KV cache and why does it dominate memory?"

What it probes: whether you understand the central object of LLM serving, the one every other technique is built around.

During decode, each new token attends over the keys and values of every previous token. Recomputing those from scratch at every step would make generation quadratic in output length, so instead you cache the key and value vectors for every past token and reuse them. That cache is what makes autoregressive generation tractable, and it is also what makes serving memory-bound: it grows linearly with sequence length and linearly with batch size, and it lives in the same high-bandwidth memory as the weights. At long context and high concurrency, the KV cache, not the model weights, becomes the dominant consumer of GPU memory, and its size is the hard ceiling on how many requests you can run at once.

The strong answer puts numbers to the shape: cache size scales as 2 * layers * kv_heads * head_dim * seq_len * batch * bytes, the factor of two for keys and values. That formula is why the number of key/value heads matters, and why the MHA to GQA to MLA progression from the attention interview is really a KV-cache-shrinking story, not a quality ranking.

The trap: describing the KV cache as an optimization you can bolt on. It is not optional; without it, decode is quadratic and unservable. The real insight is that the cache is a two-edged object: it buys speed (no recomputation) at the cost of memory (it grows without bound), and the entire serving stack is the ongoing negotiation of that trade. The full treatment is in KV cache engineering.

Rung 4: "How do you serve many users at the same time?"

What it probes: whether you know how modern servers actually schedule work, or you are still imagining static batching.

The answer that lands is continuous batching (also called in-flight batching). Naive static batching groups N requests, runs them together, and waits for all N to finish before starting the next batch, which wastes the GPU whenever requests have different output lengths: short generations sit idle waiting for the long one in their batch. Continuous batching instead operates at the granularity of a single decode step. It maintains a running set of active sequences, and the moment any sequence emits its end token and frees its slot, a waiting request is admitted in its place. The GPU stays saturated because finished sequences are evicted and new ones injected every step rather than every batch. This is the single biggest throughput lever in modern serving, and it is why frameworks like vLLM and SGLang exist.

Continuous batching depends on rung three: you can only pack more concurrent sequences if you can fit their KV caches, which is why paged KV memory (managing the cache in fixed blocks like virtual memory pages, so you avoid fragmentation and can pack the cache tightly) is the companion technique. Batching and paging are two halves of the same "fit and schedule more requests on one GPU" story.

The trap: describing static batching and stopping, or naming vLLM without being able to say what it does differently. The interviewer wants the mechanism: step-level scheduling that admits and evicts sequences continuously so the GPU never idles on a half-empty batch. A candidate who can explain why static batching wastes the GPU has shown they understand the problem continuous batching solves, which is worth more than the framework name. The applied picture is in LLM inference optimization.

Rung 5: "The model does not fit. What do you do?"

What it probes: whether you understand quantization as a set of distinct decisions with distinct costs, not a single "make it smaller" knob.

The strong answer separates what you are quantizing. Weight quantization (to 8-bit, 4-bit, or the FP4 formats current MoE models ship in natively) shrinks the model's memory footprint and, because decode is bandwidth-bound, often speeds up decode by reducing how many bytes you move per token. KV-cache quantization shrinks the cache from rung three, letting you fit longer context or more concurrent requests. Activation quantization is the aggressive frontier that lets the matrix multiplications themselves run in low precision. These are independent choices with independent risks, and conflating them is the tell of someone who has only run load_in_4bit once.

The senior addition is knowing where the damage lands. Low-bit quantization rarely moves aggregate perplexity much, which is exactly why it is dangerous: the degradation concentrates in the tail, on rare tokens and long-context recall, precisely the cases a headline benchmark does not stress. So the honest protocol is to validate the quantized model on your own distribution and your own long-context behavior, not to trust a perplexity delta. And a 4-bit-native model (quantization-aware-trained, shipping with no higher-precision checkpoint) is a floor, not a dial: you cannot fall back to more bits for the tail because the higher-precision weights do not exist.

The trap: "quantize to 4-bit, it barely changes quality." That sentence is true on average and false where it matters. The interviewer probing this rung wants to hear that you know quantization is nearly free on the mean and expensive on the tail, and that the measurement that catches it is not the one vendors report. The mechanics are in LLM quantization deep dive.

Rung 6: "How does speculative decoding speed things up, and when does it stop helping?"

What it probes: whether you understand the mechanism and, more importantly, its failure boundary, which is where most candidates who have read the abstract go quiet.

Speculative decoding attacks the memory-bound nature of decode. A small, cheap draft model proposes several tokens ahead; the large target model then verifies all of them in a single forward pass. Because that verification pass costs almost the same as generating one token (decode is bandwidth-bound, so a batch of candidate tokens moves the weights once), you get multiple tokens for roughly the price of one whenever the draft is accepted. The speedup is a function of the acceptance rate: the more often the small model guesses what the big model would have said, the more free tokens per pass.

The senior tell is the batch-size cliff. Speculative decoding is a low-concurrency latency optimization. At batch size one it can double or triple throughput, but the benefit erodes as concurrency rises, because a busy server is already compute-saturated by its batch, so the spare capacity speculative decoding was exploiting has been spent. At high enough batch sizes the extra verification work can make you slower than not speculating at all. This is why it is a single-stream latency trick, not a throughput trick, and why the honest question is always "at what batch size."

The trap: quoting a headline speedup with no batch size attached. A "2 to 3x faster" number is almost always a batch-size-one figure, which is a best case dressed as an expected case. The candidate who says "it helps most at low concurrency and can hurt at high concurrency, because it is trading spare compute for latency" has shown they understand the mechanism's boundary, not just its pitch. The details, including how the batch-size cliff plays out with EAGLE-style drafters, are in speculative decoding in vLLM.

Rung 7: "What breaks at very long context, or at very high scale?"

What it probes: whether you can reason about where a working system stops working, which is the question that most reveals real operational experience.

The strong answer names several failure surfaces and ties each to an earlier rung. Prefill goes quadratic: attention over the prompt is O(n^2), so a 200k-token prompt spends most of its wall-clock in prefill before a single output token appears, which is why TTFT explodes at long context. The KV cache grows linearly and can exceed the weights, so long-context serving is capacity-bound and forces the attention-variant and cache-quantization choices from rungs three and five. And at high concurrency with a warm cache, the bottleneck can move off the GPU entirely: the CPU-bound Python serving frontend (HTTP parsing, tokenization, detokenization, streaming) can become the ceiling, which is why the serving frameworks have been moving that work to Rust. The unifying point is that the bottleneck moves rather than vanishes, and a senior engineer names where it moves next.

The trap: treating "long context" as a single problem with a single fix. It is at least three problems (quadratic prefill, linear cache growth, and a frontend that saturates a core) with three different answers. The candidate who presents them as one connected system, each pressure traceable to a mechanism from an earlier rung, demonstrates exactly the understanding the whole ladder was built to find. The context side is covered in effective context length, and the local-serving economics in running LLMs locally.

How to actually prepare (the operational part)

Reading answers is not preparation. Inference questions expose whether you can reason from the hardware up, so rehearse the causal chains, not the vocabulary. A protocol that works:

1. Be able to state the prefill/decode split in one breath, with the bottleneck for each. Prefill is compute-bound, decode is memory-bandwidth-bound. If you cannot say why decode is bandwidth-bound (you move all the weights to make one token), rungs four through six have no foundation.
2. Draw the KV cache and write its size formula from memory. 2 * layers * kv_heads * head_dim * seq_len * batch. Every later technique (paging, GQA, cache quantization) is an edit to that formula, and if you can write it you can derive why each one helps.
3. For every optimization, name the bottleneck it relieves and the one it does not. Continuous batching relieves GPU idle time, not memory. Quantization relieves memory and bandwidth, not the tail. Speculative decoding relieves single-stream latency, not high-concurrency throughput.
4. Attach a batch size to every performance number you say out loud. "Faster" is not an answer. "Faster at batch size one, break-even by batch size 32" is. This one habit signals operational experience more than any framework name.
5. Prepare one honest "where it breaks." Name a failure boundary: the batch-size cliff for speculative decoding, the tail damage from 4-bit, quadratic prefill at long context. Interviewers trust the candidate who leads with the boundary over the one who claims everything is solved.

The unifying thread across all seven rungs: interviewers are not checking whether you can name the fastest serving framework. They are checking whether you understand where the time and memory go well enough to predict what a given trick will and will not do. Every rung has a tool-name answer that stalls and a mechanism answer that opens the next door. Prepare for the door.

Key Takeaways

1. Prefill and decode are the foundation, and they are opposites. Prefill is compute-bound (large parallel matmuls over the whole prompt); decode is memory-bandwidth-bound (move all the weights to emit one token). Almost every serving optimization targets the memory-bound decode phase, so an answer that flattens generation into one loop fails the first rung.
2. Latency and throughput are different, often opposed, objectives. Name the metrics: TTFT and TPOT for latency, aggregate tokens per second for throughput and cost. A single unqualified "speed" number invites the follow-up that exposes it. Ask which one you are serving before proposing anything.
3. The KV cache is the object the whole interview orbits. It makes decode fast and memory-bound at once, grows linearly with context and batch, and becomes the dominant memory consumer at scale. Its size formula is why the attention-variant choices exist.
4. Continuous batching is the biggest throughput lever. Step-level scheduling that admits and evicts sequences every decode step keeps the GPU saturated, unlike static batching that idles on the slowest sequence. Paged KV memory is its companion: you can only pack more sequences if their caches fit.
5. Quantization is nearly free on the mean and expensive on the tail. Separate weight, KV-cache, and activation quantization; each has distinct costs. Low-bit damage concentrates on rare tokens and long-context recall, which headline benchmarks miss, so validate on your own distribution. A 4-bit-native model is a floor with no higher-precision fallback.
6. Speculative decoding is a low-concurrency latency trick with a batch-size cliff. It trades spare compute for tokens, so it shines at batch size one and can hurt at high concurrency. Every quoted speedup needs a batch size attached, or it is a best case dressed as an expected case.
7. At scale the bottleneck moves rather than vanishes. Quadratic prefill blows up TTFT at long context, the KV cache can exceed the weights, and a warm high-concurrency server can become CPU-bound on its Python frontend. Naming where the bottleneck moves next is the mark of real operational experience.