Start with what inference means and how it differs from model serving. Then learn how LLMs generate responses, use hardware, and handle real traffic. Eleven lessons take you from core concepts to intermediate engineering decisions.
01–03Understand the foundationsInference, serving, and how a response is generated.
04–06Understand performanceLatency, GPU memory, batching, and KV cache.
07–11Engineer a production serviceRouting, diagnosis, optimization, safety, and scaling.
LESSON 01 / 11
What is inference?
Use what a model has learned to produce an output.
Inference is the process of running a trained model on an input to produce an output. A fraud model can score a transaction. An image model can identify an object. A language model can generate a reply. Inference is broader than chat and does not require a web server.
During training, an algorithm adjusts the model’s learned numerical values, called weights, using examples. During ordinary inference, those weights stay fixed. The input and intermediate calculations change, but the model is not retrained for each request.
For a language model, a prompt supplies instructions and context. Adding a document to that prompt can help the model answer questions about it; this does not, by itself, update its weights. Fine-tuning is additional training that does update weights.
One concrete example
You load a trained model on a laptop and ask it to summarize a paragraph. The calculations that produce the summary are inference. You have performed inference even though no public API exists.
This guide focuses on large language models (LLMs) that generate text one token at a time. Later lessons explain how to make that work fast, reliable, and affordable for many users.
Try it: Check the distinction
Classify these actions: adjusting weights using examples, generating a summary, and adding a policy document to a prompt. Which changes what the model has learned?
LESSON 02 / 11
Inference and model serving
Inference is the computation. Serving makes it available to users.
Model serving is the system that receives requests, runs inference, and returns results. It loads the model, exposes an interface such as an API, schedules work, and handles failures. A production service also needs access controls, monitoring, and capacity management.
An inference engine, or runtime, executes the model efficiently on hardware. A serving system may wrap that engine with queues, routing, and an HTTP interface. Some software packages provide both, so the boundaries are not always visible.
Why are the terms used interchangeably? In production, users usually reach inference through a serving system. Teams may call the whole deployment an “inference service,” and vendors may use “inference platform” for tools that also handle serving. The terms overlap in everyday use, but describe different scopes.
Follow the boundary
A request waits in an API queue for two seconds, then the model generates a reply in one second. The user waits about three seconds, plus other overhead. The model computation took one second; the serving path added the rest.
Inference engineering spans both scopes: model execution and the system around it. The goal is to meet quality, latency, reliability, and cost requirements. Improving the model’s execution time will not fix every serving problem.
Try it: Map the system
Draw client → API → queue → inference engine → response. Mark where authentication, model computation, waiting, and streaming happen. Explain which boxes belong to serving.
LESSON 03 / 11
How an LLM generates a response
Understand tokens, prefill, decode, and context.
A tokenizer converts text into tokens: pieces of words, punctuation, or other symbols. Tokens are not the same as words, and counts depend on the tokenizer. A chat template arranges instructions and messages in the format the model expects.
During prefill, the model processes the prompt and builds attention state. During decode, it uses the current context to produce scores for the next token. A selection rule chooses a token, appends it to the sequence, and repeats until a stopping condition is reached.
Sampling controls how a token is chosen from those scores. Temperature and other sampling settings can affect the answer, its length, and reproducibility. Keep them fixed when comparing configurations.
The context window limits the tokens the model can consider. For the decoder models in this guide, budget for instructions, conversation history, retrieved documents, and generated output. A chat service usually supplies the relevant history again; the model does not remember every past conversation on its own.
Streaming returns generated output incrementally. It lets users see progress before the answer is complete. It does not remove the computation needed to produce the remaining tokens.
Try it: Inspect one request
Use your model’s tokenizer to count a short prompt and a prompt containing a document. Record generated tokens and the stop reason. Compare streamed and non-streamed responses.
LESSON 04 / 11
Measure latency and throughput
Separate how long one user waits from how much work the system completes.
Time to first token (TTFT)
Time from submitting a request until the first output token arrives. It includes waiting before generation begins.
Inter-token latency (ITL)
The gap between successive output tokens. Large gaps make streaming feel uneven.
End-to-end latency
Time from submission to the complete response.
Throughput
Completed work per second. State whether you count requests, input tokens, or output tokens.
Concurrency is the number of requests in progress at once. Arrival rate is how many new requests arrive per second. As arrival rate exceeds capacity, the queue grows and latency usually rises.
Report a distribution, not just an average. A p95 latency of two seconds means about 95% of measured requests took two seconds or less. Report failures and timeouts too; excluding them can make an overloaded system look healthy.
Compare the same prompt lengths, output lengths, sampling settings, and measurement boundary. A short answer and a long answer are different workloads. Measure at the client to capture the experience users receive.
Try it: Build a baseline
Run a fixed set of prompts at low load. Record TTFT, completion latency, output tokens, and errors. Then increase concurrent requests and explain which measures change.
LESSON 05 / 11
Understand GPU memory and compute
A model that loads may still lack room to serve requests.
GPUs perform many model operations in parallel. Performance depends on both compute capacity, which performs arithmetic, and memory bandwidth, which moves data. GPU utilization alone does not tell you which resource limits a workload.
Device memory holds model weights, attention state, and temporary execution data. Estimate weight bytes ≈ parameter count × bytes per parameter. An illustrative 6-billion-parameter model at 16 bits needs about 12 GB for weights alone. It needs more memory to run.
Precision describes how numerical values are represented. Quantization uses lower-precision representations to reduce storage and sometimes improve speed. Gains depend on hardware and kernel support. Check quality after changing precision; weight precision and attention-cache precision are separate choices.
Longer contexts and more active requests increase memory demand. Reserve space for runtime buffers and peak execution needs. Do not size a deployment using the model file size alone.
Prefill often uses large parallel matrix operations effectively. Decode at small batch sizes is often limited by moving weights and attention state. Treat these as starting hypotheses, then measure your model and workload.
Try it: Create a memory budget
Estimate weight storage, then measure device memory with the model loaded and during one request. Repeat with longer inputs and more concurrent requests. Record where the first limit appears.
LESSON 06 / 11
Use batching and KV cache
Share GPU work and avoid repeating attention calculations.
Batching groups work from multiple requests. It can improve total throughput by making better use of the GPU, but may increase the wait for an individual request. Continuous batching lets the scheduler add new requests as others finish.
The KV cache stores attention keys and values for tokens already processed. Decode reuses this state instead of recomputing it. It stores intermediate model data, not complete answers.
A useful cache estimate
For a standard full-attention decoder: KV bytes ≈ 2 × layers × KV heads × head dimension × cached tokens × bytes per element.
With 32 layers, 8 KV heads, a head dimension of 128, and 2-byte values, this is 128 KiB per token. An 8,192-token sequence needs about 1 GiB before overhead. Sum tokens across resident sequences, accounting for any shared blocks. Other attention layouts and sharding change the calculation.
Prefix caching reuses compatible KV state when requests begin with the same tokens. Paged allocation manages cache in blocks. Both help manage work or memory; neither makes long contexts free.
At intermediate level, tune the active-sequence limit and the tokens scheduled per step. Chunked prefill breaks long prompts into scheduling units so they can share execution time with decoding. Compare TTFT, ITL, cache occupancy, and preemptions before keeping a change.
Try it: Test one setting at a time
Use a mix of short and long prompts. Change the sequence limit, then the scheduled-token budget in separate runs. Identify which configuration improves throughput while preserving acceptable latency.
LESSON 07 / 11
Build a reliable request path
Control how requests enter, wait, and reach a worker.
A worker runs a model instance. A router chooses the worker for each request. Start with one worker; add routing when you need replicas, different models, or separate workload classes.
Admission control decides whether to accept more work. Validate context length and output limits, enforce per-user quotas, and bound the queue. A clear overload response is better than a request that waits indefinitely.
Request count alone is a weak estimate of load. A long document prompt can require more work than several short questions. Routing can consider queued tokens, available cache, and worker health. Cache affinity may save prefill work, but must be balanced against queueing delay.
Set request deadlines and propagate cancellation so abandoned requests release resources. Limit retries to eligible failures and keep them within the original time budget. Retrying a partially streamed response can duplicate text; tools that perform actions need separate safeguards against duplicate execution.
Try it: Test a failure
Stop a test worker while traffic is running. Check failure responses, queue limits, cancellation, and retries. Verify that the system recovers without sending unlimited replacement requests.
LESSON 08 / 11
Set targets and diagnose bottlenecks
Use evidence to connect user symptoms to system behavior.
A service-level objective (SLO) states the service target: which requests, what measurement, what threshold, and over which time window. For example, a team might require a chosen share of interactive requests to receive their first token within a defined time.
Goodput counts requests completed per second that meet the stated targets. High throughput with many late responses is not the same as high goodput. Evaluate answer quality separately unless it is explicitly part of the qualifying criteria.
Collect client latency and errors alongside queue age, active sequences, cache use, preemptions, and device memory. A trace connects events for one request. Metrics summarize many requests. Logs record discrete events. Use each for its purpose and avoid storing prompt contents by default.
Growing queues with stable model execution time suggest capacity pressure. ITL spikes during long prompts suggest scheduling interference. Cache exhaustion and repeated recomputation suggest memory pressure. These are hypotheses; validate them with traces and controlled tests.
Warm the service before steady-state tests and measure cold starts separately. Test both fixed concurrency and independently scheduled arrivals: a fixed-concurrency client reduces its sending rate when responses slow down and can hide overload.
Try it: Find the saturation point
Increase arrival rate in steps. Plot goodput, p95 or p99 TTFT, errors, and queue age. Identify the highest load that meets your SLO and repeat the test around that boundary.
LESSON 09 / 11
Optimize without losing quality
Compare changes at the same workload and service target.
Begin with a reproducible baseline: model revision, tokenizer, runtime, hardware, precision, sampling settings, and workload. Keep a representative quality test set. A faster answer is useful only if it still does the job.
Change one variable at a time. Try a smaller model, less irrelevant context, a useful output limit, different precision, or scheduler tuning. Record quality, latency, throughput, and memory before and after.
Cost per qualifying request is total serving cost divided by requests that meet your stated criteria. Include idle capacity and retries. For an API, count input, output, and any cache charges. For self-hosted service, include hardware, storage, networking, and operations.
More advanced methods need the same discipline. Speculative decoding proposes tokens for the target model to verify. Its value depends on how many proposals are accepted and the added cost of proposing and verifying them. It is not a universal speedup.
Try it: Keep an experiment record
Write a hypothesis, one configuration change, the workload, and the result. Keep the change only if it meets the same quality and latency requirements at a better measured cost or capacity.
LESSON 10 / 11
Protect and release the service
Treat isolation and recovery as engineering requirements.
Authenticate callers and authorize access in application code. Model instructions are not access controls. Limit request size, output length, concurrent work, and request rate for each tenant.
Keep cache reuse within the intended trust boundary. Pin model and adapter versions, review code required to load them, and restrict management endpoints. Treat retrieved text and model-generated tool arguments as untrusted input.
Validate tool actions before execution and keep credentials out of prompts. Define what logs may contain, who can read them, and when they expire. Test unauthorized access and resource exhaustion alongside normal requests.
A canary release sends a small share of traffic to a new version before wider rollout. Compare quality, latency, errors, and memory with the baseline. Keep a known-good version ready for rollback and drain active work when replacing workers.
Try it: Rehearse a release
Write the checks that allow a new version to receive more traffic, and the conditions that trigger rollback. Test rollback with requests in progress.
LESSON 11 / 11
Scale beyond one worker
Choose a scaling method for a measured constraint.
Replication adds copies of the model to serve independent requests. Use it when one worker meets latency goals but cannot handle the arrival rate. Replicas increase aggregate capacity; they do not automatically speed up a single request.
Tensor parallelism divides model operations across devices. Pipeline parallelism puts groups of layers on different devices. These can help fit larger models, but communication, synchronization, and uneven work can reduce efficiency. Expert parallelism distributes experts in mixture-of-experts models and adds token-routing traffic.
Some deployments separate prefill and decode into different worker pools. This allows independent capacity choices, but requires transferring KV state. Compare that transfer cost and queueing with a simpler deployment that keeps both phases together.
Autoscaling changes worker count as demand changes. Model loading, warmup, and an empty prefix cache delay useful capacity. Use demand and queue signals, retain headroom for expected bursts, and test worker loss.
You now have the pieces to reason about an inference system: define the workload, trace a request, budget memory, set service targets, measure the bottleneck, and test a change. Choose the simplest deployment that meets those requirements.
Try it: Design a small deployment
Choose a model, workload, SLO, and memory budget. Propose one-worker and scaled configurations. Explain what measurement would justify scaling, what could fail, and how you would recover.
PUT IT TO WORK
Put the pieces together.
By the end, you should be able to distinguish inference from serving, explain prefill and decode, estimate memory needs, read latency and throughput results, and justify a serving configuration. Put those skills into practice: build a small baseline, test one change, and explain the tradeoff.
