An FDE’s playbook for diagnosing TTFT delays

Most discussions about AI inference performance focus on model speed. In practice, however, users experience performance through a different metric: Time to First Token (TTFT).

TTFT is the elapsed time between a user request and receiving the first generated token from the model. A poor TTFT creates the perception of a slow or unresponsive system, regardless of the model’s eventual throughput.

The mistake many organizations make is treating TTFT as a model-serving problem. In reality, TTFT is a system problem. Delays can originate anywhere across the inference path — from the network and API gateway to scheduling, model loading, prompt prefill, and token streaming.

For field engineers and platform operators, TTFT should therefore be analyzed as an end-to-end latency budget rather than a single component metric.

TTFT degradation is rarely caused by a single component. More often, it is the cumulative effect of inefficiencies across multiple infrastructure layers.

A good mental model for TTFT would be to imagine a flow of data in the below sequence:

Network + Gateway + Scheduler + Model Load + Prefill + Streaming = TTFT

A user prompt travels via the network, lands at the gateway, gets scheduled for execution, is processed during the prefill phase where the KV cache is generated, and then produces the first token during decoding.

Typically to diagnose or troubleshoot a TTFT latency, 3 prominent areas to look:

• Queue time — Network + Gateway + scheduler
• Prefill time — Model load + Cache prefill

• Decode time — Time to first token

Before we look at the latencies that can arise in each of these phases, let us set a baseline understanding of what happens from the moment a prompt is generated to TTFT.

The diagram below shows the entire architecture that I am about to describe:

Let us take a look at how each of these layers can contribute to a TTFT latency:

Network

Potential contributors:

• Congestion
• Packet loss
• Buffer overflows
• PFC storms
• DNS delays
• Cross-region routing

Symptoms:

• High request latency before reaching serving infrastructure
• Inconsistent user experience across locations

Gateway

The inference gateway acts as the front door of the platform.

Responsibilities typically include:

• Authentication
• Authorization
• Request validation
• Rate limiting
• Queue management
• Routing

Potential TTFT impact:

• Queue buildup
• Excessive rate limiting
• Authentication bottlenecks
• Request routing inefficiencies

Scheduler

Once accepted, requests must be assigned to available compute resources.

Responsibilities:

• GPU allocation
• Worker selection
• Capacity management
• Load balancing

Potential TTFT impact:

• Insufficient GPU capacity
• Fragmented GPU resources
• Poor scheduling decisions
• Excessive queue wait times

In large-scale AI environments, scheduler latency can become a significant contributor during burst traffic periods.

Model Load (Optional Phase)

This phase occurs when the serving infrastructure must load model weights into GPU memory.

Activities include:

• Container startup
• Weight download
• GPU memory population
• Runtime initialization

Potential contributors:

• Cold starts
• Scale-out events
• New model deployment
• Evicted model reloads

For warm deployments this phase may be negligible. During cold starts it can dominate TTFT significantly.

Prefill

Prefill is often the least understood component of TTFT.

Before generating any output token, the model must process the entire input prompt.

Activities include:

• Tokenization
• Embedding generation
• Attention computation
• KV cache population

The longer the prompt, the larger the prefill workload.

In modern LLM systems, prefill frequently becomes one of the largest contributors to TTFT.

First Token Delivery

After prefill is complete, the model generates the first output token.

The generated token must then:

• Pass through serving infrastructure
• Be serialized
• Be streamed to the client

Only then does the user perceive a response.

A Real-World Example

A user submits a short prompt to a 70B model.

Network = 20 ms

Gateway = 100 ms

Scheduler = 500 ms

Model Load = 0 ms (warm)

Prefill = 1.2 s

Decode = 150 ms

TTFT = 1.97 s

In this scenario, optimizing model throughput would have little impact. The dominant contributors are scheduler delay and prefill latency.

The goal of the FDE is not to optimize TTFT directly, but to identify which stage has become the bottleneck and remove the constraint.

A Practical Troubleshooting Framework

Measuring TTFT

To effectively troubleshoot TTFT, teams should collect stage-level telemetry:

• Network latency
• Gateway latency
• Scheduler wait time
• Model load time
• Prefill time
• Decode time
• End-to-end TTFT

Without stage-level visibility, engineering teams often optimize the wrong component.

Conclusion

When users complain that “the model is slow,” the model is often innocent.

The bottleneck may be hiding in the network, gateway, scheduler, cold-start path, or prefill stage.

The most effective field engineers do not optimize models first. They optimize latency budgets.

Organizations that focus solely on model optimization often overlook the larger contributors to latency hidden within networking, gateways, scheduling, model lifecycle management, and prompt processing.

For field engineers, the most effective approach is to treat TTFT as a latency budget distributed across the entire inference path. Only by measuring and optimizing every stage can teams consistently deliver responsive AI experiences at scale.