I'm a Platform Engineer / SRE based in Spain. I build and run cloud platforms, mostly on AWS and Kubernetes, and small tools for the problems I run into along the way.
Right now I'm studying how AI models are served in production: GPUs, batching, the KV cache and the infrastructure behind every API call. I'm writing down what I learn here, in the order I wish I had read it.
Writing
The first notes are on their way. Meanwhile, here is the plan:
- 01 What it costs to generate a token planned
- 02 Prefill and decode planned
- 03 Measuring it: TTFT, TPOT and throughput planned
- 04 The KV cache, with numbers planned
- 05 Batching: static, dynamic and continuous planned
- 06 PagedAttention and GPU memory planned
- 07 Quantization: what you lose and what you gain planned
- 08 Inference engines: vLLM, SGLang and friends planned
- 09 Serving on Kubernetes: GPUs, autoscaling and queues planned