Updated 10 October 2026
AI inference, from the ground up
What happens between sending a prompt and getting tokens back, and what it takes to run that in production. These are the notes I wrote while studying it, in the order I wish I had read them. Each chapter ends with the resources I used.
Chapters
- 01 What it costs to generate a token planned
- 02 Prefill and decode planned
- 03 Measuring it: TTFT, TPOT and throughput planned
- 04 The KV cache, with numbers planned
- 05 Batching: static, dynamic and continuous planned
- 06 PagedAttention and GPU memory planned
- 07 Quantization: what you lose and what you gain planned
- 08 Inference engines: vLLM, SGLang and friends planned
- 09 Serving on Kubernetes: GPUs, autoscaling and queues planned
Starting points
- Transformer Inference Arithmetic kipply
- Making Deep Learning Go Brrrr From First Principles Horace He
- Orca: A Distributed Serving System for Transformer-Based Generative Models Yu et al., OSDI 2022
- Efficient Memory Management for LLM Serving with PagedAttention Kwon et al., 2023
- vLLM documentation vLLM project