tiny-inference-clusterBuild a cluster-level LLM inference service in Go, from scratch

A streaming proxy, multi-replica load balancing and prefix-cache-aware routing, then onto Kubernetes. Experiments replay real agent traffic, and every chapter maps to interview questions.

Start with chapter 1 (Chinese)

Real agent traffic

Workloads come from public agent traces (FreeInference, SemiAnalysis/Weka, Mooncake), not random prompts.

Each chapter is a diagnosis

Predict first, watch a naive implementation fail, find the cause in per-request records, fix it, then check the fix on a workload you haven't seen.

Runs on a laptop

llm-d-inference-sim stands in for vLLM, so you don't need a GPU. Key experiments come with real-GPU runs on H200 and RTX Pro 5000.

Interview-ready

Each chapter ends with follow-up questions, troubleshooting exercises and design problems. The troubleshooting exercises use request records and replays from a misbehaving cluster.

Chapters

#	Topic	Status
1	Streaming proxy and cancellation (Chinese for now)	Experiment and replays ready; real-GPU runs in progress
2	Multi-replica load balancing	Planned
3	Prefix caching and queueing	Planned
4	Moving to Kubernetes (kind)	Planned

Work in progress. v0.1 will have four chapters, and the GitHub repository goes public with it. Chapters are written in Chinese first; English follows.