ML System Design Checklist
A stage-by-stage checklist for designing ML and LLM systems, from problem framing to monitoring, with an LLM and RAG add-on.
- Type
- Checklist
- Access
- Free
- Format
- Published
- Nov 2, 2026
Hosted on Gumroad. It opens in a new tab.
What's inside#
- Checklists for eight stages: framing, data, features and training, evaluation, serving, monitoring, responsible ML, and an LLM and RAG add-on
- A table of evaluation metrics by task, with notes on when each misleads
- A comparison of batch, online, streaming and on-device serving
- A monitoring table covering drift, data quality and system health
- A one-page launch review with twelve questions
How to use it#
- Work through the stages in order when planning a new ML or LLM feature.
- Treat each unchecked box as a risk: fix it, or accept it on purpose with an owner.
- Use the one-page review in launch meetings.
- Pair it with the Technical Design Doc Template for the written proposal.
Limitations#
- It's a checklist, not a tutorial; it assumes basic ML knowledge.
- Regulatory and legal requirements vary by country and industry; check with the right people.
- It doesn't replace a security or privacy review for sensitive systems.
License
Free to use, share and adapt, including commercially, as long as you credit Letinix and link to this page. Licensed under CC BY 4.0.
Related reading
Does Size Matter? Bigger AI Isn’t Always Better
Does size matter? There. I asked it. Humanity has spent an unreasonable amount of time arguing about this question, so naturally we eventually gave the problem to artificial intelligence. Except now we measure it in billions of parameters .
Jev Is Not a Paradigm Shift. It’s ust a Fast Classifier.
I called Jev a glorified, high-speed classifier . Then I did the thing controversial posts usually skip: I tried to prove myself wrong. I went through TypeSafe’s launch material, LangChain’s harness and Jev-as-a-Judge experiments , Vercel’s
How to Budget Retrieved Context in a RAG Pipeline
A practical RAG token budget accounts for instructions, the query, retrieved chunks, formatting, and output allowance. Build the budget around your model and retrieval workload instead of copying a universal chunk size.
Related tools
More resources
Technical Design Doc Template
A free template for proposing a system change, showing the evidence behind it, and getting a clear decision from reviewers.
Tech Lead Toolkit
Five printable worksheets for tech leads: delegation levels, stakeholder map, decision brief, pre-mortem and one-on-one template.
System Design Cheat Sheet
A dense reference to system design concepts, estimation numbers, building blocks and trade-offs.