LLM evaluation: grading a model with no right answer
Vibes do not scale and BLEU measures nothing you care about. Eval sets, LLM-as-judge without fooling yourself, and the biases that corrupt it.
Working with LLMs as a systems problem: getting reliable output, retrieving the right context, giving models tools, and knowing whether any of it actually works.
Calling models from Python, forcing structured output, choosing between fine-tuning and retrieval, and grading results.
Vibes do not scale and BLEU measures nothing you care about. Eval sets, LLM-as-judge without fooling yourself, and the biases that corrupt it.
Fine-tuning teaches behaviour, RAG supplies facts. A decision framework, the LoRA maths, real costs, and the cheaper ladder to climb first.
Asking a model to reply in JSON works until it does not. Learn why tool schemas beat prompt begging, and how to validate every response before it reaches your code.
Retrieval-augmented generation done properly — chunking, retrieval evaluation, reranking, and the vector index underneath.
Most RAG fails at retrieval, not generation. Build a golden set, measure recall@k, run chunking experiments, add a reranker when data says so.
ANN search trades accuracy for speed. How HNSW graphs and IVF-PQ quantisation work, which knobs move recall, and how to choose an index.
Retrieval-augmented generation in plain Python: split a PDF into chunks, find the relevant ones, and let Claude answer using only those. No vector database required.
The agent loop and its failure modes, plus designing tool interfaces and MCP servers a model can actually use.
An agent is a loop with tools and a stopping condition. Context growth, error handling, termination — and when to write a chain instead.
Tool calling fails at the interface, not the model. Schema design, descriptions, error contracts, and what MCP actually standardises.
Newsletter
Free essays on learning, Python, and ML — no account required. We’ll only email when there’s something worth reading.