Stage 4: Large Language Models¶

4 Understand tokens, transformers, generation, adaptation, and evaluation.

Goal¶

Build a practical mental model of LLM behavior so you can choose models, control outputs, evaluate quality, and decide between prompting, RAG, tools, and fine-tuning.

Roadmap to Master This Stage¶

Read the stage goal and diagram before opening the parts.
Move through the parts in order unless you can already pass the exit criteria.
Study each sub-part folder: overview, deep dive, and examples/practice.
Build the stage artifact in small slices and measure the listed metrics.
Use the part exam after each part, or open the global Exam tab to test across the roadmap.

Stage Structure Diagram¶

%%{init: {"flowchart": {"htmlLabels": true, "nodeSpacing": 70, "rankSpacing": 90}, "themeVariables": {"fontSize": "18px"}} }%%
flowchart LR
  P1["<b>4.1</b><br/>Token and Context<br/>Mechanics"]
  P2["<b>4.2</b><br/>Transformer Mental Model"]
  P1 --> P2
  P3["<b>4.3</b><br/>Generation Controls"]
  P2 --> P3
  P4["<b>4.4</b><br/>Model Landscape"]
  P3 --> P4
  P5["<b>4.5</b><br/>Prompting and In-Context<br/>Learning"]
  P4 --> P5
  P6["<b>4.6</b><br/>Fine-Tuning and Dataset<br/>Engineering"]
  P5 --> P6
  P7["<b>4.7</b><br/>LLM Evaluation<br/>Methodology"]
  P6 --> P7

Parts¶

Part	Simple explanation	Build focus
4.1 Token and Context Mechanics	Understand how language becomes tokens and why context is a scarce engineering resource.	Create tokenization and context-budget demos.
4.2 Transformer Mental Model	Learn the architecture concepts that explain modern LLM behavior.	Annotate a transformer block and trace a simplified forward pass.
4.3 Generation Controls	Control probabilistic text generation and make outputs fit software contracts.	Compare repeated generations across decoding settings.
4.4 Model Landscape	Choose among model families, sizes, licenses, providers, and hosting patterns.	Create a private model leaderboard for one task.
4.5 Prompting and In-Context Learning	Use instructions, examples, constraints, and decomposition before heavier adaptation.	Build a prompt testing lab.
4.6 Fine-Tuning and Dataset Engineering	Know when model weights should change and how data quality drives the result.	Prepare an instruction dataset and PEFT plan.
4.7 LLM Evaluation Methodology	Evaluate open-ended model behavior with exact checks, rubrics, judges, and comparative tests.	Create a 30-case eval set.

Sub-Part Map¶

Part	Sub-part	Why it matters
4.1	4.1.1 Tokenization and Subwords	Tokenization and Subwords is the working skill inside Token and Context Mechanics that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.1	4.1.2 Context Windows and Truncation	Context Windows and Truncation is the working skill inside Token and Context Mechanics that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.1	4.1.3 Prompt Packing and Context Efficiency	Prompt Packing and Context Efficiency is the working skill inside Token and Context Mechanics that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.1	4.1.4 Token Based Pricing	Token Based Pricing is the working skill inside Token and Context Mechanics that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.2	4.2.1 Embeddings and Positional Information	Embeddings and Positional Information is the working skill inside Transformer Mental Model that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.2	4.2.2 Self Attention QKV	Self Attention QKV is the working skill inside Transformer Mental Model that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.2	4.2.3 MLP Blocks Residuals and Normalization	MLP Blocks Residuals and Normalization is the working skill inside Transformer Mental Model that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.2	4.2.4 Causal Masking	Causal Masking is the working skill inside Transformer Mental Model that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.3	4.3.1 Logits and Softmax	Logits and Softmax is the working skill inside Generation Controls that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.3	4.3.2 Temperature Top-p and Top-k	Temperature Top-p and Top-k is the working skill inside Generation Controls that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.3	4.3.3 Stop Sequences and Max Tokens	Stop Sequences and Max Tokens is the working skill inside Generation Controls that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.3	4.3.4 Structured Outputs and JSON Schemas	Structured Outputs and JSON Schemas is the working skill inside Generation Controls that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.4	4.4.1 Closed API and Open Weight Models	Closed API and Open Weight Models is the working skill inside Model Landscape that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.4	4.4.2 Base Instruct Reasoning and Multimodal Models	Base Instruct Reasoning and Multimodal Models is the working skill inside Model Landscape that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.4	4.4.3 Licenses and Data Policies	Licenses and Data Policies is the working skill inside Model Landscape that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.4	4.4.4 Build Buy Host or Route	Build Buy Host or Route is the working skill inside Model Landscape that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.5	4.5.1 Prompt Anatomy	Prompt Anatomy is the working skill inside Prompting and In-Context Learning that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.5	4.5.2 Zero Shot Few Shot and Examples	Zero Shot Few Shot and Examples is the working skill inside Prompting and In-Context Learning that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.5	4.5.3 Task Decomposition	Task Decomposition is the working skill inside Prompting and In-Context Learning that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.5	4.5.4 Prompt Versioning and Tests	Prompt Versioning and Tests is the working skill inside Prompting and In-Context Learning that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.6	4.6.1 When to Fine Tune	When to Fine Tune is the working skill inside Fine-Tuning and Dataset Engineering that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.6	4.6.2 Instruction and Preference Data	Instruction and Preference Data is the working skill inside Fine-Tuning and Dataset Engineering that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.6	4.6.3 PEFT LoRA and QLoRA	PEFT LoRA and QLoRA is the working skill inside Fine-Tuning and Dataset Engineering that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.6	4.6.4 Fine Tuning Evaluation	Fine Tuning Evaluation is the working skill inside Fine-Tuning and Dataset Engineering that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.7	4.7.1 Exact and Functional Evaluation	Exact and Functional Evaluation is the working skill inside LLM Evaluation Methodology that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.7	4.7.2 AI as Judge	AI as Judge is the working skill inside LLM Evaluation Methodology that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.
4.7	4.7.3 Comparative Evaluation	Comparative Evaluation is the working skill inside LLM Evaluation Methodology that helps you build the stage artifact, An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases, while collecting enough evidence to trust the result.

Stage Artifact¶

An LLM fundamentals notebook comparing models, tokenization, structured outputs, embeddings, costs, and failure cases.

What to Measure¶

token counts
latency
cost estimate
structured output validity
small task accuracy

Exit Criteria¶

explain tokenization, embeddings, attention, context, and sampling
choose models by constraints
use structured outputs
know when fine-tuning is premature

Previous: Stage 3: Deep Learning | Next: Stage 5: AI Applications