Stage 8: Optimization and Hardware Acceleration¶

8 Make inference faster, cheaper, and more predictable.

Goal¶

Understand inference performance, model optimization, serving engines, distributed inference, GPU basics, edge deployment, and accelerator tradeoffs.

Roadmap to Master This Stage¶

Read the stage goal and diagram before opening the parts.
Move through the parts in order unless you can already pass the exit criteria.
Study each sub-part folder: overview, deep dive, and examples/practice.
Build the stage artifact in small slices and measure the listed metrics.
Use the part exam after each part, or open the global Exam tab to test across the roadmap.

Stage Structure Diagram¶

%%{init: {"flowchart": {"htmlLabels": true, "nodeSpacing": 70, "rankSpacing": 90}, "themeVariables": {"fontSize": "18px"}} }%%
flowchart LR
  P1["<b>8.1</b><br/>Inference Performance<br/>Model"]
  P2["<b>8.2</b><br/>Transformer Inference<br/>Internals"]
  P1 --> P2
  P3["<b>8.3</b><br/>Model Optimization"]
  P2 --> P3
  P4["<b>8.4</b><br/>Serving Engines"]
  P3 --> P4
  P5["<b>8.5</b><br/>Distributed Inference"]
  P4 --> P5
  P6["<b>8.6</b><br/>GPU and Kernel Basics"]
  P5 --> P6
  P7["<b>8.7</b><br/>Edge and Accelerator<br/>Co-Design"]
  P6 --> P7

Parts¶

Part	Simple explanation	Build focus
8.1 Inference Performance Model	Use the measurement vocabulary and workload model behind optimization.	Benchmark varied prompt, output, and batch shapes.
8.2 Transformer Inference Internals	Connect transformer computation to memory bandwidth, attention, cache layout, and sampling.	Trace one decode step for a small model.
8.3 Model Optimization	Change or approximate computation while measuring quality risk.	Compare baseline and optimized model variants.
8.4 Serving Engines	Understand how runtimes schedule, batch, stream, and manage model memory.	Serve a model or design a serving plan.
8.5 Distributed Inference	Scale inference across replicas or devices when one process is not enough.	Design a multi-GPU serving plan.
8.6 GPU and Kernel Basics	Build the hardware intuition needed to read profiles and understand bottlenecks.	Profile a simple GPU workload.
8.7 Edge and Accelerator Co-Design	Connect workloads to Jetson, mobile NPUs, FPGA prototypes, compilers, and future chips.	Create an edge deployment or accelerator workload contract.

Sub-Part Map¶

Part	Sub-part	Why it matters
8.1	8.1.1 Latency Throughput and Cost Metrics	Latency Throughput and Cost Metrics is the working skill inside Inference Performance Model that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.1	8.1.2 Prefill Decode and KV Cache	Prefill Decode and KV Cache is the working skill inside Inference Performance Model that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.1	8.1.3 Batch Shape and Concurrency	Batch Shape and Concurrency is the working skill inside Inference Performance Model that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.1	8.1.4 Benchmark Design	Benchmark Design is the working skill inside Inference Performance Model that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.2	8.2.1 Weight Memory and Activations	Weight Memory and Activations is the working skill inside Transformer Inference Internals that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.2	8.2.2 Attention and KV Cache Layout	Attention and KV Cache Layout is the working skill inside Transformer Inference Internals that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.2	8.2.3 GEMM GEMV and MLP Blocks	GEMM GEMV and MLP Blocks is the working skill inside Transformer Inference Internals that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.2	8.2.4 Sampling Hot Path	Sampling Hot Path is the working skill inside Transformer Inference Internals that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.3	8.3.1 Quantization Formats	Quantization Formats is the working skill inside Model Optimization that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.3	8.3.2 Calibration and Quality Checks	Calibration and Quality Checks is the working skill inside Model Optimization that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.3	8.3.3 Distillation and Pruning	Distillation and Pruning is the working skill inside Model Optimization that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.3	8.3.4 Speculative Decoding	Speculative Decoding is the working skill inside Model Optimization that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.4	8.4.1 vLLM SGLang TGI and TensorRT LLM	vLLM SGLang TGI and TensorRT LLM is the working skill inside Serving Engines that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.4	8.4.2 Continuous Batching	Continuous Batching is the working skill inside Serving Engines that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.4	8.4.3 Paged Attention	Paged Attention is the working skill inside Serving Engines that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.4	8.4.4 Streaming and Scheduler Policy	Streaming and Scheduler Policy is the working skill inside Serving Engines that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.5	8.5.1 Replica Parallelism	Replica Parallelism is the working skill inside Distributed Inference that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.5	8.5.2 Tensor Parallelism	Tensor Parallelism is the working skill inside Distributed Inference that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.5	8.5.3 Pipeline and Expert Parallelism	Pipeline and Expert Parallelism is the working skill inside Distributed Inference that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.6	8.6.1 CUDA Threads Blocks and Warps	CUDA Threads Blocks and Warps is the working skill inside GPU and Kernel Basics that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.6	8.6.2 Memory Hierarchy	Memory Hierarchy is the working skill inside GPU and Kernel Basics that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.6	8.6.3 Tensor Cores and Mixed Precision	Tensor Cores and Mixed Precision is the working skill inside GPU and Kernel Basics that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.6	8.6.4 Triton and Custom Kernels	Triton and Custom Kernels is the working skill inside GPU and Kernel Basics that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.7	8.7.1 Edge Runtime Targets	Edge Runtime Targets is the working skill inside Edge and Accelerator Co-Design that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.7	8.7.2 Power Thermal and Memory Budgets	Power Thermal and Memory Budgets is the working skill inside Edge and Accelerator Co-Design that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.7	8.7.3 ML Compilers and Graph Lowering	ML Compilers and Graph Lowering is the working skill inside Edge and Accelerator Co-Design that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.
8.7	8.7.4 FPGA ASIC and Dataflow Thinking	FPGA ASIC and Dataflow Thinking is the working skill inside Edge and Accelerator Co-Design that helps you build the stage artifact, An inference benchmark and optimization report for an open-weight or hosted model workload, while collecting enough evidence to trust the result.

Stage Artifact¶

An inference benchmark and optimization report for an open-weight or hosted model workload.

What to Measure¶

TTFT
TPOT
tokens per second
p95 latency
device memory
cost per 1000 requests
quality regression

Exit Criteria¶

explain prefill, decode, KV cache, batching, and memory math
apply and evaluate optimization
choose serving engines and targets
reason about CPU, GPU, NPU, FPGA, edge, and cloud

Previous: Stage 7: Model Infrastructure | Next: Stage 9: AI Security, Blockchain, and ZKML