AI Engineer Roadmap 2026: From LLM APIs to Production (Step-by-Step)

You do not need to be a Python expert to start, but AI engineering leans on a few patterns more than typical scripting.

What to be comfortable with

• Type hints: def embed(text: str) -> list[float]: — they make LLM client code and tool schemas readable and self-documenting
• Dataclasses / Pydantic: model request and response shapes; Pydantic is the de facto way to validate structured LLM outputs
• Virtual environments: python -m venv .venv and pip install, or uv for speed — isolate every project
• async/await: LLM calls are network-bound. Running 50 eval examples sequentially is slow; asyncio.gather runs them concurrently
• Generators / streaming: token streams from LLM APIs arrive incrementally — for chunk in stream:

Why it matters

Most AI engineering code is glue: call a model, validate its output, retry on failure, log the result. Clean Python with types and async turns a fragile demo into something you can ship and test.

Every AI app talks to model providers over HTTP with an API key. Leaking that key is the most common, most expensive beginner mistake.

Rules

• Never hardcode keys in source. Use environment variables: os.environ["OPENAI_API_KEY"]
• Never commit .env — add it to .gitignore. Use .env.example with blank values for documentation
• Rotate keys if one is ever exposed in a commit, screenshot, or log
• Set spend limits in the provider dashboard so a runaway loop cannot drain your budget

Reading responses

LLM APIs return JSON. You will parse fields like choices[0].message.content, usage.total_tokens, and stop_reason. Get comfortable inspecting JSON responses before building on top of them.

Chip Huyen's framing, which the rest of this roadmap follows: AI engineering is about building applications on top of foundation models that already exist, not training models from scratch.

The shift

The three layers of the AI stack

1. Application development — prompts, context, evaluation, the product. Where most AI engineers work.
2. Model development — training, finetuning, dataset engineering, inference optimization.
3. Infrastructure — serving, compute, monitoring.

Why this matters for your career

Because the model is pre-built, the differentiators are no longer 'can you train a model' but 'can you evaluate, ground, and ship one reliably.' That is why this roadmap spends entire sections on evaluation, RAG, agents, and guardrails.

Before transformers, sequence models were RNNs and LSTMs. Understanding why they were abandoned tells you what attention buys you.

The RNN problem

• RNNs process tokens one at a time, left to right. You cannot parallelize across the sequence, so training is slow.
• Information from early tokens has to survive many steps to influence later ones. In practice it fades — the vanishing gradient / long-range dependency problem.

What attention does

• Every token can look directly at every other token in one step. No information bottleneck.
• The whole sequence is processed in parallel, which is why transformers scale to huge datasets on GPUs.

This is the single architectural idea that made modern LLMs possible. Step 1.3 covers the mechanics.

Model size is reported in parameters (e.g. 8B, 70B, 405B). It is a rough proxy for capability, but the real engineering questions are about cost and latency.

Dense vs Mixture-of-Experts (MoE)

• Dense: every parameter is used for every token. A 70B dense model does 70B params of work per token.
• MoE: the model has many 'expert' sub-networks but only routes each token to a few. A model can have 100B+ total parameters but only activate ~10B per token, giving large-model quality at smaller-model inference cost.

The practical takeaway

• Bigger is not always better for your use case. A smaller model that you can evaluate, finetune, and serve cheaply often beats a frontier model you call blindly.
• Always separate total parameters (memory to hold the model) from active parameters (compute per token). MoE breaks the assumption that they are the same.

A freshly pretrained model only predicts the next token. It does not 'follow instructions' — post-training does that.

Two stages

1. Supervised finetuning (SFT): train the base model on curated (prompt, good-answer) pairs so it learns the instruction-following format.
2. Preference finetuning: teach the model which of two answers humans prefer.
   • RLHF (reinforcement learning from human feedback): train a reward model on human preferences, then optimize the LLM against it. Powerful but complex.
   • DPO (direct preference optimization): skip the separate reward model and optimize directly on preference pairs. Simpler and now widely used.

Why an AI engineer cares

You rarely run RLHF yourself, but it explains model behavior: why models hedge, refuse, or have a 'house style.' When you finetune (Step 8), you are usually doing SFT on your own data, sometimes followed by DPO.

An LLM outputs a probability distribution over the next token. Sampling decides how you pick from it. This is why the same prompt can give different answers.

The knobs

• Temperature: scales the distribution. 0 is (near) deterministic — always take the most likely token. Higher (0.7–1.0) increases randomness and creativity. For extraction and classification, use low temperature; for brainstorming, higher.
• Top-p (nucleus sampling): sample only from the smallest set of tokens whose probabilities sum to p (e.g. 0.9). Cuts off the long tail of unlikely tokens.
• Top-k: sample only from the k most likely tokens.

Test-time compute

Newer models 'think' before answering by generating reasoning tokens (chain-of-thought, multiple samples, self-consistency). Spending more compute at inference time can raise accuracy without changing the model — a key lever for hard tasks.

Engineering implication

Non-determinism is a feature, not a bug, but it makes systems hard to test. This is exactly why evaluation (Step 5) is built on distributions and judges, not exact string matches.

Beyond the prompt, a handful of request parameters control output. Know these cold.

• max_tokens: hard cap on output length. Too low truncates answers mid-sentence; it also bounds cost.
• temperature / top_p: randomness (see Step 2.4). Set one, not both.
• stop sequences: strings that halt generation. Useful to stop a model from running past a structured answer.
• stream: when true, tokens arrive incrementally. Essential for chat UX so users see output as it generates instead of waiting for the full response.

Streaming in practice

Streaming changes your code shape: you iterate over chunks and accumulate them, rather than awaiting one response object. It also complicates error handling and structured-output parsing, so add it once the basics work.

You are billed per token (input + output) and judged on latency. Both are engineering levers, not fixed costs.

Cost

• Count tokens before sending. Long system prompts and bloated RAG context are the usual budget killers.
• Pick the right tier: use a small/cheap model for routing, classification, and simple extraction; reserve frontier models for hard reasoning.
• Cache repeated calls and reuse prompt prefixes where the provider supports prompt caching.

Latency

• Time to first token (TTFT) dominates perceived speed in chat — streaming helps.
• Smaller models and shorter outputs are faster. So is batching offline workloads.

Step 10 (Inference Optimization) and Step 11 (caching, routing) go deep on this. Here, just build the habit of measuring tokens and latency on every call.

Calling the provider SDK directly from all over your codebase is a trap. Wrap it once.

What the wrapper handles

• Retries with backoff on rate limits and transient 5xx errors
• Timeouts so a hung request does not freeze your app
• Logging: prompt, model, tokens, latency, cost — the raw material for evaluation and observability
• Provider abstraction: swap OpenAI for Anthropic or a local model without touching business logic
• Structured-output parsing and validation in one place

Build vs use a library

Libraries like LiteLLM give you a unified interface for free. Building your own once (even a 50-line version) teaches you what these abstractions actually do — worth doing in your portfolio projects.

Most prompt 'tricks' reduce to a few principles from Chip Huyen's chapter.

1. Write clear, explicit instructions. Tell the model the role, the task, the format, and the constraints. Ambiguity is the #1 cause of bad outputs.
2. Provide sufficient context. The model only knows what is in the prompt. Give it the data, examples, and definitions it needs.
3. Break complex tasks into subtasks. One prompt that does five things badly becomes five prompts that each do one thing well (prompt chaining).
4. Give the model time to think. Ask for reasoning before the answer (chain-of-thought). For hard tasks, 'think step by step' measurably improves accuracy.
5. Use few-shot examples when format or style matters. Show, do not just tell.
6. Iterate and measure. Treat prompts as code — change one thing, evaluate against a fixed set of examples, keep what wins (this connects to Step 5).

A prompt buried in an f-string is unmaintainable. In production, prompts are assets you version, review, and test.

Practices

• Externalize prompts into files or a prompt registry, not inline strings scattered across the code
• Version them so you can roll back a regression and tie an output to the exact prompt that produced it
• Template variables explicitly (e.g. Jinja or simple {placeholder}) so the static instruction and the dynamic data are separate
• Test prompts against a golden set on every change — a prompt edit is a behavior change and deserves the same scrutiny as a code change

Tooling

Prompt management tools (LangSmith, Langfuse, PromptLayer, or a homegrown YAML registry) give you versioning, diffing, and linking prompts to eval runs. Start with files in git; add tooling when you have more than a handful of prompts.

Senior AI engineer JDs increasingly list 'AI safety and risk awareness: prompt injection, output filtering, data leakage.' Know the difference.

• Jailbreaking: a user tricks the model into ignoring its safety guidelines (e.g. 'pretend you are an AI with no rules').
• Prompt injection: untrusted content (a web page, an email, a retrieved document) contains instructions that hijack your app. This is the dangerous one for RAG and agents, because the model cannot reliably tell your instructions from instructions hidden in data.

Defenses (none is complete)

• Treat all retrieved/tool content as untrusted; never let it grant new permissions
• Separate instructions from data; use delimiters and structured inputs
• Filter and validate outputs before acting on them
• Apply least privilege to tools an agent can call
• Add a guardrail layer (Step 11.2) that scans inputs and outputs

There is no prompt that fully prevents injection — defense is architectural, which is why this reappears in the agents and architecture sections.

If you take one thing from this roadmap for your job search: AI engineers who can evaluate systems are rare and in demand. Almost every senior JD asks for eval pipeline experience.

Why it is hard

• No single ground truth. 'Summarize this' has many good answers. You cannot diff against one correct string.
• Open-ended outputs. Quality is multi-dimensional: relevance, faithfulness, tone, safety, format.
• Non-determinism. The same prompt gives different outputs, so a single run tells you little.
• Benchmark-reality gap. A model topping a public leaderboard can still fail on your data.

The mindset

Evaluation is not a final step — it is how you make every other decision: which model, which prompt, whether RAG helped, whether finetuning was worth it. Build the habit of asking 'how would I measure that?' before building anything.

These are the intrinsic metrics of language models. You will not use them to evaluate a chatbot's answer quality, but they explain training and model comparison.

• Entropy: how uncertain the model is about the next token, on average. Lower means more confident/predictable.
• Cross-entropy: the training loss — how surprised the model is by the true next token. Training minimizes this.
• Perplexity: exp(cross-entropy). Intuitively, the number of equally-likely choices the model is deciding between. A perplexity of 10 means it is as confused as if choosing uniformly among 10 tokens. Lower is better.

Where they help and where they do not

• Helpful: comparing base models, detecting whether finetuning is learning, spotting data issues.
• Not helpful: judging if an answer is correct or useful. A model can have low perplexity and still be wrong. For task quality you need the methods in 5.3–5.6.

Some tasks have checkable answers. Use exact methods there — they are cheap, fast, and objective.

Functional correctness

For code generation, run the code against tests. For SQL, execute it and compare results. This is the gold standard when available: it measures whether the output actually works.

Similarity to reference

When you have reference answers:

• Lexical (exact match, BLEU, ROUGE): compare overlapping words. Crude but useful for translation/summarization sanity checks.
• Semantic (embedding similarity): embed the output and the reference, measure cosine similarity. Captures meaning even when wording differs.

The limit

Most interesting AI tasks have no reference answer. That is when you reach for AI-as-a-judge (5.4) and comparative evaluation (5.5).

Use a strong LLM to score the outputs of your system. This is how teams evaluate open-ended tasks at scale, and it shows up by name in job descriptions.

How to use it

• Scoring: 'Rate this answer's faithfulness to the source from 1-5, and explain.'
• Pairwise: 'Which answer is better, A or B?' — often more reliable than absolute scores.
• Reference-guided: give the judge a rubric or a reference answer to compare against.

Limitations to manage

• Position bias: judges favor the first (or last) option — randomize order.
• Verbosity / self-preference bias: judges prefer longer answers, or answers from the same model family.
• It is still an LLM: it can be wrong. Validate the judge against human labels on a sample before trusting it.

What makes a good judge

Usually a strong, well-aligned model, given a clear rubric and asked to reason before scoring. Cheaper than humans, faster, and good enough to drive iteration — as long as you have spot-checked it.

When absolute scores are unreliable, rank by comparison. This is how public leaderboards like Chatbot Arena work.

The idea

Humans (or an LLM judge) see two anonymous outputs and pick the better one. Run thousands of these pairwise battles and compute an Elo rating (the same system used in chess) to rank models or system versions.

Why it works

• People are bad at assigning '7/10' consistently but good at 'A is better than B.'
• It is robust to scale drift and judge calibration.

How you use it

For your own systems: when you ship a new prompt or model, run it head-to-head against the current version on a fixed set of inputs and see which wins more often. This 'A/B by comparison' is more sensitive to small improvements than absolute scoring.

Chip Huyen's recipe for evaluating a system, not just a model. This is what you build in the Step 5 project.

Step 1: Evaluate every component

A RAG system has retrieval and generation. An agent has planning and tool calls and final output. Evaluate each part — a great generator cannot fix bad retrieval.

Step 2: Create an evaluation guideline

Define, in writing, what 'good' means for your task: the criteria (faithfulness, relevance, safety), what counts as a pass, and edge cases. Without a guideline, scores are noise.

Step 3: Define methods and data

• Pick the method per criterion: exact match, embedding similarity, or LLM-as-judge.
• Build a golden dataset: a fixed, representative set of inputs with known-good expectations. This is your regression suite — run it on every change.

The payoff

Once this exists, every decision (new model, new prompt, RAG vs finetune) becomes a measurement instead of a guess.

Choosing a model is an engineering decision with cost, latency, privacy, and quality trade-offs.

Build vs buy

• Buy (API models): fastest to start, best frontier quality, no infra. Costs scale per token, and your data leaves your perimeter.
• Build (open models you host): control, privacy, predictable cost at scale, finetunable. Requires serving infra and ops (Step 10).

Reading public benchmarks without being fooled

• Contamination: benchmark data may be in training sets, inflating scores.
• It is not your task: MMLU rank says little about your customer-support RAG quality.
• Leaderboard gaming: optimize a number, lose the plot.

The honest workflow

Shortlist 2-3 candidate models from benchmarks and price, then run your own eval pipeline (5.6) on your golden dataset. The model that wins on your data wins, full stop.

RAG = retrieve relevant documents, then put them in the prompt so the model answers grounded in your data.

Problems it solves

• Stale knowledge: a model's training has a cutoff. RAG injects fresh information at query time.
• Private data: the model never saw your internal docs. RAG gives it access without training on them.
• Hallucination: grounding answers in retrieved sources (with citations) makes outputs verifiable and reduces made-up facts.
• Cost: cheaper and faster than finetuning when the need is 'know about these documents,' not 'behave differently.'

The mental model

Think open-book exam. The model is smart but does not have your textbook memorized. RAG hands it the right page before it answers. Step 8 contrasts this with finetuning (changing how the model behaves rather than what it knows).

Retrieval works by turning text into vectors and finding the closest vectors to your query.

Embeddings

An embedding model maps text to a fixed-length vector (e.g. 1536 numbers) such that similar meanings land near each other in vector space. 'How do I reset my password' and 'forgot login credentials' end up close, even with no shared words.

Vector search

• Embed every document chunk once, store the vectors in a vector database.
• At query time, embed the question and find the nearest neighbors by cosine similarity.
• Exact nearest-neighbor is slow at scale, so vector DBs use approximate nearest neighbor (ANN) indexes (HNSW, IVF) that trade a little recall for big speed gains.

What to know

Pick an embedding model (its dimension and max input length matter), pick a vector store (pgvector, Pinecone, Qdrant, Chroma, Weaviate), and understand that retrieval quality is capped by embedding quality — garbage embeddings, garbage retrieval.

Every RAG system has the same skeleton. Build it once and you understand 90% of them.

Indexing (offline, done once)

1. Load documents (PDFs, web pages, tickets).
2. Chunk them into passages (Step 6.5 covers strategy).
3. Embed each chunk.
4. Store vectors + metadata in a vector database.

Retrieval + generation (online, per query)

5. Embed the query and retrieve the top-k most similar chunks.
6. Construct the prompt: system instructions + retrieved context + the question.
7. Generate the answer, ideally with citations to the retrieved chunks.

The two failure points

• Retrieval failure: the right chunk was not retrieved — the model never had a chance.
• Generation failure: the right chunk was retrieved but the model ignored or misused it.

Evaluating these separately (Step 6.6) is the whole game.

Vector search is not always best. Production retrieval usually combines methods.

The three approaches

• Keyword search (BM25): classic lexical match. Great for exact terms, codes, names, acronyms — where embeddings often blur meaning.
• Vector search: semantic match. Great for paraphrases and concepts, weak on exact identifiers.
• Hybrid search: run both and fuse the results (e.g. reciprocal rank fusion). Usually beats either alone.

Rerankers

Retrieval is recall-oriented: get candidates fast. A reranker (a cross-encoder model) then scores each query-document pair precisely and reorders the top candidates. Two-stage retrieve-then-rerank is the standard high-quality pattern: cheap broad retrieval, then expensive precise ranking on a small set.

Engineering takeaway

Start with hybrid + a reranker as your default. Measure (6.6) before adding complexity.

Most 'the model is dumb' complaints are really retrieval problems. These levers fix more than swapping the LLM.

Chunking

• Too big: irrelevant text dilutes the context and wastes tokens.
• Too small: you lose the surrounding meaning.
• Strategies: fixed-size with overlap, sentence/paragraph-aware, semantic chunking, and structure-aware (split by headings/sections). Match chunking to your document type.

Metadata filtering

Store metadata (date, source, author, doc type) and filter before or during retrieval. 'Only search 2026 policy docs' avoids retrieving correct-but-irrelevant chunks.

Query rewriting / expansion

User queries are messy. Rewrite or expand them (e.g. HyDE, multi-query) before retrieval to better match how documents are phrased.

Context construction

How you order and format retrieved chunks in the prompt matters — models attend more to the start and end of context ('lost in the middle'). Put the most relevant chunks where the model will use them.

You cannot improve what you do not measure, and RAG has two things to measure.

Retrieval metrics

Given a query, did you fetch the right chunks?

• Recall@k: of all relevant chunks, how many were in the top k? (Did the answer-bearing chunk make it in at all?)
• Precision@k / MRR: how high up were the relevant chunks?

Generation / end-to-end metrics (usually LLM-as-judge)

• Faithfulness / groundedness: does the answer only use the retrieved context, or does it hallucinate?
• Answer relevance: does it actually address the question?
• Context relevance: were the retrieved chunks on-topic?

How to use it

Frameworks like Ragas compute these for you. Build a golden set of (question, ideal answer, relevant docs). When quality drops, the metrics tell you which stage broke — retrieval or generation — so you fix the right thing.

An agent is an LLM that runs in a loop: it reasons, takes an action (calls a tool), observes the result, and repeats until the task is done. The model, not your code, decides the next step.

Agent vs workflow

• Workflow: you hardcode the steps (retrieve, then summarize, then format). Predictable, testable, cheaper.
• Agent: the model chooses steps dynamically. Flexible, but slower, costlier, and harder to control.

The discipline JDs reward

The GitLab AI engineer JD explicitly values saying 'this does not need AI' — and the same applies to agents: 'this does not need an agent.' Reach for an agent only when the task genuinely needs dynamic, multi-step decision-making with tools. For everything else, a fixed workflow is more reliable. Knowing the difference is a senior signal.

The loop

Reason → Act (tool call) → Observe → Reason → ... → Answer. Everything in this section (tools, planning, memory, failure modes) makes that loop reliable.

A tool is a function the model can choose to call. Tools turn a text generator into something that can search, compute, query a database, or hit an API.

How it works

1. You describe each tool to the model: name, description, and a JSON schema for its arguments.
2. The model, instead of answering, returns a structured request to call a tool with specific arguments.
3. Your code executes the function and returns the result to the model.
4. The model uses the result to continue or answer.

What makes tools work well

• Clear descriptions: the model picks tools based on your descriptions — vague ones cause wrong calls.
• Tight schemas: constrain argument types so the model cannot produce invalid calls.
• Validate before executing: never run a tool call blindly — it may carry injected or malformed input.

Standards

The Model Context Protocol (MCP) is an emerging open standard for exposing tools/data to any model, so you build an integration once and reuse it across apps.

Hard tasks need more than one model call. Planning patterns structure multi-step work.

ReAct (Reason + Act)

The model interleaves thoughts ('I need the user's order history') with actions (call the orders tool) and observations (the returned data). This explicit reasoning trace makes agent behavior debuggable and more reliable than acting blindly.

Task decomposition

Break a big goal into sub-tasks, solve each, then combine. Sometimes the model plans all steps upfront; sometimes it plans one step at a time based on what it observes.

Reflection / self-critique

After producing a result, the model critiques its own output and revises. Catches errors a single pass misses, at the cost of more calls.

Engineering reality

More steps = more cost, more latency, and more places to fail. Add planning capability only when simpler approaches fail your evals, and cap the number of steps so an agent cannot loop forever.

Complex tasks are often split across specialized agents, a pattern named explicitly in senior AI engineer job descriptions (e.g. ShopFully's 'planner, retriever, executor, reviewer').

Why split into roles

One agent juggling everything degrades. Specialized agents with focused prompts and tools are more reliable:

• Planner: decomposes the goal into steps
• Retriever: gathers the information each step needs
• Executor: performs actions / tool calls
• Reviewer / critic: checks the output, catches errors, can send work back

Orchestration patterns

• Supervisor: a lead agent routes work to sub-agents and assembles results.
• Sequential / graph: agents arranged in a pipeline or state graph (LangGraph models this well).

Human-in-the-loop

For high-stakes actions (sending money, deleting data, posting publicly), insert a human checkpoint before the executor commits. This is non-negotiable in regulated domains like finance and is called out in real JDs.

Cost warning

Every agent is more LLM calls. Multi-agent is powerful but expensive — justify it with evals, not vibes.

Models are stateless — each call only knows what is in its context window. Memory is how you create continuity.

Short-term (working) memory

The conversation so far, kept in the context window. The challenge is the window is finite, so you:

• Truncate old turns, or
• Summarize earlier conversation into a running summary, or
• Retrieve only the relevant past turns (RAG over the conversation).

Long-term memory

Persist facts across sessions (user preferences, past decisions, learned outcomes) in a database or vector store, and retrieve them when relevant. This is what makes an assistant feel like it 'remembers you.'

State management

Beyond text, agents track structured state: which steps are done, intermediate results, tool outputs. Frameworks like LangGraph make this state explicit and persistable, so an agent can pause, resume, and recover from failures.

The recurring tension

More memory in context = better continuity but higher cost, higher latency, and more room for the model to get distracted. Memory design is a retrieval problem in disguise.

Demos are easy; reliable agents are hard. The gap is failure handling and evaluation, which is exactly what separates a portfolio toy from a hireable skill.

Common failure modes

• Wrong tool / wrong arguments: the model calls the wrong function or fills bad parameters.
• Infinite loops: the agent repeats the same failing action — always cap steps.
• Compounding errors: a small mistake in step 1 derails every later step.
• Hallucinated tool results: the model invents data instead of using the tool's actual output.
• Prompt injection via tool output: untrusted retrieved/tool content hijacks the agent (see Step 4.4).

Guardrails and recovery

• Verification and repair: check each step's output; on failure, retry or escalate.
• Validation: schema-check tool calls before executing; sanity-check final outputs.
• Human-in-the-loop for irreversible actions.

Evaluating agents

Evaluate the trajectory, not just the final answer: did it pick the right tools, in a sensible order, without wasted steps? Build a set of tasks with known good trajectories and measure success rate, step count, and cost. This is the agent equivalent of the eval pipeline from Step 5.

Finetuning is often the first thing beginners reach for and usually the wrong first move. Chip Huyen's guidance: exhaust cheaper options first.

The ladder (cheap to expensive)

1. Prompt engineering: try this first. Fast, free, no infra.
2. RAG: when the problem is 'the model lacks knowledge.' Add the right context.
3. Finetuning: when the problem is 'the model lacks a behavior, format, or style' that prompting cannot reliably produce.

Finetune when

• You need a consistent output format/style prompting cannot enforce
• You want a smaller, cheaper model to match a bigger one on a narrow task
• You have a domain the base model handles poorly, and you have quality training data

Do NOT finetune when

• The real need is fresh or private knowledge — use RAG (finetuning bakes facts in poorly and they go stale)
• You lack a good dataset (Step 9) — finetuning on bad data makes things worse
• You have not yet built evals to prove it helped

Key point

RAG and finetuning are complementary, not either/or: finetune for behavior, RAG for knowledge.

Whether a finetune fits on your GPU comes down to arithmetic. Understanding it tells you why LoRA and quantization exist.

What consumes GPU memory during training

• Model weights: parameters x bytes-per-parameter
• Gradients: roughly same size as the weights
• Optimizer states: Adam stores 2 extra values per parameter — often the biggest consumer
• Activations: intermediate values kept for backpropagation; grow with batch size and sequence length

The rough rule

Full finetuning in 16-bit needs on the order of ~16+ bytes per parameter across weights, gradients, and optimizer states. A 7B model can need well over 100 GB — far beyond a single consumer GPU.

Why this matters

This math is exactly why you almost never do full finetuning. Parameter-efficient finetuning (Step 8.4) trains a tiny fraction of parameters, and quantization (8.3) shrinks bytes-per-parameter. Together they bring finetuning a large model down to a single GPU.

Quantization stores model weights in fewer bits, shrinking memory and speeding inference with usually small quality loss.

The idea

Weights are normally 16-bit or 32-bit floats. Quantization represents them in 8-bit or 4-bit integers. A 4-bit model is ~4x smaller than 16-bit, so it fits on smaller GPUs and runs faster.

Where it is used

• For serving: run a big model on cheaper hardware (Step 10).
• For finetuning: QLoRA (8.4) quantizes the frozen base model to 4-bit so you only need memory for the small trainable adapter.

The trade-off

Fewer bits = some precision loss. Modern methods (GPTQ, AWQ, bitsandbytes NF4) keep quality loss small for most tasks, but you must evaluate the quantized model on your task (Step 5) — never assume it is free.

Connection to numerical formats

This is the same precision idea as the memory math in 8.2: bytes-per-parameter is a dial you can turn.

A complete, evaluable finetune — the kind worth putting in a portfolio — follows a fixed loop.

The workflow

1. Define the task and metric first. What behavior are you adding, and how will you measure success? Build the eval before you train.
2. Prepare data (Step 9): a clean, formatted dataset of (input, desired output) pairs.
3. Baseline. Measure the base model on your eval. This is the bar to beat.
4. Finetune with LoRA/QLoRA (small, cheap, single-GPU) using a library like TRL.
5. Evaluate the finetuned model on the same eval set.
6. Compare. Did it actually beat the baseline? By how much? At what cost?

The lesson

Most of the value is in steps 1, 3, and 5 — the evaluation. A finetune you cannot prove improved anything is not an achievement. Showing a measured before/after is what makes it interview-worthy.

For both finetuning and evaluation, your dataset caps your ceiling. Chip Huyen frames curation along three axes.

Quality

Clean, correct, consistent examples. A few hundred high-quality examples often beat tens of thousands of noisy ones for finetuning. Bad labels teach bad behavior.

Coverage

Does the data span the real distribution of inputs you will see — including edge cases, rare categories, and hard examples? Gaps in coverage become blind spots in the model.

Quantity

How much you need depends on the method: parameter-efficient finetuning can work with hundreds to thousands of examples; the right answer is empirical — add data, re-evaluate, stop when the curve flattens.

The connection

This is why dataset engineering sits next to finetuning and evaluation. Your golden eval set (Step 5) and your finetuning set (Step 8) are both products of this discipline.

Where do training and eval examples come from? Usually a mix of sources, all needing labels you can trust.

Acquisition

• Existing logs: real production inputs are the most representative data you have. Mine them (with privacy care).
• Public datasets: a starting point, but rarely match your exact task.
• Manual creation: write examples by hand for narrow, high-stakes tasks.

Annotation

• Write a clear annotation guideline (same discipline as the eval guideline in 5.6) so labelers agree on what 'correct' means.
• Measure inter-annotator agreement — if humans disagree, the model has no chance and your eval is noise.
• Annotation is expensive, which motivates synthesis (9.3).

Privacy and rights

Real user data carries PII and usage-rights obligations. Anonymize, get consent where required, and respect data licenses — especially in regulated domains.

Hand-labeling does not scale. Synthetic data — often generated by a strong LLM — is how teams build training and eval sets quickly.

AI-powered data synthesis

Use a capable model to generate examples: paraphrase existing inputs, create edge cases, generate (input, output) pairs for a task. Powerful, but you must filter — synthetic data can be repetitive, biased, or wrong. Always validate a sample by hand.

Model distillation

Use a large, expensive 'teacher' model to generate high-quality outputs, then finetune a small 'student' model to imitate them. The student approaches teacher quality on the target task at a fraction of the serving cost. This is a common, practical reason to finetune (Step 8).

Watch out for

• Distribution collapse: synthetic data drifting away from real inputs
• Amplified bias: the generator's flaws baked into your dataset
• Licensing: some model terms restrict using outputs to train competing models

Synthesis is a force multiplier, not a free lunch — pair it with the curation checks in 9.1.

Raw data is never training-ready. The processing pipeline is unglamorous and decisive.

The steps

1. Inspect: actually read samples. Plot length distributions, label balance, and look for garbage. You cannot fix what you have not seen.
2. Deduplicate: duplicate examples bias the model toward repeated content and inflate eval scores (train/test leakage). Use exact and near-duplicate (embedding/MinHash) dedup.
3. Clean: fix encoding issues, strip boilerplate/HTML, remove broken records.
4. Filter: drop low-quality, off-distribution, toxic, or PII-laden examples by rules or a classifier model.
5. Format: convert to the exact structure the training tool expects (chat templates, prompt/completion fields, special tokens).

Why it is in this roadmap

The CS336 'Data' assignment is literally turning raw Common Crawl into usable pretraining data — because at every scale, data processing is where quality is won or lost. The same care applies to your finetuning and eval datasets.

To optimize serving you must measure the right things. These are the metrics teams track.

Latency

• Time to first token (TTFT): how long until the user sees anything. Dominates perceived responsiveness in chat.
• Time per output token / inter-token latency: how fast tokens stream after the first.
• Total latency: end-to-end for the full response.

Throughput

• Tokens per second and requests per second the system can sustain. This drives cost-efficiency at scale.

Cost

• Cost per token (or per request). For self-hosted models, this ties back to GPU utilization and throughput.

The tension

Latency and throughput trade off: batching more requests together raises throughput (cheaper per token) but can raise individual latency. The right balance depends on whether you are serving an interactive chat (optimize latency) or a batch pipeline (optimize throughput/cost).

Knowing the bottleneck tells you which optimization will actually help.

The two regimes

• Memory-bandwidth bound: the GPU spends most time moving weights, not computing. LLM decoding (generating one token at a time) is usually memory-bound — you read the whole model from memory per token. This is why quantization (smaller weights = less to move) speeds up decoding.
• Compute bound: the GPU is busy doing math. Prefill (processing the input prompt in parallel) is more compute-bound.

Why it matters

• If decoding is memory-bound, throwing more raw compute at it does nothing — you need to move less data (quantization) or move it once for many requests (batching, KV cache).
• AI accelerators (GPUs, TPUs) are characterized by both compute (FLOPS) and memory bandwidth; the cheaper card is often memory-starved.

The MIT 6.5940 course goes deep on efficient inference and the hardware-software interface if you want the full picture.

These shrink or speed up the model itself, before any serving tricks.

Quantization

Covered in 8.3: fewer bits per weight means less memory to move, which directly speeds up memory-bound decoding. The most impactful single lever for self-hosted inference.

Distillation

Covered in 9.3: a small student model trained to imitate a large teacher serves far cheaper. Best when you have a narrow task.

Speculative decoding

A small, fast 'draft' model proposes several tokens; the big model verifies them in one pass. When the draft is right (often), you get multiple tokens for roughly the cost of one big-model step — same outputs, lower latency.

Pruning and sparsity

Remove weights that contribute little. Effective but more involved; covered in efficient-ML courses like MIT 6.5940.

The rule

Every optimization can change outputs. Re-run your eval (Step 5) after applying one — speed is worthless if quality silently drops.

These optimize how you serve a fixed model — the biggest wins for self-hosted deployments.

KV cache

When generating token by token, the model would normally recompute attention over all previous tokens every step. The key-value (KV) cache stores those intermediate values so each new token only computes against the cache. Essential — without it, generation is quadratically slow. The trade-off: the KV cache consumes GPU memory that grows with context length and concurrency.

Continuous batching

Naive batching waits for a full batch and for all requests to finish together. Continuous (in-flight) batching swaps finished requests out and new ones in at each step, keeping the GPU saturated. This is the core trick behind high-throughput servers like vLLM and is often a 2-20x throughput win over naive serving.

PagedAttention

vLLM's technique for managing KV cache memory like virtual memory pages, reducing waste and allowing more concurrent requests.

Takeaway

You rarely implement these yourself — you use a serving framework (vLLM, TGI, TensorRT-LLM) that does. Knowing what they do lets you configure and debug them.

Chip Huyen presents production architecture as a series of layers you add to a bare model call, each solving a real problem. You do not build it all at once — you add the next layer when you need it.

The progression

1. Start: a single model API call.
2. Enhance context (Step 6): add RAG / retrieval so the model has the right information.
3. Add guardrails (11.2): validate inputs and outputs for safety and correctness.
4. Add a model router and gateway (11.3): route requests to the right model; centralize keys, rate limits, and logging.
5. Add caching (11.4): cut cost and latency on repeated work.
6. Add agent patterns (Step 7): when the task needs dynamic multi-step actions.
7. Add monitoring and orchestration (11.5): observe, evaluate in production, and coordinate pipelines.

Why this framing helps

It turns 'how do I build a production AI app' from an overwhelming question into an ordered checklist. Each layer is independently testable and maps to a section of this roadmap.

Guardrails are the validation layer around the model. They are explicitly named in AI engineer JDs alongside prompt injection and data leakage.

Input guardrails

• Detect and block prompt injection and jailbreak attempts (Step 4.4)
• Strip or block PII before it reaches the model or logs
• Enforce topic/scope limits ('this assistant only answers billing questions')

Output guardrails

• Schema validation: ensure structured outputs match the expected format before downstream code uses them
• Safety filters: block toxic, unsafe, or policy-violating content
• Data-leak checks: ensure the model is not echoing secrets, other users' data, or system-prompt contents
• Groundedness checks: in RAG, verify the answer is supported by retrieved context

How they fit

Guardrails run before and after the model, can reject or rewrite, and should fail safe. They never fully replace good prompting and architecture — defense in depth. In regulated domains (finance, health) they are a hard requirement, not a nice-to-have.

As soon as you use more than one model, you need a layer that decides which model handles a request and centralizes how you call all of them.

Model router

Route each request to the most appropriate model:

• By cost/capability: cheap small model for easy requests, frontier model for hard ones (often the small model or a classifier decides).
• By task: a code request goes to a code-strong model; a vision request to a multimodal one.
• Fallback: if the primary model errors or times out, retry on a backup. Improves reliability.

Gateway

A single entry point in front of all model providers that handles:

• API key management and provider abstraction
• Rate limiting and quotas
• Centralized logging, cost tracking, and caching
• Retries and timeouts

Why it matters

Routing can cut cost dramatically (most requests are easy and do not need a frontier model), and a gateway keeps provider-specific logic in one place. Tools like LiteLLM or a cloud AI gateway provide this off the shelf; the wrapper you built in Step 3.5 is the seed of it.

LLM calls are slow and expensive. Caching avoids paying twice for the same (or similar) work.

Types of caching

• Exact-match cache: identical request, return the stored response. Trivial and effective for repeated queries.
• Semantic cache: embed the query; if a previous query is similar enough (vector similarity above a threshold), reuse its answer. Catches paraphrases — but tune the threshold carefully or you serve wrong answers.
• Prompt caching (provider-side): reuse the processing of a long, stable prompt prefix (system prompt, large context) across calls. Providers like Anthropic and OpenAI support this and it can cut cost and TTFT substantially.

Cautions

• Cache invalidation: stale answers when underlying data changes (especially in RAG).
• Personalization: do not serve one user's cached answer to another if it contains private context.

When to add it

Add caching once you see repeated or similar traffic and have evals to confirm cached answers stay correct.

Traditional monitoring (latency, errors) is necessary but not sufficient for AI — you also have to watch quality, which can degrade silently.

Tracing

Capture the full path of each request: prompts, retrieved context, tool calls, intermediate steps, final output, tokens, latency, cost. For agents and RAG, a trace is the only way to debug why an answer was wrong (bad retrieval? bad tool call? bad generation?).

Evals in production

Run your eval methods (Step 5) on a sample of live traffic — LLM-as-judge on real outputs, groundedness checks on RAG answers. This catches quality regressions that uptime dashboards never would.

Drift and regressions

• Input drift: real queries shift away from what you tested on.
• Model drift: a provider updates a model and behavior changes under you.
• Regression alerts: tie production evals to alerts so quality drops page someone.

The loop

Observability feeds your golden dataset: real failures become new eval cases, which prevent the same failure twice. This closes the loop with user feedback (11.6).

Production users are your best (and cheapest) source of evaluation signal — if you design to capture it.

Explicit feedback

Thumbs up/down, star ratings, 'was this helpful?' Simple, but sparse — most users do not click. Make it low-friction and act on it.

Implicit / conversational feedback

Signals embedded in behavior: the user rephrased the question (the answer missed), copied the code (it worked), abandoned the session (it failed), or said 'no, I meant...'. Extracting these from conversation logs is a rich, underused source — Chip Huyen dedicates a section to it.

Feedback design

• Place feedback prompts where intent is clearest
• Avoid leading or annoying users
• Capture enough context (the trace) to make the feedback actionable

Limitations

Feedback is biased (loud minority, positivity/negativity skew) and noisy. Treat it as signal, not ground truth.

The payoff

Feedback + traces (11.5) become new examples for your golden dataset and finetuning set — the data flywheel that makes the product improve with use. This is the loop that ties the whole roadmap together.

Build these three, evaluate them, and put them on GitHub. They map directly to what AI engineer job descriptions ask for (RAG, agents, and evaluation appear in almost every senior posting).

1. Production RAG system

Chat over a real document set with hybrid search, a reranker, citations, and — the differentiator — a retrieval + answer evaluation showing recall and faithfulness numbers. (Project in this roadmap.)

2. LLM agent with tools

An agent that plans, calls real tools, handles failures, and is evaluated on its trajectory and failure modes, not just happy-path demos. (Project in this roadmap.)

3. LLM evaluation pipeline

A reusable eval harness: golden dataset, LLM-as-judge, and regression testing you can point at any prompt or model change. (Project in this roadmap.)

What makes them stand out

Most candidates ship demos. You ship evaluated systems with numbers, trade-off discussion, and guardrails. That is the exact gap between 'used an LLM once' and 'can build reliable AI products' that senior JDs screen for.

Your README is read before your code. For AI projects, it should prove you think like an engineer, not a demo-maker.

What to include

• Problem and approach: what it does and why you built it this way
• Architecture diagram: the pipeline (retrieval, model, guardrails, etc.)
• Evaluation results: the numbers — retrieval recall, faithfulness, agent success rate, before/after a change. This is what sets you apart.
• Trade-offs: what you chose and rejected (why RAG not finetuning, why this chunk size, why this model) — shows judgment
• Limitations and next steps: honesty signals seniority
• How to run it: reproducible setup

The principle

Anyone can wire an API call. Showing that you measured your system and reasoned about trade-offs is the signal hiring managers look for. Let the evals tell the story.

From the real job postings analyzed for this roadmap, here is what AI engineer hiring actually rewards.

Lead with what JDs ask for

• Evaluation: 'designed eval pipelines, golden datasets, LLM-as-judge.' This is the rarest and most-requested skill.
• RAG and agents: 'built a production RAG system'; 'built and evaluated agentic workflows with tool use.'
• Reliability and safety: 'added guardrails, handled prompt injection, monitored quality in production.'
• Systems thinking: several JDs (e.g. GitLab) value diagnosing whether AI is even the right tool before building.

Quantify outcomes

'Improved retrieval recall from 0.62 to 0.89'; 'cut answer hallucination rate by half via groundedness guardrails'; 'reduced cost 70% by routing easy queries to a small model.' Numbers from your projects beat adjectives.

Stack to name

Python is universal. Then: RAG, an LLM API (OpenAI/Anthropic), a vector DB, LangChain/LangGraph, an eval framework, and at least one cloud. Match your resume to the specific JD's stack.

AI engineering interviews blend classic software interviews with AI-specific system design and judgment questions.

System design (the core AI round)

Expect 'design a RAG system for X' or 'design an agent that does Y.' Practice talking through: data ingestion and chunking, retrieval choices, prompt and context construction, evaluation strategy, guardrails, cost/latency trade-offs, and monitoring. Mentioning evaluation unprompted is a strong signal.

AI fundamentals

Be ready to explain transformers and attention at a high level, what temperature does, why models hallucinate, when to use RAG vs finetuning, and how LLM-as-judge works and where it fails. (All covered in this roadmap.)

Practical / coding

Writing a prompt, parsing structured output, implementing a simple retrieval or eval loop, debugging an agent.

Judgment questions

'When would you NOT use an LLM / an agent?' Saying 'this does not need AI' when true is something JDs explicitly value.

Standard SWE rounds

Data structures, algorithms, and system design still apply — AI engineering is engineering first.