LLM and Generative AI Fundamentals

This guide covers the foundations needed to build modern LLM applications and explain design decisions in interviews.

Overview

Large language models are neural networks trained to predict the next token over massive text corpora. Generative AI systems use those models to produce text, code, structured outputs, and increasingly images, audio, and multimodal responses.

This matters because modern AI products depend on:

• prompt design
• model selection
• context management
• embeddings and retrieval
• evaluation and safety layers

Core Concepts

Tokenization

Tokenization converts text into model-readable pieces. In practice, tokenization determines:

• context window usage
• cost
• truncation behavior
• downstream chunking strategies

Prompt engineering

Prompt engineering is not just phrasing. It includes:

• task framing
• output constraints
• few-shot examples
• tool-use instructions
• failure-handling instructions

Embeddings

Embeddings map text or other inputs into dense vectors so semantically similar items land near each other. They power semantic search, clustering, retrieval, deduplication, and recommendation.

Context windows

The context window is the total amount of input and output tokens the model can process in one interaction. Good engineers plan for limited context instead of assuming everything can be stuffed into one prompt.

Key Skills

Designing effective prompts

In practice, this means writing prompts that are explicit about goal, constrained in format, robust to ambiguity, and testable across example cases.

Understanding context windows

This shows up as trimming noisy history, summarizing earlier turns, chunking documents correctly, and reserving output tokens.

Using embeddings for similarity

A strong engineer knows when embedding search is better than keyword match, and when hybrid search is better than either one alone.

Choosing models for tasks

This is a tradeoff problem across quality, cost, latency, context length, tool-use reliability, and multimodal support.

Tools

Projects

Chatbot

• Goal: Build a domain-specific conversational assistant with streaming responses.
• Key components: session memory, prompt templates, conversation summarization, evaluation prompts.
• Suggested tech stack: OpenAI or Anthropic API, FastAPI, Redis, React.
• Difficulty: Intermediate.

Text summarizer

• Goal: Summarize long reports while preserving key facts and action items.
• Key components: chunking, map-reduce summarization, citation or section references, quality rubric.
• Suggested tech stack: Python, LangChain or plain SDK, Pydantic.
• Difficulty: Intermediate.

Semantic search system

• Goal: Retrieve relevant documents using embeddings and vector search.
• Key components: embedding pipeline, vector index, query rewriting, reranking, relevance evaluation.
• Suggested tech stack: OpenAI embeddings, FAISS or Chroma, FastAPI.
• Difficulty: Intermediate.

Prompt experimentation repo

• Goal: Track prompts, outputs, and evaluations as a reproducible engineering asset.
• Key components: prompt versioning, fixtures, regression tests, side-by-side output comparison.
• Suggested tech stack: LangSmith or custom test harness, pytest, JSON fixtures.
• Difficulty: Advanced.

Example Code

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-5.4-mini",
    input=[
        {"role": "system", "content": "Answer as a concise ML interview coach."},
        {"role": "user", "content": "Explain precision vs recall with an example."},
    ],
)

print(response.output_text)

Suggested Project Structure

prompt-experiments/
├── prompts/
├── datasets/
├── evals/
├── scripts/
│   ├── run_eval.py
│   ├── compare_outputs.py
│   └── embed_corpus.py
├── results/
└── README.md

LLM Engineering in Practice (2026)

Knowing the theory isn't enough: interviews and real work test whether you can choose a model, budget tokens, control cost, and tune generation. This section is the practical layer.

Choosing a Model

Pick the smallest model that clears your quality bar, then scale up only where you must. A useful mental model across providers:

Other providers offer the same tiering (a frontier model, a balanced model, a small fast model). Always pull current model IDs, context windows, and pricing from the provider's own docs: they change often. For Claude specifics see the Anthropic & Claude guide.

Approximate Claude pricing (per 1M tokens, late 2026; check Anthropic's pricing page before quoting):

Context Windows & Token Budgeting

• Context window = the max tokens (input + output) the model can attend to in one request. Modern frontier models reach 1M tokens.
• Rough heuristic: ~1 token ≈ 4 characters ≈ 0.75 English words. Code and non-English text tokenize less efficiently.
• Don't estimate with the wrong tokenizer. OpenAI's tiktoken does not match Claude's tokenizer (and tokenizers change between model generations), so counts can be noticeably off. Use the provider's own token-counting endpoint.
• Budget the whole request: prompt + few-shot + retrieved context + reasoning + output ≤ window. Reserve headroom for the output (max_tokens) and for reasoning/thinking tokens.

Prompt Caching (the biggest cost lever)

If many requests share a large, stable prefix (system prompt, tool definitions, few-shot examples, a long document), cache it:

• Cache reads cost ~10% of normal input price; cache writes cost ~1.25×.
• It's a prefix match: any byte change anywhere in the prefix invalidates everything after it. So keep stable content first and put volatile content (the user's varying question, timestamps) last.
• Silent cache-killers: datetime.now() or a UUID in the system prompt, non-deterministic JSON key order, a tool set that varies per request.

Generation / Decoding Parameters

Note (2026): the newest reasoning models drop manual sampling params in favor of adaptive thinking + an effort control (low→max). On current Claude models such as Sonnet 5 and Opus 4.7 and later, a request that sets temperature, top_p or top_k is rejected with a 400; older models and most open-weight models still accept them. The model decides how much to reason and you set how hard it should try. Steer behavior with prompting and effort, not temperature=0.

Estimating Cost

cost ≈ (input_tokens  / 1e6) × input_price
     + (output_tokens / 1e6) × output_price
# with caching:
     + (cached_tokens / 1e6) × (≈0.1 × input_price)   # cache reads

Example: a 50K-token cached system prompt + 2K question → 4K answer on Sonnet 4.6, warm cache ≈ (50K×0.1 + 2K)/1e6 × $3 + 4K/1e6 × $15 ≈ $0.021 + $0.060 ≈ $0.08/request. Without caching the input alone would cost 52K/1e6 × $3 ≈ $0.156.

Reducing Hallucination

• Ground the model with retrieval (RAG) and instruct it to answer only from provided context, citing sources.
• Constrain the output with structured outputs / strict tool schemas.
• Lower temperature for factual tasks, on models that still accept sampling params. Current reasoning models reject them, so rely on grounding and prompting.
• Add an eval + guardrail layer: LLM-as-judge, schema validation, and refusal handling. See Evaluation & Guardrails.

Interview Q&A

How do you pick a model for a new feature?

Start with the cheapest tier that meets the quality bar (measured on an eval set), scale up only where needed, and route easy traffic to a small model.

What's the single biggest lever for LLM cost at scale?

Prompt caching of a stable prefix, plus right-sizing the model per task.

Why not use tiktoken to count Claude tokens?

Wrong tokenizer: its counts differ from Claude's; use the provider's token-counting API.

temperature vs top_p?

Both control randomness; temperature scales the distribution, top_p truncates it to a probability mass. Tune one, not both hard.

What is a context window and how do you stay within it?

Max tokens per request; budget prompt + context + output, trim/retrieve selectively, and use compaction/summarization for long sessions.

How do you reduce hallucinations?

Ground with RAG + citations, constrain output structure, add evals/guardrails, and (on models that still accept it) lower the temperature.

Related Topics

• RAG
• Agent Systems
• Evaluation & Guardrails
• Multimodal AI