AI Engineering Part 1: Fundamentals of Foundation Models

A fundamental paradigm shift has taken place in software engineering over the last few years: AI Engineering has emerged as a distinct, highly technical software discipline.

Traditionally, Machine Learning Engineers built custom models from scratch using PyTorch or TensorFlow, curating features, training models on proprietary datasets, and shipping single-task models (e.g. sentiment classifiers or object detectors).

In contrast, AI Engineers build production applications on top of pre-trained Foundation Models (such as GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and open-weight models like LLaMA 3 and DeepSeek V3).

In this 5-part masterclass series, we systematically unpack the engineering principles, architectural patterns, and operational systems required to build production-grade AI applications.

In Part 1, we lay down the bedrock: Foundation Model Lifecycles, Pre-training vs Alignment, GPU Memory Math, Quantization, and Build vs Buy Trade-offs.

1. The Anatomy of a Foundation Model

A Foundation Model is a large-scale deep learning network trained on vast quantities of unstructured data (text, code, images, audio, video) using self-supervised learning at scale.

Unlike traditional software that executes deterministic, hand-written rules, foundation models act as probabilistic reasoning engines. They process input token sequences and output probability distributions over potential completion tokens.

Step 1: Pre-training (Self-Supervised Learning)

The base model is trained on trillions of tokens (e.g. LLaMA 3 was pre-trained on 15+ trillion tokens).

The pre-training objective is deceptively simple: Causal Language Modeling (Next-Token Prediction). Given a sequence of tokens (t_1, t_2, ..., t_{n-1}), maximize the log-likelihood of predicting the true target token t_n:

Loss_CLM(θ) = - ∑ log P(t_i | t_1, t_2, ..., t_{i-1}; θ)

During this phase, the model compresses vast amounts of human world knowledge, grammar, coding syntax, and logical patterns into its high-dimensional weight matrices.

Crucial Distinction: A raw Base Model is NOT a helpful assistant chatbot! If you prompt a Base Model with “What is the capital of France?”, it might complete it as “and what is the capital of Germany?” because it is merely completing text patterns found on web pages!

Step 2: Supervised Fine-Tuning (SFT / Instruction Tuning)

To transform a raw Base Model into a helpful assistant, we fine-tune it on thousands of high-quality, hand-crafted instruction-response pairs:

{
  "instruction": "Summarize the following legal clause in two bullet points...",
  "response": "1. The tenant must pay rent by the 1st of each month.\n2. Late fees apply after day 5."
}

Through SFT, the model learns the conversational format (<|im_start|>user ... <|im_end|> <|im_start|>assistant ...) and learns to respond directly to user directives.

Step 3: Alignment (RLHF, DPO, & KTO)

While SFT teaches formatting, the model can still generate toxic text, fabricate false statements (hallucinations), or assist in malicious activities. The Alignment phase aligns model outputs with human values of Helpfulness, Honesty, and Harmlessness (HHH).

1. Reinforcement Learning from Human Feedback (RLHF)

Pioneered by OpenAI for InstructGPT and ChatGPT:

1. Train a secondary Reward Model on human preference rankings (“Response A is better than Response B”).
2. Fine-tune the base model using PPO (Proximal Policy Optimization) to maximize the score given by the Reward Model while penalizing divergence from the base model using a KL-divergence constraint:

Objective(θ) = E_{(x,y) ~ D} [ R_φ(x,y) - β * D_KL( π_θ(y|x) || π_ref(y|x) ) ]

2. Direct Preference Optimization (DPO)

Pioneered in 2023 (Rafailov et al.), DPO mathematically proves that you can bypass training a separate Reward Model and running complex PPO entirely! Instead, DPO directly optimizes the policy model on preferred y_w (winning) vs dispreferred y_l (losing) responses using a simple binary cross-entropy loss:

Loss_DPO(θ) = - E_{(x, y_w, y_l)} [ log σ ( β * log( π_θ(y_w|x) / π_ref(y_w|x) ) - β * log( π_θ(y_l|x) / π_ref(y_l|x) ) ) ]

DPO is computationally stable and is now the industry standard for aligning open-weight models like LLaMA and Mistral.

2. GPU Memory Math: Calculating VRAM Requirements

As an AI Engineer, you must know how to calculate whether a model can physically run on specific GPU hardware (e.g. an NVIDIA A100 80GB or H100 80GB).

TOTAL GPU VRAM REQUIRED = Model Weights + KV Cache + Activation Memory + Overhead

1. Model Weights Memory

Model weights are represented as parameters stored in specific precision formats:

• FP32 (32-bit Floating Point): 4 bytes per parameter
• FP16 / BF16 (16-bit Floating Point): 2 bytes per parameter
• INT8 (8-bit Quantized Integer): 1 byte per parameter
• INT4 (4-bit Quantized Integer): 0.5 bytes per parameter

Formula for Model Weight VRAM:

VRAM_weights = Parameters (in Billions) * Bytes per Parameter

Examples:

• LLaMA 3 70B in FP16: 70 * 2 GB = 140 GB VRAM (Requires 2x 80GB GPUs!)
• LLaMA 3 70B in INT4: 70 * 0.5 GB = 35 GB VRAM (Fits easily on a single 80GB GPU!)

2. Key-Value (KV) Cache Memory

During inference, to avoid recalculating Key and Value vectors for previous tokens at every step, the model caches them in GPU VRAM (KV Cache).

Formula for KV Cache Memory:

VRAM_KV = 2 * Layers * Heads * Dimension * Sequence_Length * Batch_Size * Bytes_Per_Precision

For long context windows (e.g. 128k tokens) or high concurrent batch sizes, the KV Cache can easily consume more VRAM than the actual model weights!

3. Quantization: Shrinking Models without Losing Intelligence

Quantization is the technique of converting high-precision floating-point weights (FP16) down to lower-precision representations (INT8, INT4, or FP4) with minimal loss in model accuracy.

FP16 Weight (16 bits: 1 Sign + 5 Exponent + 10 Mantissa)
                     │
                     ▼ (Quantization Transformation)
INT4 Weight  (4 bits: Representing values -8 to +7 with Scale & Zero-Point)

Popular Quantization Techniques:

1. GGUF (formerly GGML): Designed for CPU and mixed CPU/GPU execution (popularized by llama.cpp). Uses k-quants (e.g. Q4_K_M, Q5_K_S) to keep critical attention layers at higher precision while aggressively quantizing feed-forward layers.
2. AWQ (Activation-aware Weight Quantization): Protects the top 1% salient weights that correspond to large activation magnitudes, quantizing the remaining 99% down to 4-bit. Achieves near-FP16 accuracy while running 3x faster on GPUs.
3. GPTQ: Post-training quantization based on second-order matrix derivatives (Hessian matrices). Ideal for server-side GPU batch inference.

4. The Spectrum of Model Customization

When an off-the-shelf model fails to satisfy your application requirements, you have four distinct levers along the Customization Spectrum:

5. Proprietary APIs vs Self-Hosted Open-Weight Models

The foundational architecture decision for any AI team is deciding between Proprietary SaaS APIs (OpenAI, Anthropic, Google) and Self-Hosted Open-Weight Models (LLaMA, Mistral, DeepSeek).

Summary of Part 1

In Part 1 of our AI Engineering masterclass series, we established:

1. Foundation Models transition from Base models (pre-training) to Instruct models via SFT and Alignment (DPO/RLHF).
2. VRAM Math is mandatory to size hardware: Model Weights + KV Cache + Activations.
3. Quantization (AWQ, GGUF) enables running massive models at INT4 precision with minimal loss in reasoning capability.
4. RAG solves knowledge freshness, while Fine-Tuning (LoRA) solves formatting, style, and domain syntax.

Up next: AI Engineering Part 2: Prompt Engineering, Context Windows & Structured JSON Outputs.