AI Engineering Part 2: Prompt Engineering, Context Windows & Structured Outputs

When building production software applications powered by Large Language Models, treating prompts as casual text strings is a recipe for failure.

If your prompt produces slightly different JSON output keys on 2% of user requests, your application backend will throw unhandled runtime exceptions (JSON.parse() syntax errors, missing field errors, type mismatch crashes).

In Part 2 of our AI Engineering masterclass series, we treat prompts as code. We cover advanced reasoning prompting patterns, context window dynamics, tokenization mechanics, and modern frameworks for 100% deterministic structured outputs.

1. Advanced Prompt Architecture & Reasoning Tactics

Prompt Engineering is the practice of designing input context to steer a probabilistic model’s token distribution toward a desired high-accuracy response.

Technique 1: Few-Shot Prompting (In-Context Learning)

Providing 2 to 5 concrete input-output demonstration pairs inside your system prompt is frequently more effective than writing multiple paragraphs of complex text rules.

System Prompt:
You are an enterprise support triage assistant. Classify the customer message into JSON format with keys "category", "urgency", and "suggested_action".

Example 1:
Input: "Our production server is completely down and returning 500 errors across all endpoints!"
Output: {"category": "Infrastructure Outage", "urgency": "CRITICAL", "suggested_action": "Page On-Call SRE"}

Example 2:
Input: "Hi, I forgot my password and cannot sign in on my phone."
Output: {"category": "Account Access", "urgency": "LOW", "suggested_action": "Trigger Password Reset Email"}

User Input:
"I was charged $99 twice on my credit card statement this morning!"

Why Few-Shot Prompting Works: Modern LLMs act as pattern-completion engines. By providing consistent input-output exemplars, you anchor the model’s conditional probability distribution to match the exact syntax, key naming, and value ranges of your examples.

Technique 2: Chain of Thought (CoT) Prompting

Introduced by Wei et al. (Google Brain, 2022), Chain of Thought (CoT) forces the model to generate intermediate reasoning steps before emitting a final answer.

Consider a multi-step math problem:

BAD (Direct Prompting):
Q: A store has 12 boxes of apples. Each box contains 45 apples. 15% of the apples are spoiled. 
How many non-spoiled apples remain?
A: [Model guesses 450 quickly and incorrectly!]

GOOD (Chain of Thought Prompting):
Q: A store has 12 boxes of apples. Each box contains 45 apples. 15% of the apples are spoiled. 
How many non-spoiled apples remain? 
Think step by step before providing the final answer.

A: 
1. Total apples = 12 boxes * 45 apples/box = 540 apples.
2. Spoiled apples = 15% of 540 = 0.15 * 540 = 81 apples.
3. Non-spoiled apples = 540 total - 81 spoiled = 459 apples.
Final Answer: 459.

The Computational Reason CoT Works:

In an autoregressive Transformer, compute is allocated per token generated. When you force the model to answer directly without CoT, it must perform all mathematical reasoning inside a single forward pass layer calculation. When you force it to write out intermediate reasoning steps, each output token grants the model an additional forward pass of matrix multiplication compute!

Technique 3: Tree of Thoughts (ToT) Framework

Pioneered by Yao et al. (2023), Tree of Thoughts (ToT) extends CoT by allowing LLMs to explore multiple reasoning paths concurrently, evaluate intermediate progress, and backtrack when a path hits a dead end.

ToT is essential for complex software tasks such as codebase refactoring, multi-step bug diagnoses, or strategic planning.

2. Context Window Mechanics & Tokenization

To write efficient prompts, engineers must understand how text is physically processed by the LLM tokenizer.

Tokenization (Byte-Pair Encoding / BPE)

LLMs do not process characters or words. Text is broken down into Tokens using algorithms like Byte-Pair Encoding (BPE, used by OpenAI tiktoken).

Input String: "Antigravity AI Engineering"
Tokens:       ["Anti", "gravity", " AI", " Eng", "ineer", "ing"]
Token IDs:    [ 47211,     29482, 15321,   3821,   4922,  301 ]

Tokenization Gotchas for AI Engineers:

1. Numbers & Math: Tokenizers split numbers unpredictably (1000000 might become ["1000", "000"]). This is why LLMs struggle with digit alignment in math!
2. Whitespace & Formatting: Extra leading/trailing spaces alter token IDs, which can alter model output behavior.
3. Non-English Languages: English text averages ~0.75 words per token. Languages like Japanese or Arabic require 2x to 4x more tokens per sentence due to tokenizer vocabulary compression bias, making non-English API calls significantly more expensive!

Context Window Dynamics: The “Lost in the Middle” Problem

Modern LLMs boast massive context windows (128k to 1M+ tokens). However, researchers (Liu et al., 2023) demonstrated that model recall accuracy is non-uniform across the context window:

Architectural Rule for Prompt Construction:

• Place critical system guidelines and constraints at the very top of the prompt.
• Place the user’s specific query and final execution instructions at the very bottom of the prompt.
• Place long, retrieved document context in the middle.

3. Guaranteeing 100% Structured Outputs (JSON Schema)

In production backend engineering, unstructured prose is unusable. Your backend requires typed objects (UserProfile, OrderPayload, TicketTriage).

If you simply ask an LLM in prose: “Return your answer as JSON with keys ‘a’ and ‘b’”, the model will frequently emit: Here is the requested JSON:\n```json\n{"a": 1, "b": 2}\n```\nHope this helps!

Parsing this with standard JSON.parse() crashes your application.

Solution 1: Constrained Decoding (Grammar-Based Sampling)

The most robust way to guarantee structured output is Constrained Decoding at the token sampling level (used by Outlines, vLLM, and OpenAI Structured Outputs).

During autoregressive generation, before predicting token t_n, the inference engine evaluates the current string against a strict BNF Grammar or JSON Schema. Any token in the vocabulary that would violate the schema is assigned a probability score of zero (-infinity logit mask)!

Solution 2: Schema Enforcement with Zod & Pydantic (Instructor Framework)

In TypeScript and Python, we define strict schema contracts using Zod or Pydantic, and pass them to libraries like Instructor.

TypeScript Example with Zod:

import { z } from "zod";
import OpenAI from "openai";

// 1. Define strict type schema
const UserAnalysisSchema = z.object({
  userId: z.string().uuid(),
  sentimentScore: z.number().min(-1.0).max(1.0),
  keyTopics: z.array(z.string()).min(1).max(5),
  flagForReview: z.boolean(),
  summary: z.string().max(200)
});

// Infer TypeScript type automatically
type UserAnalysis = z.infer<typeof UserAnalysisSchema>;

// 2. OpenAI Structured Output API call
const openai = new OpenAI();

async function analyzeUserFeedback(rawText: string): Promise<UserAnalysis> {
  const completion = await openai.beta.chat.completions.parse({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: "Extract structured user analysis metrics." },
      { role: "user", content: rawText }
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "user_analysis",
        strict: true,
        schema: {
          type: "object",
          properties: {
            userId: { type: "string" },
            sentimentScore: { type: "number" },
            keyTopics: { type: "array", items: { type: "string" } },
            flagForReview: { type: "boolean" },
            summary: { type: "string" }
          },
          required: ["userId", "sentimentScore", "keyTopics", "flagForReview", "summary"],
          additionalProperties: false
        }
      }
    }
  });

  return completion.choices[0].message.parsed as UserAnalysis;
}

4. Prompt Versioning & Software Best Practices

To manage prompts cleanly in production repositories:

1. Never Hardcode Prompts in Application Logic: Store prompts in template files (.prompt or YAML files) decoupled from business code.
2. Treat Prompts as Code: Check prompt templates into Git version control. Run automated test suites whenever a prompt file is updated.
3. Template Substitution: Use template variables (e.g. {{user_name}}, {{context_documents}}) to assemble prompts dynamically at runtime.

# prompts/triage_v2.yaml
version: "2.1.0"
model: "gpt-4o-mini"
temperature: 0.1
system_prompt: |
  You are an enterprise support triage assistant.
  Extract user intent according to the provided schema.
user_template: |
  Customer ID: {{user_id}}
  Message: {{message_body}}

Summary of Part 2

In Part 2 of our AI Engineering masterclass series, we established:

1. Chain of Thought (CoT) works by allocating additional token generation compute to intermediate reasoning steps.
2. Tokenizer BPE splits words, numbers, and non-English scripts unpredictably, directly impacting cost and accuracy.
3. Context Window Recall follows a U-shaped curve; place critical guidelines at the top/bottom and document dumps in the middle.
4. Constrained Decoding & JSON Schema Enforcement (Zod/Pydantic) guarantee 100% deterministic outputs for backend integration.

Up next: AI Engineering Part 3: RAG Pipelines & Vector Databases (Chunking, Embeddings, Hybrid Search, & Reranking).