Pre-Training, Post-Training & Alignment: How Language Models Are Actually Built
Language models don't get built in a single training run. They go through three distinct phases — pre-training, post-training (fine-tuning), and alignment — each with different goals, data sources, compute requirements, and techniques.
Language models are built through three distinct training phases — pre-training on raw text, post-training (fine-tuning) for domain specialisation, and alignment for helpfulness and safety. Each phase uses different data, techniques, and compute. Here's how they work and why they matter.
How do pre-training, fine-tuning, and alignment actually differ?
Pre-training teaches a model language and world knowledge from raw text using next-token prediction. Fine-tuning adapts it to specific tasks with labelled examples. Alignment teaches it to be helpful, honest, and safe through human preferences.
Sources: RLHF — Reinforcement Learning from Human Feedback · DPO — Direct Preference Optimisation · LoRA — Low-Rank Adaptation
The Three Phases of Model Training
When people talk about "training an AI model," they usually mean one thing — but modern language models go through three distinct training phases, each with different goals, data sources, and techniques. Understanding these phases matters whether you're evaluating models for production, fine-tuning for your domain, or just trying to grasp why your model sometimes sounds confident while being completely wrong.
The three phases are:
- Pre-training — learning the world from raw text
- Post-training (often called "fine-tuning") — learning to follow instructions
- Alignment (RLHF, DPO, etc.) — learning to be helpful, honest, and safe
Let's break down each one, how they differ, and what's happening under the hood.
Phase 1: Pre-Training — Teaching the Model the World
What It Is
Pre-training is where a model learns language, facts, reasoning patterns, and world knowledge from vast amounts of raw, unlabeled text. This is the most expensive and compute-intensive phase.
The model reads trillions of tokens — books, articles, code, forums, scientific papers — and learns to predict the next token in each sequence. There are no labels, no questions, no instructions. Just raw text.
How It Works
- Architecture: A transformer decoder (or encoder-decoder) is initialised with random weights.
- Tokenisation: Text is converted into tokens — subword units that balance vocabulary size with coverage.
- Masked Prediction: For each token in the sequence, the model learns to predict the next one based on everything before it (causal LM) or from context on both sides (masked LM, used in BERT).
- Loss Function: Cross-entropy between predicted and actual next tokens. The model adjusts weights via backpropagation.
- Scale: Modern pre-training runs on thousands of GPUs for weeks or months, consuming hundreds of millions of dollars in compute.
What the Model Learns
- Grammar, syntax, and semantics of multiple languages
- Factual knowledge (dates, names, scientific facts)
- Reasoning patterns (math, logic, code generation)
- Style and register (formal vs. casual, technical vs. conversational)
- Cross-domain transfer (knowledge from one domain helps in another)
Key Characteristics
| Aspect | Detail |
|---|---|
| Data | Raw, unlabelled text (web crawls, books, code) |
| Objective | Next-token prediction (self-supervised) |
| Compute | Massive — thousands of GPUs, weeks-months |
| Cost | $10M–$100M+ per model |
| Output | A base model that can generate coherent text but cannot reliably follow instructions |
Limitations of Pre-Training
A pre-trained model is essentially a very sophisticated autocomplete system. It can continue a sentence, answer questions if they appear in its training data, and generate plausible-sounding text — but it has no built-in mechanism for following instructions, staying on topic, or refusing harmful requests. It also cannot learn anything new after pre-training ends.
Phase 2: Post-Training / Fine-Tuning — Teaching the Model to Follow Instructions
What It Is
Post-training (commonly called "fine-tuning") adapts a pre-trained model to a specific task or domain using labelled data. Unlike pre-training, which learns from raw text, fine-tuning uses examples of desired input-output pairs.
This is where the model goes from "knows everything but can't do anything specific" to "specialised for your use case."
How It Differs from Pre-Training
| Aspect | Pre-Training | Post-Training / Fine-Tuning |
|---|---|---|
| Data | Raw, unlabelled text | Labelled, task-specific examples |
| Objective | Next-token prediction (self-supervised) | Supervised learning (input → desired output) |
| Compute | Thousands of GPUs, weeks-months | Single GPU or small cluster, hours-days |
| Cost | $10M–$100M+ | $1K–$100K |
| Scope | General-purpose knowledge | Domain-specific or task-specific |
| Duration | One-time (model is built once) | Repeated (re-tuned as needs change) |
Common Fine-Tuning Techniques
Full Fine-Tuning
Every parameter in the model is updated. This produces the best performance but requires significant compute and risks catastrophic forgetting — the model "forgets" its general language abilities while learning the new task.
Parameter-Efficient Fine-Tuning (PEFT)
Only a small subset of parameters is updated, while the rest are frozen. Popular methods include:
- LoRA (Low-Rank Adaptation): Adds low-rank matrices to existing weight layers. A 70B parameter model can be fine-tuned with just a few hundred MB of adapter weights.
- Prefix Tuning / Prompt Tuning: Learns additional "soft prompts" prepended to the input, without changing any model weights.
- Adapter Layers: Small neural networks inserted between transformer layers, trained while the base model stays frozen.
PEFT methods are now the default for most production fine-tuning because they're cheaper, faster, and avoid catastrophic forgetting.
What Fine-Tuning Is Used For
- Domain adaptation: A medical LLM trained on clinical notes and research papers
- Style transfer: Writing in a specific brand voice or tone
- Task specialisation: Code generation, legal document review, customer support
- Format compliance: Structured JSON output, specific response templates
- Language adaptation: Improving performance in low-resource languages
The Fine-Tuning Pipeline
- Data preparation: Curate high-quality examples (input → desired output pairs). Quality matters more than quantity.
- Base model selection: Choose a pre-trained model that's a good starting point for your domain.
- Training: Run fine-tuning with LoRA or full updates, monitoring loss and validation performance.
- Evaluation: Test on held-out examples and real-world scenarios.
- Deployment: Merge adapter weights into the base model (for LoRA) or deploy separately.
Phase 3: Alignment — Teaching the Model to Be Helpful, Honest & Safe
What It Is
Alignment is the process of making a model's outputs consistent with human values and intentions. After pre-training and fine-tuning, a model can follow instructions and produce domain-specific output — but it may still be unhelpful, verbose, misleading, or dangerous.
Alignment methods teach the model how to respond, not just what to say.
How It Differs from Pre-Training and Fine-Tuning
| Aspect | Pre-Training | Fine-Tuning | Alignment |
|---|---|---|---|
| Data | Raw text (unlabelled) | Task-specific examples | Human preferences / feedback |
| Objective | Predict next token | Match desired output | Optimise for human preference |
| Method | Self-supervised learning | Supervised fine-tuning | Reinforcement learning or preference optimisation |
| Key Question | "What comes next?" | "What's the right answer?" | "Which response do humans prefer?" |
Alignment Methods
RLHF (Reinforcement Learning from Human Feedback)
The original and most well-known alignment method:
- Supervised Fine-Tuning: The model is first fine-tuned on a dataset of high-quality responses.
- Reward Model Training: Humans rank multiple model outputs for the same prompt. A separate reward model is trained to predict human preferences.
- Reinforcement Learning: The model is further trained using PPO (Proximal Policy Optimisation) to maximise the reward model's scores.
This is expensive and complex — it requires a reward model, an RL training loop, and careful tuning to avoid reward hacking (where the model learns to game the reward function rather than actually being better).
DPO (Direct Preference Optimisation)
A simpler alternative to RLHF that skips the reward model and RL loop entirely. Instead:
- Collect pairs of responses — a "chosen" (preferred) response and a "rejected" (less preferred) response for each prompt.
- Train the model directly to increase the probability of chosen responses and decrease rejected ones, using a single loss function derived from the Bradley-Terry model.
DPO is cheaper, more stable, and produces comparable results to RLHF in most benchmarks. It's now the default alignment method for many open-source models.
Other Alignment Approaches
- KTO (Kahneman-Tversky Optimisation): A preference-free alignment method that uses individual examples rather than pairs, comparing outputs against a reference model.
- ORPO (Odds Ratio Preference Optimisation): Combines SFT and DPO into a single training objective, simplifying the pipeline further.
- Constitutional AI: Instead of human feedback, the model is trained to follow a set of principles (a "constitution") and self-critique its own outputs.
What Alignment Achieves
- Helpfulness: Responses are concise, relevant, and directly address the user's intent
- Honesty: The model avoids fabricating facts and expresses uncertainty when appropriate
- Safety: The model refuses harmful requests (bomb-making instructions, self-harm guidance)
- Tone control: The model maintains an appropriate tone (professional, empathetic, casual) based on context
Training Outside Language Models
The concepts above apply specifically to language models, but similar three-phase training patterns exist across other AI modalities:
Computer Vision Models
| Phase | Description |
|---|---|
| Pre-training | Learn visual features from raw images (ImageNet, self-supervised contrastive learning) |
| Fine-tuning | Adapt to specific tasks: object detection, segmentation, medical imaging |
| Alignment | Less common in vision; sometimes involves human preference ranking for image generation outputs |
Speech / Audio Models
| Phase | Description |
|---|---|
| Pre-training | Learn acoustic representations from raw audio (wav2vec 2.0, HuBERT use self-supervised learning on unlabeled speech) |
| Fine-tuning | Adapt to specific tasks: transcription, speaker identification, language detection |
| Alignment | Fine-tune for natural prosody, accent neutrality, or style control in text-to-speech |
Multimodal Models (Vision + Language)
| Phase | Description |
|---|---|
| Pre-training | Learn joint image-text representations (CLIP, ALIGN, Flamingo) |
| Fine-tuning | Adapt to specific tasks: visual question answering, image captioning, document understanding |
| Alignment | Align generated descriptions with human preferences, ensure factual accuracy about images |
Diffusion Models (Image Generation)
| Phase | Description |
|---|---|
| Pre-training | Learn to denoise random noise into coherent images from millions of image-text pairs |
| Fine-tuning | Adapt style (watercolour, photorealistic, anime) or domain (fashion, architecture, product photography) |
| Alignment | Safety filters, content policy enforcement, style consistency via human feedback |
The Common Pattern
Across all modalities, the pattern is consistent:
- Learn general representations from massive unlabeled data
- Specialise using labelled examples for a specific task or domain
- Align to human preferences for quality, safety, and usefulness
The compute cost and complexity decrease dramatically at each phase.
Why This Matters for Practitioners
Choosing the Right Phase for Your Use Case
- You need general knowledge and reasoning? Use a pre-trained base model. No fine-tuning needed.
- You need domain-specific expertise (legal, medical, financial)? Fine-tune a pre-trained model on your data.
- You need the model to follow a specific format or tone? Fine-tune with instruction examples.
- Your model is helpful but sometimes unhelpful, verbose, or unsafe? Alignment (DPO) is the right intervention.
The Data Problem
The bottleneck has shifted:
- Pre-training era: Compute was the bottleneck. Data was abundant (web crawls).
- Fine-tuning era: High-quality labelled data is the bottleneck. There isn't enough domain-specific data for every use case.
- Alignment era: High-quality preference data is the bottleneck. DPO and related methods require carefully curated chosen/rejected pairs, which are expensive to produce.
This is why synthetic data generation (using stronger models to create training data for weaker ones) has become such a critical area of research.
The Post-Training Landscape Is Evolving Fast
What we call "post-training" today will look different in a year. Emerging trends:
- Unified pipelines: Methods like ORPO combine SFT and alignment into one step.
- Data-centric focus: Better data quality matters more than model architecture.
- Automated preference generation: Using stronger models to generate training data for weaker ones.
- Continuous training: Models that can learn from new data without full retraining (still research-stage).
Summary
| Phase | Goal | Data | Compute | Cost |
|---|---|---|---|---|
| Pre-training | Learn language and world knowledge | Raw text (trillions of tokens) | Thousands of GPUs, months | $10M–$100M+ |
| Post-training / Fine-tuning | Specialise for a task or domain | Labelled examples | Single GPU to small cluster, hours-days | $1K–$100K |
| Alignment | Make outputs helpful, honest, safe | Human preferences / feedback | Moderate (less than fine-tuning) | $5K–$50K |
The key insight: these are not alternatives. They're sequential stages in building a useful AI system. A pre-trained model without alignment is unpredictable. An aligned model without domain fine-tuning lacks expertise. Understanding which phase you need — and how each one differs — is the foundation of working with modern AI models effectively.
Tags: LLM, Machine Learning, Pre-Training, Fine-Tuning, Post-Training, RLHF, DPO, Alignment, AI Training, MLOps
Feature image: Rosenblatt, F. / Public domain / Wikimedia Commons.