🔴 Advanced ⚙️ Type: Educational Framework / AI Training 💸 Free & Open Source ⭐ 6,400+ GitHub Stars
What is Train LLM From Scratch?
Train LLM From Scratch is an incredibly popular open-source educational repository created by AI Engineer Fareed Khan. It provides a complete, end-to-end framework for building, training, and aligning Large Language Models (LLMs) entirely from scratch using pure PyTorch.
Most AI tutorials teach you how to use heavy abstraction libraries like Hugging Face’s transformers or trl to fine-tune existing models. This project takes the opposite approach. It forces you to build the core architecture—based on the famous “Attention is All You Need” paper—line by line. You will manually code the tokenizers, the Multi-Head Attention mechanisms, the Feed-Forward networks, and the training loops.
Recently, the repository underwent a massive update. It now goes far beyond basic pre-training and includes a complete, hand-written Post-Training Suite. It guides you through the exact pipeline used to make modern AI chatbots polite and helpful, providing from-scratch implementations of Supervised Fine-Tuning (SFT), Reward Modeling, PPO, DPO, and GRPO.
Who is it for?
- AI/ML Students and Researchers who want to deeply understand the underlying math and architecture of modern LLMs rather than just treating them as “black-box” APIs.
- Data Scientists looking to learn how to train highly customized, small-scale models (like 13 million or 2 billion parameters) on highly specific proprietary data.
- Python Developers who want a structured, hands-on masterclass in PyTorch, Object-Oriented Programming (OOP), and Reinforcement Learning from Human Feedback (RLHF).
- Hobbyists who want to experience the magic of typing a prompt and getting a response from an artificial “brain” that they built entirely by themselves.
What makes it special?
- Zero Magic Libraries — By explicitly banning libraries like
peft,trl, andtransformers, the project ensures you learn how the AI actually computes its answers at a matrix-multiplication level. - Hardware Accessible — You don’t need a $300,000 server cluster. The project is specifically engineered to allow you to train a functional 13-million parameter miniature LLM on a standard, free Google Colab T4 GPU.
- Comprehensive Alignment Code — Finding raw code for pre-training is easy, but finding pure, hand-written code for complex alignment algorithms like GRPO (used in DeepSeek) or DPO is incredibly rare. This repository has them all.
- Runnable Textbook — The GitHub README is formatted like a high-quality textbook, complete with flowcharts, math explanations, and code breakdowns that explain why each block of Python is necessary.
Requirements before you start
Because you are building an AI from the ground up, you will need a solid technical foundation and the right hardware:
- Python 3.8+ & PyTorch — The core programming language and machine learning framework.
- A Dedicated GPU — While you can read the code anywhere, actually training the model in a reasonable timeframe requires a CUDA-enabled GPU. A free Google Colab notebook is sufficient for the small-scale models.
- Programming Knowledge — You need a basic understanding of Object-Oriented Programming (OOP) in Python, as well as a conceptual understanding of Neural Networks.
- Storage Space — If you intend to run this locally rather than in the cloud, downloading the recommended chunks of “The Pile” dataset requires tens of gigabytes of free hard drive space.
Step-by-step installation
Note: It is highly recommended to run these steps inside a Jupyter Notebook or Google Colab environment for easy debugging.
Step 1 — Clone the Repository
Open your terminal or notebook and pull the source code to your machine:
git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git
cd train-llm-from-scratchStep 2 — Install PyTorch
Ensure you have PyTorch installed with GPU acceleration. (If you are on Google Colab, this is already done for you). Otherwise, run:
pip install torch numpyStep 3 — Download the Training Data
The project uses a script to fetch chunks of “The Pile” (a massive, diverse text dataset). Run the data preparation script to download your first batch:
python src/data_prep.py --train_max 1(Note: Even setting it to 1 will download an ~11 GB file, so be prepared to wait!).
Step 4 — Train the Base Model
With the data ready, execute the core training loop. This script will initialize your hand-written Transformer blocks and begin teaching the neural network how to predict the next token:
python src/train.pyYou will see the terminal output the “Training Loss.” As the number goes down, your model is getting smarter!
Step 5 — Generate Text
Once the training finishes and the .pt model checkpoint is saved, you can chat with the brain you just built:
python src/generate.py --model_path path/to/your/model.pt --input_text "The future of artificial intelligence is" --max_new_tokens 100Common errors and fixes
| Error | What it means | How to fix it |
|---|---|---|
CUDA Out of Memory (OOM) | Your GPU does not have enough VRAM to hold the batch of training data and the model gradients simultaneously. | Open the training script and reduce the batch_size or the context_length. For smaller GPUs, try training the 13M parameter configuration instead of the 2B configuration. |
| Data download fails or hangs indefinitely | The server hosting The Pile is timing out, or your local hard drive is completely full. | Check your storage capacity. If the connection keeps dropping, you can bypass the downloader script entirely and simply supply your own .txt file (like a Shakespeare book or a large code file) to train the model on a custom micro-dataset. |
FutureWarning: torch.load weights_only=False | Newer versions of PyTorch flag the loading of arbitrary .pt checkpoint files as a potential security risk. | This is just a warning and won’t stop execution. To remove it, open generate.py and update the load function to explicitly state torch.load(path, weights_only=True). |
Free vs Paid comparison
| Feature | Train-LLM-From-Scratch (GitHub) | Premium AI Bootcamps (e.g., DeepLearning.AI) |
|---|---|---|
| Cost | $0 (Free code, runs on free Google Colab) | $50/month to $2,000+ for certified courses |
| Code Transparency | ✅ 100% Pure PyTorch (No abstractions) | Often relies on high-level APIs (Hugging Face) |
| RLHF / Alignment Coverage | ✅ Extensive (Includes PPO, DPO, and GRPO) | Usually omitted or skimmed in beginner courses |
| Video Lectures | ❌ Text and code only | ✅ Professional video instruction |
Bottom line: If you truly want to call yourself an AI Engineer, you must know what happens inside the black box. Fareed Khan’s repository is one of the most accessible, comprehensive, and hardware-friendly ways to learn the brutal math and code of LLM training. It replaces thousands of dollars of paid tuition with a single, incredibly well-documented GitHub repository.
Alternatives — 3 similar tools
1. nanoGPT (by Andrej Karpathy)
The legendary benchmark for training a miniature language model from scratch. Written by a founding member of OpenAI, nanoGPT is incredibly sleek, fast, and optimized. However, its primary focus is on base pre-training, whereas Fareed Khan’s repository goes much deeper into modern post-training and RLHF alignment methodologies.
2. LLMs-from-scratch (by Sebastian Raschka)
The official code repository for the highly acclaimed book “Build a Large Language Model (From Scratch)”. It provides a phenomenal, highly academic step-by-step approach to coding an LLM. It is arguably the best starting point for absolute beginners who prefer a structured, book-driven learning environment.
🔗 github.com/rasbt/LLMs-from-scratch
3. train-tiny-llm
Another excellent repository by Fareed Khan. While train-llm-from-scratch focuses heavily on alignment algorithms (DPO/PPO), train-tiny-llm focuses on the practical deployment pipeline. It walks you through training a 29M parameter model, creating a custom tokenizer, fine-tuning it, and directly deploying it into a Streamlit web app.
🔗 github.com/FareedKhan-dev/train-tiny-llm
🚀 Want more free AI tools like this?
We find, test, and write setup guides for the best free and open-source AI tools — so you don’t have to dig through GitHub yourself.Browse Free AI Tools at globalaiforce.com/shop →
📸 Follow us for daily AI tool tips and tutorials: instagram.com/globalaiforce