How to Build an LLM From Scratch in PyTorch

I implemented a Transformer/LLM from scratch in PyTorch and trained it as a small GPT like language model. The goal of this video is to understand how modern LLMs work under the hood by implementing a language model from scratch, inspired by the Attention Is All You Need paper. We start with the dataset and tokenization, then build embeddings, positional encoding, masked multi head attention, decoder blocks, and the full PyTorch training loop.This is a decoder only Transformer trained with next token prediction, similar in spirit to GPT. For the dataset I use Yu Gi Oh cards, which makes the final output a bit more fun than another Shakespeare example. Code: https://github.com/vossenwout/llm-from-scratch Paper: https://arxiv.org/abs/1706.03762 Timestamps: 00:00 - Building a Transformer from Scratch in PyTorch 01:15 - Why train on Yu-Gi-Oh cards? 03:05 - Downloading and preprocessing the dataset 08:25 - Building tokenizers for our language model 12:40 - Byte Pair Encoding tokenizer explained 19:15 - Creating a PyTorch dataset 20:35 - Next-token prediction explained 21:45 - Transformer input and output shapes 27:40 - Decoder-only Transformers explained 30:55 - Token embeddings and positional encoding 39:10 - Implementing the Transformer decoder 41:30 - Multi-head attention explained 49:40 - Causal masking for GPT-style models 57:05 - Implementing multi-head attention in PyTorch 1:00:20 - Optimizing attention (vectorization) 1:05:00 - Residual connections, dropout, and layer norm 1:09:25 - Feed-forward networks in Transformers 1:13:25 - Stacking decoder blocks 1:16:40 - Building the PyTorch training loop 1:21:35 - Cross entropy loss and optimization 1:29:50 - Validation loss, perplexity, and checkpoints 1:32:00 - Training the Transformer model 1:34:25 - Training results and overfitting 1:35:45 - Building the inference script 1:38:45 - Sampling tokens with temperature 1:41:20 - Generating AI Yu-Gi-Oh cards 1:44:00 - Opening AI-generated booster packs 1:48:30 - Final thoughts #pytorch #llm #deeplearning