Andrej Karpathy

How I use LLMs

2.9M views · 1y ago · 2:11:12

The example-driven, practical walkthrough of Large Language Models and their growing list of related features, as a new entry to my general audience series on LLMs. In this more practical followup, I take you through the many ways I use LLMs in my own life.

Topics: ChatGPT interaction under the hood · Basic LLM interactions examples · Be aware of the model you're using, pricing tiers · Thinking models and when to use them · Tool use: internet search · Tool use: deep research

Deep Dive into LLMs like ChatGPT

10.4M views · 1y ago · 3:31:24

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications.

Topics: pretraining data (internet) · tokenization · neural network I/O · neural network internals · inference · GPT-2: training and inference

▶️ Mentioned videos: AlphaGo Move 37 video

📄 Mentioned papers: Llama 3 paper Meta · InstructGPT paper on SFT · DeepSeek-R1 paper

Let's reproduce GPT-2 (124M)

1.2M views · 2y ago · 4:01:26

We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations.

Topics: exploring the GPT-2 (124M) OpenAI checkpoint · SECTION 1: implementing the GPT-2 nn.Module · loading the huggingface/GPT-2 parameters · implementing the forward pass to get logits · sampling init, prefix tokens, tokenization · sampling loop

📄 Mentioned papers: Attention is All You Need paper · OpenAI GPT-3 paper

Let's build the GPT Tokenizer

1.3M views · 2y ago · 2:13:35

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings.

Topics: strings in Python, Unicode code points · Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 · daydreaming: deleting tokenization · Byte Pair Encoding (BPE) algorithm walkthrough · starting the implementation · counting consecutive pairs, finding most common pair

▶️ Mentioned videos: Playlist of the whole Zero to Hero series so far

[1hr Talk] Intro to Large Language Models

4.4M views · 2y ago · 59:48

This is a 1 hour general-audience introduction to Large Language Models: the core technical component behind systems like ChatGPT, Claude, and Bard. What they are, where they are headed, comparisons and analogies to present-day operating systems, and some of the security-related challenges of this new computing paradigm. As of November 2023 (this field moves fast!).

Topics: LLM Inference · LLM Training · LLM dreams · How do they work? · Finetuning into an Assistant · Summary so far

Let's build GPT: from scratch, in code, spelled out.

8.3M views · 3y ago · 1:56:20

We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.

Topics: reading and exploring the data · tokenization, train/val split · data loader: batches of chunks of data · simplest baseline: bigram language model, loss, generation · training the bigram model · port our code to a script

▶️ Mentioned videos: Playlist of the whole Zero to Hero series so far

📄 Mentioned papers: Attention is All You Need paper · OpenAI GPT-3 paper

Building makemore Part 5: Building a WaveNet

306K views · 3y ago · 56:22

We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered).

Topics: starter code walkthrough · let’s fix the learning rate plot · pytorchifying our code: layers, containers, torch.nn, fun bugs · overview: WaveNet · re-running baseline code on blocksize 8 · implementing WaveNet

📄 Mentioned papers: WaveNet 2016 DeepMind

Building makemore Part 4: Becoming a Backprop Ninja

391K views · 4y ago · 1:55:24

We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd.

Topics: starter code · exercise 1: backproping the atomic compute graph · brief digression: bessel’s correction in batchnorm · exercise 2: cross entropy loss backward pass · exercise 3: batch norm layer backward pass · exercise 4: putting it all together

📄 Mentioned papers: BatchNorm paper

📰 Mentioned news: Yes you should understand backprop

Building makemore Part 3: Activations & Gradients, BatchNorm

572K views · 4y ago · 1:55:58

We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network.

Topics: starter code · fixing the initial loss · fixing the saturated tanh · calculating the init scale: “Kaiming init” · batch normalization · batch normalization: summary

📄 Mentioned papers: Kaiming init · BatchNorm paper · Good paper illustrating some of the problems with batchnorm in practice

Building makemore Part 2: MLP

616K views · 4y ago · 1:15:40

We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.).

Topics: Bengio et al. 2003 (MLP language model) paper walkthrough · (re-)building our training dataset · implementing the embedding lookup table · implementing the hidden layer + internals of torch.Tensor: storage, views · implementing the output layer · implementing the negative log likelihood loss

The spelled-out intro to language modeling: building makemore

1.3M views · 4y ago · 1:57:45

We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).

Topics: reading and exploring the dataset · exploring the bigrams in the dataset · counting bigrams in a python dictionary · counting bigrams in a 2D torch tensor ("training the model") · visualizing the bigram tensor · deleting spurious (S) and (E) tokens in favor of a single . token

Stable diffusion dreams of psychedelic faces

55K views · 4y ago · 4:02

Prompt: "psychedelic faces" Stable diffusion takes a noise vector as input and samples an image. To create this video I smoothly (spherically) interpolate between randomly chosen noise vectors and render frames along the way.

Stable diffusion dreams of steampunk brains

39K views · 4y ago · 19:26

Prompt: "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine" Stable diffusion takes a noise vector as input and samples an image. To create this video I smoothly (spherically) interpolate between randomly chosen noise vectors and render frames along the way.

Stable diffusion dreams of tattoos

106K views · 4y ago · 1:46

Dreams of tattoos. (There are a few discrete jumps in the video because I had to erase portions that got just a little , believe I got most of it) Links Stable diffusion Code used to make this video

The spelled-out intro to neural networks and backpropagation: building micrograd

4.5M views · 4y ago · 2:25:52

This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. micrograd on github jupyter notebooks I built in this video

Topics: micrograd overview · derivative of a simple function with one input · derivative of a function with multiple inputs · starting the core Value object of micrograd and its visualization · manual backpropagation example #1: simple expression · manual backpropagation example #2: a neuron

Stable diffusion dreams of "blueberry spaghetti" for one night

98K views · 4y ago · 5:03

Prompt: "blueberry spaghetti" Stable diffusion takes a noise vector as input and samples an image. To create this video I simply smoothly interpolate between randomly chosen noise vectors and render frames along the way.

Stable diffusion dreams of steam punk neural networks

74K views · 4y ago · 2:35

A stable diffusion dream. The prompt was "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine" the new and improved v2 version of this video is now here

▶️ Mentioned videos: the and improved v2 version of this video is