I make videos about machine learning research papers, programming, and issues of the AI community, and the broader impact of AI in society. Twitter: https://twitter.com/ykilcher Discord: https://ykilcher.com/discord BitChute: https://www.bitchute.com/channel/yannic-kilcher LinkedIn: https://www.linkedin.com/in/ykilcher BiliBili: https://space.bilibili.com/2017636191 If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribe
Watched Yannic Kilcher? Tell other learners what you think below, or read the discussion around the channel.
13:4613K views · 8mo ago · 13:46
All information about GTC and the DGX Spark Raffle is here Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
2:33:37
47:0225K views · 10mo ago · 47:02
Abstract Diffusion language models hold the promise of fast parallel generation, while autoregressive (AR) models typically excel in quality due to their causal structure aligning naturally with language modeling. This raises a fundamental question: can we achieve a synergy with high throughput, higher GPU utilization, and AR level quality?
📄 Mentioned papers: arXiv 2511.08923
32:3125K views · 11mo ago · 32:31
Abstract Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context.
📄 Mentioned papers: arXiv 2501.00663
40:1023K views · 1y ago · 40:10
Abstract We propose an extension of the decoder Transformer that conditions its generative process on random latent variables which are learned without supervision thanks to a variational procedure. Experimental evaluations show that allowing such a conditioning translates into substantial improvements on downstream tasks.
📄 Mentioned papers: arXiv 2510.17558
13:1910K views · 1y ago · 13:19
Theo's Video Cloudflare article Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
▶️ Mentioned videos: Theo's Video
48:5737K views · 1y ago · 48:57
Abstract Vector embeddings have been tasked with an ever-increasing set of retrieval tasks over the years, with a nascent rise in using them for reasoning, instruction-following, coding, and more. These new benchmarks push embeddings to work for any query and any notion of relevance that could be given.
📄 Mentioned papers: arXiv 2508.21038
7:09
37:4933K views · 1y ago · 37:49
Guest: Context Rot
Abstract Large Language Models (LLMs) are typically presumed to process context uniformly—that is, the model should handle the 10,000th token just as reliably as the 100th. However, in practice, this assumption does not hold. We observe that model performance varies significantly as input length changes, even on simple tasks. In this report, we evaluate 18 LLMs, including the state-of-the-art GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 models.
47:5133K views · 1y ago · 47:51
Abstract Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards).
📄 Mentioned papers: arXiv 2507.02092
56:2617K views · 1y ago · 56:26
An in-depth look at Anthropic's Transformer Circuit Blog Post Part 1 here Abstract We investigate the internal mechanisms used by Claude 3.5 Haiku — Anthropic's lightweight production model — in a variety of contexts, using our circuit tracing methodology.
▶️ Mentioned videos: Part 1
54:0559K views · 1y ago · 54:05
An in-depth look at Anthropic's Transformer Circuit Blog Post Abstract We investigate the internal mechanisms used by Claude 3.5 Haiku — Anthropic's lightweight production model — in a variety of contexts, using our circuit tracing methodology.
1:09:00176K views · 1y ago · 1:09:00
GRPO is one of the core advancements used in Deepseek-R1, but was introduced already last year in this paper that uses a combination of new RL techniques and iterative data collection to achieve remarkable performance on mathematics benchmarks with just a 7B model.
📄 Mentioned papers: arXiv 2402.03300
1:28:175K views · 1y ago · 1:28:17
TabNine Code Completion (Referral) YouTube BitChute Minds Parler BiliBili If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
36:1548K views · 1y ago · 36:15
Guest: Byte Latent Transformer
This paper does away with tokenization and creates an LLM architecture that operates on dynamically sized "patches" instead of tokens. By controlling the patch size, they gain a level of control over the tradeoff between model size and FLOPs and use that to achieve more favorable scaling behavior than classically tokenized LLMs.
48:5313K views · 1y ago · 48:53
This paper demonstrates in a series of experiments that current safety alignment techniques of LLMs, as well as corresponding jailbreaking attacks, are in large part focusing on modulating the distribution of the first few tokens of the LLM response.
📄 Mentioned papers: Paper on openreview.net
28:2319K views · 1y ago · 28:23
Guest: Tokenized Model Parameters
A deep dive into the TokenFormer and an opinion about its impact, novelty, and relation to prior work. Paper Abstract Transformers have become the predominant architecture in foundation models due to their excellent performance across various domains. However, the substantial cost of scaling these models remains a significant concern. This problem arises primarily from their dependence on a fixed number of parameters within linear projections.
📄 Mentioned papers: arXiv 2410.23168
37:0621K views · 2y ago · 37:06
This paper (by Apple) questions the mathematical reasoning abilities of current LLMs and designs a synthetic template-based dataset distribution to investigate various aspects around LLM performance of high-school level math questions.
📄 Mentioned papers: arXiv 2410.05229
27:4860K views · 2y ago · 27:48
This paper posits the interesting question: How much of the performance of Mamba, S4, and other state-space-like models is actually just attributable to some very core concepts - rather than their elaborate architectures. The authors construct minimal versions of GRUs and LSTMs and report competitive performance.
📄 Mentioned papers: arXiv 2410.01201
53:0231K views · 2y ago · 53:02
How can one best use extra FLOPS at test time? Paper Abstract Enabling LLMs to improve their outputs by using more test-time computation is a critical step towards building generally self-improving agents that can operate on open-ended natural language.
📄 Mentioned papers: arXiv 2408.03314
1:03:5618K views · 2y ago · 1:03:56
Guests: Privacy Backdoors, Corrupted Pretrained Models
Can you tamper with a base model in such a way that it will exactly remember its fine-tuning data? This paper presents a method of doing exactly that, and implements it in modern transformers. OUTLINE Paper Code Abstract
Topics: Core idea: single-use data traps · Backdoors in transformer models · Additional numerical tricks · Experimental results & conclusion
📄 Mentioned papers: arXiv 2404.00473
49:4535K views · 2y ago · 49:45
Matrix multiplications (MatMuls) are pervasive throughout modern machine learning architectures. However, they are also very resource intensive and require special accelerators (GPUs). This paper explores architectures that do away with MatMuls and use quantization and recurrence to keep performance up.
Topics: MatMul is everywhere · Ternary accumulation as a substitute for matrix multiplication · Replacing attention layers with recurrent layers · Replacing dense layers with ternary channel mixing · Language modelling results & scaling laws · Other experimental results
📄 Mentioned papers: arXiv 2406.02528
1:11:5842K views · 2y ago · 1:11:58
An in-depth look at a recent Stanford paper examining the degree of hallucinations in various LegalTech tools that incorporate LLMs. OUTLINE Paper Abstract Legal practice has witnessed a sharp rise in products incorporating artificial intelligence (AI). Such tools are designed to assist with a wide range of core legal tasks, from search and summarization of caselaw to document drafting.
Topics: What are legal research tools and how are large language models used by them? · Overview and abstract of the paper · What is a hallucination and why do they occur? · What is retrieval augmented generation (RAG)? · Why LLMs are a bad choice when reasoning is involved · The products that were tested
📄 Mentioned papers: arXiv 2405.20362
57:0045K views · 2y ago · 57:00
xLSTM is an architecture that combines the recurrency and constant memory requirement of LSTMs with the large-scale training of transformers and achieves impressive results. Paper Abstract In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM).
📄 Mentioned papers: arXiv 2405.04517
29:2233K views · 2y ago · 29:22
After the release of their flagship model GPT-4o, OpenAI finds itself in multiple controversies and an exodus of senior personnel - notably Ilya Sutskever References Homepage YouTube If you want to support me, the best thing to do is to share out the content :)
📰 Mentioned news: Article on vox.com · Article on forbes.com
33:2626K views · 2y ago · 33:26
Authors: Jiwoo Hong, Noah Lee, James Thorne Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
📄 Mentioned papers: arXiv 2403.07691
39:1429K views · 2y ago · 39:14
If you want to support me, the best thing to do is to share out the content :)
Topics: Our next-generation Meta Training and Inference Accelerator · ALOHA Unleashed · Apple Inks $50M Deal with Shutterstock for AI Training Data · OpenAI Researchers, Including Ally of Sutskever, Fired for Alleged Leaking · Adobe's Ethical Firefly AI was Trained on Midjourney Images · Trudeau announces $2.4billion for AI-related investments
📄 Mentioned papers: arXiv 2404.07839 · arXiv 2404.07965 · arXiv 2404.05719 · arXiv 2404.12387
📰 Mentioned news: Article on microsoft.com
37:0139K views · 2y ago · 37:01
Abstract While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences.
📄 Mentioned papers: arXiv 2404.09173
17:4741K views · 2y ago · 17:47
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
Topics: Debunking Devin: "First AI Software Engineer" Upwork lie exposed! · NeurIPS 2024 will have a track for papers from high schoolers. · Opus can operate as a Turing machine. · An AI-Powered, Self-Running Propaganda Machine for $105 · TechScape: How cheap, outsourced labour in Africa is shaping AI English · Is ChatGPT Transforming Academics' Writing Style?
▶️ Mentioned videos: Video on youtube.com · Video on youtube.com
📄 Mentioned papers: arXiv 2402.05120 · arXiv 2404.08627
📰 Mentioned news: Article on news.ycombinator.com · Article on wsj.com · Article on theguardian.com
37:1760K views · 2y ago · 37:17
Guest: Leave No Context Behind
Google researchers achieve supposedly infinite context attention via compressive memory. Paper Abstract This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention.
📄 Mentioned papers: arXiv 2404.07143
31:1947K views · 2y ago · 31:19
Meta's Llama 3 is out. New model, new license, new opportunities. References Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
18:0131K views · 2y ago · 18:01
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
9:5521K views · 2y ago · 9:55
Some updates from industry in the Machine Learning world Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
27:3225K views · 2y ago · 27:32
A flurry of new models continues to appear. Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
56:16114K views · 2y ago · 56:16
Flow matching is a more general method than diffusion and serves as the basis for models like Stable Diffusion 3. Paper Abstract We introduce a new paradigm for generative modeling built on Continuous Normalizing Flows (CNFs), allowing us to train CNFs at unprecedented scale. Specifically, we present the notion of Flow Matching (FM), a simulation-free approach for training CNFs based on regressing vector fields of fixed conditional probability paths.
📄 Mentioned papers: arXiv 2210.02747
44:0537K views · 2y ago · 44:05
Abstract While Transformers have enabled tremendous progress in various application settings, such architectures still lag behind traditional symbolic planners for solving complex decision making tasks. In this work, we demonstrate how to train Transformers to solve complex planning tasks and present Searchformer, a Transformer model that optimally solves previously unseen Sokoban puzzles 93.7% of the time, while using up to 26.8% fewer search steps than standard A∗ search.
📄 Mentioned papers: arXiv 2402.14083
27:0034K views · 2y ago · 27:00
Guest: AI Act
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
Topics: XAI releases Grok-1 · Nvidia GTC · Comment of the Week · Brute-forcing OpenAI model names · Inflection AI gets eaten by Microsoft · EU AI Act moving forward
📄 Mentioned papers: Paper on spectrum.ieee.org · arXiv 2403.09611
📰 Mentioned news: Article on techcrunch.com · Article on techcrunch.com · Article on axios.com · Article on techcrunch.com
26:5052K views · 2y ago · 26:50
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
Topics: Devin: AI software engineer · Mira Murati on Sora training data · Inflection accused of copying Claude · Tools & papers · GPT-4.5-turbo mystery · US government report: total extinction by AI
📄 Mentioned papers: arXiv 2403.06634 · arXiv 2403.04652
📰 Mentioned news: Article on bloomberg.com · Article on bloomberg.com · Article on bloomberg.com · Article on bloomberg.com
53:1532K views · 2y ago · 53:15
Guest: More Gemini Drama
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
Topics: Elon sues OpenAI · Mistral Large · ML Espionage · More Gemini Drama · Copilot generates spicy images · Gemma bugs
15:1244K views · 2y ago · 15:12
No, Anthropic's Claude 3 is not conscious or sentient or self-aware. References Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
📰 Mentioned news: Article on lesswrong.com
42:3442K views · 2y ago · 42:34
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
Topics: Gemma & Gemini · Groq · Nvidia EOS Supercomputer · Gpulist.ai · Demis Hassabis on scale · Hardware wars
📄 Mentioned papers: Paper on nature.com
📰 Mentioned news: Article on arstechnica.com · Article on arstechnica.com · Article on businessinsider.com · Article on techcrunch.com
17:3654K views · 2y ago · 17:36
Google turned the anti-bias dial up to 11 on their new Gemini Pro model. References Homepage YouTube If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this)
50:0364K views · 2y ago · 50:03
V-JEPA is a method for unsupervised representation learning of video data by using only latent representation prediction as objective function. Weights & Biases course on Structured LLM Outputs OUTLINE Blog Paper Abstract This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders,…
Topics: Predictive Feature Principle · Weights & Biases course on Structured LLM Outputs · The original JEPA architecture · V-JEPA Concept · V-JEPA Architecture · Experimental Results
1:23:5932K views · 2y ago · 1:23:59
Your regularly irregular dose of Machine Learning News! W&B Course on LLM Structured Outputs OUTLINE References (they were too long for this description) Homepage YouTube If you want to support me, the best thing to do is to share out the content :)
Topics: OpenAI Sora · Gemini 1.5 with 1 Million Tokens context window · V-JEPA · Sam Altman raises 7 TRILLION dollars for AI chips · Bard becomes Gemini · GOODY-2: The world's most responsible model
54:2430K views · 2y ago · 54:24
LUMIERE by Google Research tackles globally consistent text-to-video generation by extending the U-Net downsampling concept to the temporal axis of videos. OUTLINE Paper Abstract We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis.
Topics: Problems with keyframes · Space-Time U-Net (STUNet) · Extending U-Nets to video · Multidiffusion for SSR prediction fusing · Stylized generation by swapping weights · Training & Evaluation
📄 Mentioned papers: arXiv 2401.12945
35:2740K views · 2y ago · 35:27
AlphaGeometry is a combination of a symbolic solver and a large language model by Google DeepMind that tackles IMO geometry questions without any human-generated trainind data. OUTLINE Paper Abstract Proving mathematical theorems at the olympiad level represents a notable milestone in human-level automated reasoning1,2,3,4, owing to their reputed difficulty among the world’s best talents in pre-university mathematics.
Topics: Problem Statement · Core Contribution: Synthetic Data Generation · Sampling Premises · Symbolic Deduction · Traceback · Auxiliary Construction
📄 Mentioned papers: Paper on nature.com
34:3267K views · 2y ago · 34:32
Authors: Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed
Topics: Mixture of Experts · Classic Transformer Blocks · Expert Routing · Sparse Expert Routing · Expert Parallelism · Experimental Results
📄 Mentioned papers: arXiv 2401.04088
3:4013K views · 2y ago · 3:40
If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this) Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
Showing the 48 most recent videos. Everything plays right here on The Pod Atlas.
Have thoughts about Yannic Kilcher? Share them with other learners.
Write a review← Back to all educational channels