AI Explained: Tokenization for Non-Math People (Part 2)
Ever wonder why AI makes simple math or spelling mistakes? This video demystifies tokenization, breaking down how AI cuts up English words into puzzle pieces.
Ever wonder why AI makes simple math or spelling mistakes? This video demystifies tokenization, breaking down how AI cuts up English words into puzzle pieces. We’ll simplify complex concepts so you can finally understand token boundaries without needing a math degree. Discover the secrets of how chatbots ‘read’ and process information, giving you a deeper insight into the AI black box.
What’s in the video (5m 36s)
- 0:00 — Introduction: How does a AI actually reads?
- 0:36 — Chapter 1 - The Concept - How AI Reads Language
- 1:18 — What is Tokenization?
- 2:25 — Chapter 2 - The Example - The Strawberry Problem
- 3:38 — Chapter 3 - The Takeaway - Tokens are Money
- 5:06 — Key Recap: How does a AI actually reads?
Resources
- Full AI Explained series: YouTube playlist
- Previous episode: https://youtu.be/etgBDpExVug
- Next episode: https://youtu.be/nXzL1OHLWrY
For more in this series, visit the #ai tag page or jump to the channel uploads list for everything else.
Related posts
Securing AI Agents from Doing Bad Things
Show notes for AI Explained Part 31 — sandboxing, permission scoping, instruction hierarchy, and the metrics that tell you whether your agent is safe to ship.
AI Explained: Finally Understand AI Errors & Human-in-the-Loop (HITL) (Part 30)
Feeling overwhelmed by the fear of AI making huge mistakes? In this video, we break it down into simple pieces.
AI Explained: Semantic Caching & State Management for AI Agents (Part 29)
Feeling overwhelmed by high AI API costs and latency? In this video, we break it down into simple pieces.