Bootcamp / Session 1 / 1.2
Session 1.2 · Act one · Eric Ma, Moderna
How language models actually work.
One trick, next-token prediction, explained well enough that you can teach it to your own lab and trace any answer a model gives you back to it. No math, no code. Every demo runs a real, tiny model in your browser.
Recording
What the lesson covers
- One trick. A language model is a very well-read autocomplete, predicting the next piece of text.
- Tokens. Text becomes numbers, which is why models miscount letters and garble exact strings.
- Dice and temperature. A probability list, then a weighted roll. Why answers vary between runs.
- Training. Where the numbers come from, and why the model copies flaws in its training text.
- The big library. It interpolates within what it read. It does not look things up, and it stops at a cutoff date.
- Confidently wrong. Hallucination is the mechanism working normally. Quiz: spot the invented references.
- Scale. What changes as models grow, and what does not.
- Foundation models. The same machinery for proteins, genes, and cells: AlphaFold 2, Geneformer, scGPT, Enformer.
- Limits. Where the trick breaks, and the habit that catches each failure.
Carry this into the rest of today
Eric's closing table is the starting point for Part 1.4, where each row gets a practice from Tommy and a rule from us.
| What you see | Cause | Habit that catches it |
|---|---|---|
| It makes something up | Prediction without lookup | Verify against primary sources |
| Same prompt, different answer | Sampling | Log model, settings, and prompts |
| Out-of-date answers | Training cutoff date | Check current sources |
| Repeats the field's old mistakes | Copies its training data | Read with the literature's biases in mind |
| Echoes sensitive text you pasted | Nothing in the mechanism keeps secrets | Keep patient and unpublished data out of consumer tools |
Materials
LessonHow language models actually workThe interactive page used in class.biocore.umassmed.edu/next-token
SourceRun it yourselfThe lesson and its tiny model.
docker compose up, then open port 8000.github.com/UMMS-Biocore/next-token
ReadingLanguage models for biological research: a primerNature Methods, 2024.nature.com
Going deeper3Blue1BrownVisual explanations of neural networks and transformers.youtube.com/@3blue1brown
Going deeperWelch LabsRecommended by Eric for the history and intuition of modern AI.youtube.com/@WelchLabsVideo