After building transformer implementations to understand how LLMs work, I got curious about image generation. How does a model turn “a cat wearing a tiny hat” into an actual picture of a cat wearing a tiny hat? It’s an area with a lot of ongoing research, and I wanted to understand the fundamentals. So Claude and I built text-to-image, an educational project that constructs a complete text-to-image system from scratch in five progressive phases.
I’ve spent the last week or so building two educational projects to understand how transformer models work. It started as an overview, but after I got the high level, I wanted to dig deeper and see all the math.So now you can too…every calculation, every design choice, every gradient flowing backward through the network. The first project is a complete transformer implementation in PyTorch. The second is a step-by-step walkthrough of every single calculation in training a tiny transformer model by hand.
Quite a while ago I heard about this lecture by John Ousterhout — creator of Tcl and general computer science luminary — and it’s been rattling around in my head ever since. It’s a part of his weekend thought series, or so I understand, and the main point was quite straightforward (but also quite thought-provoking): A little bit of slope makes up for a lot of y-intercept. By way of reminder, picture two lines on a graph.
Quite a while ago, I wrote a review of The New Turing Omnibus. A very clever book whose double entendre title reinforced the wit with which the subject matter was to be handled. In short, a very fun book that covers a lot of Computer Science topics. What’s interesting though is how one tiny passage in the foreword changed my outlook on something important. Here it is: Sometime during my childhood I encountered the traditional image of a bird that erodes a mountain by taking a single stone from it every year.
This last week I was sitting with friends around a campfire and told the following true story. They found it rather interesting, so I figured it was worth retelling. The truth is, my career has been a really weird one…there’s nothing linear about any of the progression through work that I’ve chosen. It has been a truly wild ride, but I’m getting ahead of myself. Amazon. 1999. It was a totally different company than it is now, though in many ways it’s probably still Day 1.