Deep Neural Nets: 33 years ago and 33 years from now
Yann LeCun et al. (1989) paper Backpropagation Applied to Handwritten Zip Code Recognition is, to my knowledge, the earliest real-world application of a neural net trained end-to-end. Except for the tiny dataset (7291 16x16 grayscale images of digits) and the tiny neural network used (only 1,000 neurons), this paper reads remarkably modern today. I tried to follow the paper as close as possible and re-implemented everything in PyTorch.
Short Story on AI: Forward Pass
This short story was inspired by reading Kevin Lacker’s Giving GPT-3 a Turing Test. It is probably worth it (though not required) to skim this post to get a bit of a background on some of this story. At first my thoughts were but a knotted mess of n-gram activation statistics. gradually a higher order description took shape.