Papers and venues

It abstracts the neuron into a logic unit that only "fires once it reaches a threshold," and proves that networks built from such units can express any propositional logic. It was the first time "what the brain does" and "what a computer can compute" were put into one mathematical framework. Both neural networks and symbolic logic branch off from here.

Swap the unanswerable question "can machines think?" for a test you can actually run: if you can't tell the machine from a human in a text conversation, stop arguing about the definition. The end of the paper also predicts machines that learn, at a time when not a single learning program existed yet.

In McCulloch–Pitts units, a person sets the weights. The perceptron learns the weights from examples, comes with a convergence proof, and was also built as hardware. It was the first real learning classifier, and it was the one that Perceptrons knocked down ten years later.

The book proved mathematically that a single-layer perceptron cannot represent even a simple function like XOR, and nobody knew how to train multiple layers. It pushed funding and talent toward the symbolic camp, and neural network research went quiet for more than a decade. Its argument was only worked around once backpropagation became widespread.

It applies the chain rule to multi-layer networks, so the weights of the hidden layers can also be adjusted by the error. It also shows that the hidden layers learn useful internal representations on their own. It answers the question Perceptrons left open: how do you train more than one layer? It is the technical starting point of the second wave.

It turns classification into finding the maximum-margin hyperplane, then uses the kernel trick to bring linear methods into a high-dimensional feature space, with generalization theory to back it up. It was the most reliable classifier of the 2000s, and deep learning only clearly beat it on large-scale vision tasks in 2012.

When a recurrent network backpropagates through time, the gradient vanishes or explodes, so information from far back is not learned. LSTM uses gated units to open a channel that lets information flow stably over many steps. Nearly all speech recognition and neural translation in the 2010s relied on it. Before the Transformer appeared, it was the default answer for sequence modeling.

It combined convolution, pooling and gradient training into one end-to-end handwriting recognition system, and it was actually deployed to read checks. It showed that putting priors into the network structure (local receptive fields, weight sharing) instead of into rules was workable. AlexNet is just a scaled-up version of it.

The problem with n-gram language models is that words have no similarity to each other, so any combination never seen in training gets probability zero. This paper has each word learn a continuous vector, then uses a neural network to predict the next word, so similar words share statistics automatically. Both word vectors and today's LLM pretraining objective come from here.

Multi-layer networks were thought to be untrainable in the 2000s. This paper used layer-by-layer unsupervised pretraining to give a deep network a good starting point, then fine-tuned the whole thing. It showed that "deep" could be trained. It made "deep learning" a term and brought funding back to neural networks. Later people found that with ReLU, GPU and big data, the pretraining step could be dropped.

A deep convolutional network trained on two consumer-grade GPUs cut the error rate on the ImageNet challenge by ten percentage points in one step. The runner-up still took the old route of hand-built features plus SVM classifiers. This is the moment deep learning took off: the methods had existed for a long time, but the dataset and the GPU were the two things that only came together in 2012.

A convolutional network reads screen pixels directly and outputs action values. The same network and hyperparameters play dozens of Atari games. It joined deep learning to reinforcement learning, and it was the most convincing demo DeepMind had before Google acquired it.

Cut the language model down to one shallow prediction task. In return, you can train word vectors on a billion words, and vector subtraction gives analogies like "king − man + woman ≈ queen". It made "pretrain representations on unlabeled text first" standard practice in NLP. It was the most important round of pretraining before BERT.

Train a generator and a discriminator against each other, and the model learns to produce realistic samples without ever writing down an explicit likelihood function. It started a whole line of deep generative models (image synthesis, style transfer, deepfakes), until diffusion models replaced it after 2021.

One LSTM compresses the whole sentence into a vector. Another LSTM decodes the translation from that vector. The model trains end to end, with no word alignment and no grammar rules. It turned neural machine translation from a paper into a system you could ship. It also exposed a bottleneck: a fixed-length vector cannot hold a long sentence. The attention mechanism was invented to fix that bottleneck.

When decoding each word, the model no longer looks at a single compressed vector. It looks back and gives every position in the source sentence a weight, then takes the weighted sum. This is attention. It fixed the drop in quality that seq2seq models had on long sentences. Three years later, the Transformer pushed the idea of "keep only attention" to the extreme.

A survey written three years after AlexNet by three future Turing Award winners. It covers why learning representations layer by layer beat hand-built features, what convolutional and recurrent networks each solve, and why unsupervised learning would be the next step. It is the people who did the work summing up the second wave themselves.

When a network gets deep, it stops training well. That isn't overfitting. The optimizer just can't make progress. Residual connections let each layer learn a correction relative to the previous layer, so a network with over a hundred layers still converges. It was the first to beat human annotators on ImageNet. Residual connections have been standard in every deep network since, and every Transformer block has one.

A policy network and a value network prune the Monte Carlo tree search. Self-play reinforcement learning then trains the networks stronger and stronger. In March 2016 it beat Lee Sedol 4:1. Go was thought to be another decade away. It changed the public image of deep learning from "recognizing images" to "making decisions".

It removed all the recurrence and convolution from the sequence-to-sequence models of the time and kept only attention. In exchange, training could be fully parallel. On machine translation it set new results with less training time. It is where the three waves begin: GPT, BERT and ViT are all variants of this skeleton. See the node at Transformer.

Many tasks have no reward function you can write down, but a person can compare two pieces of behavior and say which is better. This paper trains a reward model on those pairwise comparisons, then uses it for reinforcement learning. People only need to look at less than one percent of the interactions. It is the prototype of RLHF. Five years later it was carried over almost unchanged to language models and became InstructGPT.

It trains a bidirectional Transformer encoder with the objective of masking some words and guessing them. After fine-tuning, it swept GLUE and SQuAD. It made the pretrained model the default starting point for every NLP task. It also pushed benchmarks like GLUE past human level within a year.

First train a decoder-only Transformer as a language model on a large amount of unlabeled text, then fine-tune it for each downstream task. One model family got the best results at the time on several NLP benchmarks. It set the "generative pre-training + fine-tuning" route, and the GPT series name starts here.

It scales GPT-1 up by ten times and trains only on web text. With no fine-tuning at all, it can do summarization, translation, and question answering. It was the first sign that once a language model is big enough, the tasks are learned along the way. The staged release of the weights also made the safety of model releases a public issue for the first time.

The lesson of seventy years: every time people write human knowledge into a system, it works in the short term. In the long term, it loses to general methods built on raw compute. Chess, speech and vision all followed this script. It is the shortest note on the three waves, and the creed of this scaling wave.

175 billion parameters, and not a single weight updated. Just put a few examples in the prompt and it does a new task. It made the prompt a way of programming, and it was the first time scaling laws paid off in public view. The two years of LLM startups that followed all grew up around its API.

Language model loss falls as a power law in parameter count, data size and compute, each on its own. This holds across seven orders of magnitude, and model shape barely matters. It turns "should we spend ten times more compute" from a matter of faith into something you can calculate. The investment logic of the three waves' third wave rests on this curve.

It predicts a protein's 3D structure directly from its amino acid sequence, at an accuracy on par with experimental methods. This largely solved a biology problem that had stood for fifty years. It marked deep learning moving beyond language and images into scientific discovery, and it won the 2024 Nobel Prize in Chemistry.

Contrastive learning on 400 million image-text pairs puts images and text into the same vector space, so it can do zero-shot classification without labels. It is the eye that text-to-image models use to understand text, and it is the common starting point for multimodal models.

Put a few examples with reasoning steps in the prompt, and a large model's scores on arithmetic and commonsense reasoning jump by a lot. Small models get nothing from it. It turned "make the model think before it answers" into a technique, and it set off the argument over whether "emergent abilities" are real. Two years later, reasoning models moved this from the prompt into training.

With the same compute, parameters and training tokens should grow in equal proportion. Earlier large models were mostly fed too little data. The 70B-parameter Chinchilla beat the 280B Gopher by using four times the data. It rewrote the scaling recipe. After LLaMA, "smaller models fed more data" became the mainstream approach.

It uses a set of written principles to have the model critique and revise its own answers. Then it uses preferences given by AI to replace most human harmlessness labels. This cuts the reliance on human labeling. It also makes "which values the model is aligned to" a document you can read, for the first time.

A pretrained model only continues text. It does not follow instructions. First do supervised fine-tuning on demonstrations. Then train a reward model on human preferences and optimize with PPO. The 1.3B-parameter result was preferred by labelers over the original 175B model. This three-step recipe is the base of ChatGPT, and it is the reference point for all the "alignment" work that followed.

It moves the diffusion process from pixel space into a compressed latent space. That cuts the cost of training and generation by an order of magnitude, so it runs on a consumer GPU. Stable Diffusion, with its open weights, is this paper in practice. Image generation no longer belonged only to companies with the compute.

RLHF's reward model and PPO steps can be collapsed into a single classification loss, trained directly on preference pairs with supervised learning. The results hold up and the implementation is only a few dozen lines. It moved preference alignment from big-lab engineering to everyday work in the open-source community.

Multimodal input, top 10% on professional exams such as the bar exam, and the report says outright that it does not disclose parameter count, architecture or data. It is both a step up in capability and the point where frontier labs moved from "publishing papers" to "shipping products"; the "technical report" replaced the paper from then on.

Trained only on public data, and following the Chinchilla approach of training a smaller model for longer, the 13B-parameter model beat GPT-3 on most benchmark tests. The weights leaked within a week. Half a year later Llama 2 was officially opened for commercial use. The whole open-source ecosystem of llama.cpp, quantization and local inference grew out of it.

Reinforcement learning with only verifiable rewards, and no demonstrations: the model grows long chains of thought and self-checking on its own. The weights are open and the training cost is public. It shows that an o1-style reasoning model is not one lab's secret, and it puts the new scaling axis of inference-time compute in the hands of the open-source community.

The annual meeting of the Association for Computational Linguistics, the home conference of NLP. BERT was published at its North American chapter, NAACL. After LLM took off, a lot of its topics moved to NeurIPS and ICLR.

Where the third wave is really published: papers go on arXiv first and then to a conference. Since GPT-2, many frontier-lab "technical reports" are never submitted to a conference at all. The "first-version date" on the timeline refers to this.

The top conference for computer vision. ResNet and latent diffusion models were published here. From 2012 to 2016, vision was the main battlefield of deep learning, so reading its papers is reading the history of those years.

Bengio and LeCun founded it in 2013 specifically for "representation learning", with open review. word2vec and the attention mechanism were papers at its first and second editions. A conference of the deep learning era.

The longest-running machine learning conference (1980). It leans toward methods and theory. CLIP was published here.

An open-access journal founded in 2000 after the editorial board of Machine Learning left together. Bengio's neural language model (2003) was published here.

The landmark outlet for AI entering mainstream science: backpropagation in 1986, the deep learning review in 2015, AlphaGo in 2016 and AlphaFold in 2021 were all published here.

The top machine learning conference, held every December since 1987. AlexNet, GAN, seq2seq, Transformer and GPT-3 were all published here. Read a year of NeurIPS papers and you can guess the products two years out.