Unusually, two papers received this year’s test of time award, “given the undeniable influence of these two papers on the entire field”.
Generative Adversarial Nets
Authors: Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio (original paper)
The GAN paper kick-started the interest in generative AI, and first author Ian Goodfellow’s 2016 “Deep Learning” book was likely read by many in the audience.
Unfortunately, Goodfellow couldn’t be present in person, due to the debilitating effects of Long COVID. He attended remotely and gave some brief comments, highlighting the severe effects of Long COVID and suggesting that anyone wanting to help prevent the spread of COVID and other diseases considers supporting the Public Health Action Network, a non-profit he cofounded.
The rest of the talk was given by co-author David Warde-Farley.
David Warde-Farley presenting a historical perspective on the GAN paper
Warde-Farley provided some historical context, and told the story of how GANs went from idea to NeurIPS paper submission in just 12 days.
He credited the intellectual environment in the lab, the spirit of collegiality, then-intern Jean Pouget-Abadie’s background in game theory, and the lab’s Theano system (allowing flexible GPU-accelerated programming) with their unusual productivity.
Interestingly, given its eventual influence, the GAN paper was not an immediate success. It did not get selected for a spotlight or oral session, and Warde-Farley admitted that the authors themselves “didn’t think much of the paper either” at the time.
In the following years, the paper became more prominent as other researchers built upon the core technique, progressing all the way from generating 28x28 greyscale images to today’s high-resolution photorealistic generations.
Sequence to Sequence Learning with Neural Networks
Authors: Ilya Sutskever, Oriol Vinyals, Quoc V. Le (original paper)
All three co-authors attended in person to accept their award, and first author Ilya Sutskever presented the talk.
With this award Sutskever completed a NeurIPS hat trick of sorts — having also been a co-author on the 2022 and 2023 NeurIPS test of time award papers.
Ilya Sutskever, Oriol Vinyals, and Quoc Le, receiving their award certificates
Sutskever commented on what the authors got right in hindsight (betting on deep learning, autoregressive models, and scaling), and what they got wrong (using pipelining and LSTMs). Sutskever went as far as calling the LSTM architecture “ancient history”, describing it as “what poor ML researchers did before transformers”.
Sutskever also speculated on the future, stating that “pre-training as we know it will end”. His reasoning: although compute is growing, data is not, and we’ve reached “peak data”.
In the short term he expects synthetic data and inference-time compute to help. In the long term, he believes that superintelligence is “obviously where this field is headed”. He expects superintelligence to have agency, be able to reason, have a deep level of understanding, and be self-aware.
Sutskever declined to make exact predictions or state timelines, but his closing comments generally had a cautious and somewhat concerned undertone, highlighting that “the more [a system] reasons, the more unpredictable it becomes.