At Wednesday morning’s invited talk, Sepp Hochreiter — whose 1991 thesis introduced the LSTM architecture — spoke about Industrial AI.
Sepp Hochreiter
By comparison with historical technologies like the steam engine and the Haber-Bosch process for producing fertiliser, Hochreiter posited three phases of technology: basic development, scaling up, and industrialisation.
For AI research, under this paradigm, basic development included the invention of backpropagation and stochastic gradient descent as well as architectural innovations like CNNs and RNNs. The end of the scaling up phase is now approaching, with resources being focused on a few methods which scale well (ResNets and attention, mainly through the transformer architecture), as variety is compromised for the sake of scaling.
In the final phase — industrialisation — variety is once again introduced, as more application-specific techniques are developed. Hochreiter stated that Sutton’s bitter lesson is over, and that scaling is just one step on the road to useful AI. Echoing François Chollet and Yann LeCun’s views, he also described LLMs as “modern databases”, useful for knowledge representation but providing just one component of a larger and more complicated end system.
As for the next step, Hochreiter introduced some industrial applications of AI — in particular, approximate simulations of physical systems at a scale where numerical methods struggle, to replace traditional computational fluid dynamics or discrete element methods.
Lastly, he mentioned a new architecture developed by his lab, the xLSTM, which addresses some limitations of the original LSTM architecture while scaling better than transformers. More on the xLSTM in the poster session on Thursday evening.
xLSTM poster session