Danqi Chen’s invited talk tackled a question that’s been top of mind for the academic ML community lately: in an era where frontier language model research makes use of enormous compute budgets, what is academia’s role?
Chen started by pointing out that all three of this year’s ICLR outstanding paper award winners present language model research done by academic labs, demonstrating that academia is still doing something right in this space, and highlighted some areas in which academia has important contributions to make:
- Understanding what works (and what doesn’t)
- Writing good papers (with openness and scientific rigour)
- Developing novel and compute-efficient solutions
- Contributing great open models to the broader community
Danqi Chen
More specifically, Chen identified three promising areas and highlighted some research in each:
- Working on small language models (1-10B parameters), and leveraging techniques like pruning and distillation
- Curating data for language model training, and developing an understanding of the impact of data quality on scaling laws
- Post-training open-weights models — including holistic, controlled studies on post-training research, taking into account that different base models can give different results.
Chen called out direct preference optimisation (DPO) as a “major win for academic research”, enabling computationally efficient post-training that is more accessible than the previous reinforcement learning from human feedback (RLHF) methods.
A recurring message throughout this talk was that there are many aspects of the language model training pipeline which are still poorly understood, and there is plenty of impactful research to be done within academic compute budgets (100-1,000 GPU hours) that can contribute to improving our shared understanding.