Lidong Zhou — Chief Scientist of Microsoft Asia Pacific as well as an ACM Fellow and IEEE Fellow — provided a systems perspective on AI and scaling.
After a brief intro recapping the industry’s progress since the most recent AI winter, Zhou contrasted the AI view on scaling (where it is not unusual to predict scaling laws out several orders of magnitude) with the systems view (“you can’t scale by throwing hardware at the problem forever”). Zhou stated that increasing scale comes with increasing complexity, and requires designing new systems and solving new problems.
The good news: improvements in AI can be used to improve systems themselves, in a symbiotic relationship.
Zhou laid out three core ambitions, each with its own grand challenge. In brief:
- Efficiency: better architectures and modelling approaches are possible (e.g. ternary instead of floating point inference), but to perform best they will need specialised hardware. It’s hard to build that hardware until the architectures are competitive.
- Trust: leveraging formal methods to enable AI to think and reason about systems, making it easier to build trusted and verified systems.
- Infusion: building “AI-infused” systems, harnessing new capabilities to improve systems at the design level.
To dig into some of the research highlights Zhou mentioned, see BitNet (ternary representations) and Verus (formal verification).