Scaling AI is Hitting a Dangerous Wall

At Microsoft Ignite 2024, Satya Nadella addressed the growing scepticism surrounding AI scaling laws, likening them to the long-held but not eternal truth of Moore’s Law. “These are not physical laws; they are empirical observations,” Nadella noted, suggesting the debate could spur much-needed innovation in areas like model architecture and test-time computing.

Microsoft is betting on OpenAI’s o1 scaling law, ‘test time scaling’, a compute-heavy method reshaping efficiency metrics. Nadella introduced “tokens per watt plus dollar” as the new standard for measuring AI performance, hinting at the critical need for sustainable AI growth.