It’s Now or Never for Meta

Meta’s LLaMA is in a tight spot. It’s not only facing threats from closed source OpenAI, Gemini and others, but fighting it out on the open source front as well from the likes of Falcon 180B.

Meta is not oblivious to the competitive landscape. According to several leaks on discussion platforms, it has been working on Llama 3, and there have been diverse expectations and predictions around it.

The developer ecosystem anticipates that Llama 3 will have an infusion of high-quality training data, perhaps something akin to Phi 1.5, to catapult its performance to new heights. Excitement is high concerning the potential expansion of tokens and further forays into exploring the scaling laws. Another hot topic revolves around the concept of Mixture-of-Architecture, a statistical approach poised to address the shortcomings of parametric architecture, possibly surpassing individual experts or submodels.