Breaking Down Muse, Xbox's Generative AI Push

Every tech company in the world is angling towards incorporating artificial intelligence into its services and line of products in one way or another. Years after its parent company Microsoft invested big in the ubiquitous and divisive technology, it seems that Xbox is following suit with a generative AI of its own.

Microsoft Gaming executives announced Muse, a generative AI model made for games. Muse is an AI model that has a “detailed understanding of the 3D game world, including game physics and how the game reacts to players’ controller actions,” according to the tech company’s official announcement. The company says Muse can “create consistent and diverse gameplay rendered by AI” which can “empower game creators.”

The model is the result of a partnership between Microsoft Research and Xbox Game Studios developer Ninja Theory (Hellblade 2). It was trained on seven years' worth of human gaming sessions of Ninja Theory’s 2020 multiplayer game Bleeding Edge. The findings of the two teams’ research were published in the journal Nature.

How Muse works is by generating images of a game that reflect a player’s input. In the case of Bleeding Edge, for example, a player can press the inputs required to sprint forward and jump over a gap in the environment, and the Muse will reflect that in a series of AI images. In theory, this kind of tool can then be used by developers to quickly iterate on ideas such as object placement in a level.

Microsoft is far from the only company looking into applying AI to the gaming space. Tech billionaire Elon Musk tweeted as early as last November that he intends to launch a gaming studio that uses AI to guide development.

Where Microsoft’s tech stands now, the trade-off of generating these images will be in its quality. Muse will generate images in an extremely choppy and fuzzy 300 × 180 resolution. It’s a long way from the high-definition standard of video games today. However, Microsoft’s top brass believes there to be at least one potential use case for primitive tech: game preservation.

“You can imagine a world where from gameplay data and video that a model can learn old games and really make them portable to any platform where these models can run,” Microsoft Gaming CEO Phil Spencer said during a 7-minute introductory video about Muse. “These models and their ability to learn completely how a game plays without the necessity of the original engine running on the original hardware opens up a ton of opportunity.”

While Xbox could certainly leverage this technology for something altruistic as preserving old games, it’s tough to believe in such a mission without considering the potential negative impact it can have on an already strained industry. Thousands of developers have lost their jobs in recent years, regardless of how their games perform, as corporations endlessly pursue ways to reduce financial expenditure.

By Xbox’s admission, Muse is nowhere near replacing the critical human touch that makes video games special. However, if the day comes when Muse can somehow properly recreate old games from thousands of hours of gameplay footage, it's tough to imagine the games industry won’t follow the lead of other industries (like film, social media, education, and journalism) AI has affected.

And that’s before getting to the fact that the proposed use cases for Muse seem nebulous at best. The viability of feeding an AI model replicating the design decisions, gameplay feel, and broader experience-based expertise of trained professionals isn’t quite as exciting as real people doing real development, regardless of its use case. As a player, I’m not interested in a game company pouring hundreds of millions of dollars into a technology that might be able to replicate an old game 12 years from now when that money could be used to support the people who are hoping they’ll have their job 12 months from now.

I’m far from a practiced developer. There’s a strong chance that developers outside of the ones who’ve already turned their noses at Microsoft’s grand reveal see potential in this technology. And ultimately, Microsoft says that for now, its usage will come down to an individual development team’s choice.

“We have empowered creative leaders here at Xbox to decide on the use of generative AI,” a Microsoft spokesperson told Inverse. “There isn’t going to be a single solution for every game or project, and the approach will be based on the creative vision and goals of each team.”

But as a player, this was far from a convincing rollout. Even its best-proposed use case seems unnecessary when there are cheaper, more reliable alternatives in need of support.

Muse will likely continue to develop regardless of the public’s response to it. But like so many AI technologies proposed before it, Muse seems like an elaborate, expensive solution in search of a problem.