Market Map: Generative AI for Virtual Worlds | by Jon Radoff | Building the Metaverse | Feb, 2023

    Published on:

    In the near future, we will speak as if the entire universe exists.

    This article is a guide to companies building generative artificial intelligence technologies that lead to these virtual worlds (games, simulations, metaverse application).

    Existing market maps describing the generative AI landscape lack compelling composition and instead look like random boxes based on features. Most of my readers are interested in the technologies and companies that power games, simulations, metaverse applications, etc., so this map is a great way to chart who is driving these particular experiences forward. Helpful.

    Here’s the market map for version 1.2 (updated 2/3/2023):

    Large companies appear multiple times on the chart if they have significant investments, research, or operations in any category. For smaller companies, I try to keep them in one focused category.

    It details how to interpret the different layers of the value chain used to compose this chart, as well as the highly complex virtual world generation AI.

    Game development provides a perspective on how virtual worlds can be supported by generative AI. There are different types of creators. A world built by a studio (the normal game development mode where one person to her 1000+ builds a virtual world). Modders that extend them. People that populate and create while playing. Even the world itself may be imbued with generative functions.

    Source: Five levels of generative AI for games (click to enlarge)

    the virtual world complicated Larger and more variable internal structures can lead to unexpected behavior. They are not just his three-dimensional world, but a multidimensional world that includes time, social networks, economies, and living stories.

    but they also complexity: In the production process, there are countless jigsaw pieces that are constantly evolving and difficult to fit. This diagram shows just a few of these.

    From: The Direct from Imagination The era has begun

    Even creating a 3D model requires going from concept to modeling, optimization, texturing, UV unwrapping, rigging, animation, composition and lighting. Along the way, you can go back to the early stages and make various improvements. And we need to make this content available to participants in an ever-changing world. All of this requires a high level of expertise and a wide range of support technologies at each step.

    You could add music, sound effects, narration, and other creative pipelines to the diagram above, but adding everything would make the diagram unintelligible.

    Generative AI helps with this configuration aspect, making it easy to link these different tasks with the appropriate verbs required for your workflow. But there are also many missing pieces. The technology to generate ready-to-use 3D models in virtual worlds is currently in its early stages.

    Let’s go back to the market map. Here’s what the categories mean and how they relate to each other:

    • experience These are the playgrounds, applications and virtual worlds most affected by generative AI. To be included here, a company must not only be a product built on the productivity-enhancing aspects of generative technology, but also have generative elements directly “in the loop” of the experience. For example, a game like AI Dungeon is an experience. ChatGPT is no different. ChatGPT is basically an application for playing with GPT-3.
    • discover is a company that makes it easy to discover and connect to content and experiences in virtual worlds. Companies here have a social, community or search side that directly leverages generative AI or supports creators building virtual worlds.
    • Creator Economy is a company that creates tools and configuration frameworks that facilitate the creation of content for virtual worlds. It also includes SaaS or API-driven approaches to enabling AI applications, such as the approach used by OpenAI.
    • spatial computing A company bridging the realms of generative AI technology and 3D environments (model generation, model animation, neural radiance fields, etc.).
    • Decentralization A company that makes AI accessible to the world. While much AI software (such as nearly all of OpenAI) is highly centralized, advances in generative technology are accelerating exponentially and driven by the widespread availability of accessible research and models. This includes open source AI communities (such as Hugging Face), open source models (such as Stability AI’s work), and core open source libraries for generative AI.
    • human interface It is a technology that enables the utilization of AI. On my metaverse market map, this is mostly packaged hardware products such as AR/VR devices. For generative AI, however, it has largely converged on natural language and speech as the simplest human interfaces for a wide range of creative tasks.
    • infrastructure It is a fundamental technology that enables AI. This is the realm of the physical machine. ASML’s chip manufacturing equipment, chip makers like NVIDIA, and companies that deploy networks of equipment.

    The biggest companies in AI are investing heavily to support virtual worlds.

    • NVIDIA Given that we manufacture the most widely used chips in AI, AI technology is a major enabler of all AI technologies. Given their strong background in enabling 3D graphics, it’s no surprise that they do research in most categories related to virtual worlds. Their Omnivese is a platform that serves as a collaborative workspace for his 3D creations, including generative input. Our research across many types of models also allows us to co-develop semiconductors and software like most other companies.
    • meta has research and products in virtually every area, from supercomputing clusters (infrastructure) for training AI models to experience with platforms like Quest that directly benefit from generative technology.
    • Similarly, Google has products in almost every category, from chips to end user experience.
    • Microsoft’s Generative AI today is mostly centered around creator economy technologies that enable others to build applications.It seems likely to expand dramatically, especially given Their investment in OpenAI.
    • apple are the most secretive and rarely publish their research, but their chips now offer world-class AI performance in devices (the phone’s A16 Bionic has 17 TOPS in its neural engine). Do it in 2023!)
    • Open AI is very powerful for certain AI models (especially LLM and images), but is primarily focused on API-oriented systems for the creator economy. ChatGPT is really an end-user application that considers experiences (even virtual worlds) built on top of the underlying model.

    There are other big companies, like Tesla, that invest heavily in AI in general, but I didn’t include them just because I couldn’t identify any that could be applied to virtual worlds (sorry No, but Enabling Steam in the Center Console not counted much). If they make their supercomputing his infrastructure available for third-his party generative use, or if the generative elements of Optimus surface, things could change. Track in detail.

    Decentralized AI is another interesting battlefield to watch. There are companies like Stability whose mission is open source access to models. By contrast, companies like OpenAI, who guard it heavily, are sealed behind their APIs. Big tech companies have so far been hesitant to provide access to pre-trained AI models. However, some of these same companies are making significant contributions to open source software that directly supports distributed AI development. For example, TensorFlow was invented at Google and Meta contributed heavily to his PyTorch. These are the two most popular software libraries for building AI systems.

    Value Chain: Why Are These People ASML machine (infrastructure) Ultimately, the entire universe can be brought into existence (experience)

    A quick digression for the geeks: Gradation It’s a way of understanding how to turn the knobs deep into the network and get amazing results. emergency properties. Similarly, the value chain is just a way of looking at how the nudging of a basic technology affects a network of other interrelated and dependent technologies.That’s why it’s like improvement ASML Advanced Chip Making Machine Ultimately, this means being able to speak to beings in the entire world from your home computer. Understanding the exponential tilt of the market years ahead, not next month, will be key to building a successful R&D and investment strategy. Similarly, seeing where the loss function can be optimized shows the greatest opportunities for value creation.

    There are many startups solving key pieces of the virtual world generation puzzle, but I would like to highlight three in particular.

    • (decentralized): Most people are familiar with Stable Diffusion, a generative AI model for 2D art. Stability is notable for creating an open-source version of the diffusion model and being at the forefront of more decentralized and open AI technologies. They invest in a wide range of models focused on creative industries such as music and audio. All this applies to games and virtual worlds.
    • (creator economy): Create game assets and tweak your own models to help maintain artistic consistency. We will soon release an API that will allow games to generate assets on-the-fly (while the player is experiencing the game, not just pre-preparing it). This is like pushing some games to level 3 in the generative AI hierarchy in the near future.
    • On Your Journey (Creator Economy): produces 2D art that is especially suitable for creating ready-to-use concept art and other assets. These days, I use Midjourney far more than stock photography in all my articles and presentations.

    Scientific research within generative AI is a major driver of new capabilities. Much of the generative AI research funding comes from the industry itself (NVIDIA, Meta, Google, Google and OpenAI are at the forefront). Many also continue to rely on traditional institutional relationships.

    The market map focuses on the role of commercial technology. That is, what has moved from labs to startups and products. Due to the importance of this emerging science, the next article will provide an overview of the state of the art in the areas most relevant to the above topics.

    The application of generative AI to virtual worlds is still in its infancy, but it will grow faster than many expected.

    • The Direct from Imagination Era has Begun describes convergence techniques that give you the ability to “talk the world into existence”.
    • The Five Levels of Generative AI is an article about how to frame the progress of generative technology in virtual worlds (beginning with procedural generation and various forms of automation before generative AI was practical) .
    • The Metaverse market map is the underlying structure that inspired this slice of generative AI, including core technologies, 3D graphics, creative tools, and other people creating experiences such as games.
    • Experiences of the Metaverse provide a guide not only to games, but also to the many applications that may be placed in virtual worlds, such as education, collaboration, and other forms of media.


    Leave a Reply

    Please enter your comment!
    Please enter your name here