Building the Metaverse with Jon Radoff

Perplexity allows you to ask questions and receive answers supported by citations from online sources: Wikipedia-meets-AI. I spoke with Aravind Srinivas, founder and CEO of Perplexity.ai, the company behind it. They use a combination of GPT, good UI/UX and a lot of behind the-scenes magic to equip you with an artificial intelligence that can make you more informed, smarter--and maybe even more compassionate.

Ali El Rhermoul and Jon Radoff's team at the a16z Tech Week AI/Virtual Worlds Hackathon won grand prize: it's a multiplayer online game featuring a persistent world, generative AI for storytelling (Anthropic Claude) and immersive environments (Blockade Labs, Scenario) and a client/server architecture using Beamable + Unity to create a persistent world using a new technique called "semantic programming," which uses an XML-based format to provide consistency between generative and game components.

I spoke with Adam B. Levine, cofounder of Blockade Labs, the company responsible for the generative skybox AI that can create 3D skyboxes from text prompts or sketches. One of the best applications of this technology is created environments for use in game engines like Unity and Unreal. We explored some demos of what this can do covered, and discussed the future of games, creativity and technology.

Beyond the technology and its applications, we discussed the merits of decentralized AI vs. decentralized approaches ] (i.e., open-source models like Stable Diffusion compared to API-centric models like OpenAI).

Voice Synthesis is about to transform game development, virtual worlds and simulations of all kinds. I spoke with Zach Johnson, cofounder of LMNT, an AI voice synthesis company that uses the magic of prosody imitation to make voices that sound strikingly like the original speaker.

We covered how these technologies work; demonstrated cases with David Attenborough and a flight simulator; discussed the new type of creator economy this will form; technologies to enable scalability and reliability (such as memory safe languages, e.g., Rust) as well as other things Zach learned from his time working on Google Glass.

One of the fastest growing areas of creative expression in games has been the emotes and animations you can apply to your character in games like Fortnite. Yassine Tahi saw this realm of self-expression coming, and created a platform that democratizes the creation of character animations. With Kinetix, individual creators can capture movement and make their avatar animations available through a marketplace--and end-users can invoke them through text-to-animation generative prompting. This requires a huge number of challenging technology problems to be solved: motion capture with a normal camera that maps to the character rig; adjusting animation to the geometry and physics of an environment; labeling the data in a way that generative AI technology can make use of it; and then making animations interoperable across different games, virtual worlds and metaverse platforms. Some of the games and platforms we discuss include Zepeto, Inworld, The Sandbox Fortnite and NextDancer.

Lovelace Studios is building Nyric, a world-generation platform that starts with a text prompt and brings entire worlds to life. I spoke with Kayla Comalli (co-founder and CEO) about how they're doing this. This is a great episode for anyone who is interested in learning how you can combine generative AI technologies like ChatGPT, Polyhive, Sloyd and Convai -- along with Unreal's Nanite and Lumen capabilities to speak entire worlds into existence. (Note: this is an episode you may want to watch on Metavert.tv because the clips of the technologies used are really interesting).

Linus Ekenstam is a long-time product leader, AI educator, and the creator of Bedtimestory.ai. Parents use Bedtimestory to create personalized stories for their children. We discussed how language models enable new types of products that deliver higher levels of personalization and new forms of narrative structure; his learnings from product leadership at companies such as Typeform. We also spent time on the big picture: what it means to transition from the Age of Information to the Age of Intelligence; whether Prompt Engineering is an actual job; and how AI might help restore nuance to the currently-polarized conversations that pervade the internet.

Cameron Wills used ChatGPT to make a ChatGPT plugin for Unity, which became #1 in AI, and #2 overall on the Unity Asset Store. We talked about his experience as an indie game developer, his Unity asset for making games with ChatGPT, and the how generative AI is changing the craft of game-making for Indies.

Kris Kashtanova is an AI artist who published the world's first copyrighted AI-assisted comic book. Her journey with the US Copyright Office has been documented in the news. I spoke to her about her work as an artist and photographer, what she learned about copyrighting AI art, the safe community she is building--and some of her newer work around ControlNet and training her own models.

I spoke with Jussi Kemppainen about the future of game development, from the standpoint of a veteran game developer and VFX artist. Jussi has worked at game publishers including Rovio and Remedy. He's currently at Mainframe Industries, working on Pax Dei--a next-gen MMORPG with deep sandbox mechanics in the spirit of Eve Online. During our conversation we covered how to build graphics pipelines from Midjourney all the way into Unity; how generative AI is likely to change the industry, job openings and careers; methods involving projection-mapping; and Jussi's journey as a game developer as well as an indie game-maker (working on his own generative-AI-assisted title, Echoes of Somewhere). He had some words of wisdom for artists and VFX folks looking to get into game development. There's also a ton of hands-on, practical guidance for anyone trying to figure out how to accelerate their creative process using generative AI tools.

My conversation with Edward Saatchi (CEO, Fable Simulation) is about his vision for virtual beings: digital characters that not only have conversations, but act within virtual worlds: living, playing, creating and forming communities. We covered the birth of all the new art forms and media resulting from generative AI, agent AIs, conversational vs. action-oriented characters, the integration of virtual beings into live games and immersive experiences, and inspiration from science-fiction such as the Culture series and the Holodeck from Star Trek.

Emmanuel de Maistre ("Emm") is leading Scenario, which is a platform for creating the art for games using generative AI. One of their unique approaches is to focus on helping developers build their own fine-tuned models based on their own art style--helping to establish aesthetic consistency. During this conversation we spoke about the future of the game industry (across many forms of generative AI beyond art), Emm's background as an entrepreneur, and even got into some generative applications within the sciences & biology.

Hilary is CEO of Hidden Door, a gaming startup that is using language models to create new types of play. Previously, Hilary was head of machine learning for Cloudera; the founder of independent research lab Fast Forward Labs; Chief Scientist of Bitly; and was a professor of computer science at Johnson & Wales. In this conversation, I spoke with Hilary Mason about how language models may change interactive experiences like games, finding new forms of playfulness, details of the enabling technology, emergent behavior -- and even the very nature of intelligence and consciousness.

Do you want to learn more about scaling customer acquisition for an online product? The lessons here will apply whether you are shipping a game, a metaverse experience or any type of mobile application.  Eric Seufert worked at legendary game publishers like Rovio, Wooga, and Digital Chocolate--where he literally wrote the book on Freemium Economics for games. Eric's concern for several years has been scaling games by optimizing unit economics and accessing scalable user acquisition channels. In this conversation, Jon and Eric talk about how the market for mobile user acquisition has changed; how the market for advertising networks has evolved; and even how the strategy for game marketing has shifted from the "Harry Crane" measurement paradigm to the "Don Draper" paradigm which is all about audience resonance. We cover the differences (or lack thereof) in performance vs. brand marketing; how new privacy rules such as ATT (App Tracking Transparency) have impacted the user acquisition markets; and how to staff a user acquisition team at your publisher.  If you're looking for an overview of how to gain access to the mass market of consumers on mobile--both the history, present and possible future--this is the conversation you'll want to hear!

Three-way conversation between Jon Radoff and two amazing technology experts: Keith Adams was Chief Architect for Slack--the pervasive chat platform used by hundreds of millions of people every day (fun fact: did you know that Slack started as a game company?) Ali El Rhermoul works with Jon at Beamable, where Ali is CTO. We covered engineering philosophy, and new technologies like serverless and microservies as paradigms that live games can use to scale up. We also talk about team dynamics, agile approaches, high-velocity iteration, reaching product-market fit, and the values that an engineering team can incorporate to bring curiosity and success to a product endeavor.

Alexander Brazie has built a career working on the design of some of the largest online games: World of Warcraft and League of Legends. We start with the inspiring story of how Alexander landed a career in game design, starting as a teenage game developer. Along the way he's learned how to create amazing, engaging, sustainable game businesses--as well as how the business models to support them have evolved. We talk about MMORPGs, MOBAs, free-to-play (f2p), subscriptions and NFT-based games.

Avatars are a way for you project your identity into virtual space. In the metaverse, this will also mean the ability to transport your identity from place to place, or to take on different identities as your mood or the experience calls for. Along with that is the rise of digital fashion: both a new form of self-expression and a boon to the creator economy for artists and designers. Cathy Hackl and I discuss Roblox, avatar platforms including Ready Player Me, as well as digital fashion platforms such as Fabricant and others who are building within the blockchain/NFT ecosystem.

The metaverse is opening up a new range of opportunities for online jobs: one of these is in the realm of online performance. Emma (known online as "Swebliss") is a Twitch streamer and TikTok personality who has assembled an audience approaching a million people by streaming her gameplay and artistry. How did she do it, what are the opportunities in streaming, and how does one "break in" to this business?  What are some of the challenges and problems with being a streamer today? We also cover the challenges women face around online toxicity. If you're curious about the personal and business side of the live-streaming business, this is your chance to learn more about one of the original "play to earn" businesses in gaming.

Finn is the founder of VR, AR and video game production company Chicken Waffle (creator of VR games including Baby Hands and Shadow of Valhalla); co-founder of virtual music concert company The Wave XR; and VR-based comedy club Failed to Render. I caught up with Finn before he headed down to Art Basel Miami where he planned to show off some of his work in metaverse art.  This conversation covers a lot of ground: we cover how far virtual reality (VR) has come in the last few years, and the opportunities as it expands into AR and new hardware like smartglasses (e.g., Snap Spectacles)--and then get into some of the new experiences that involve art, music and even comedy. We talk about video games, and applications for the blockchain and NFTs to support creators across this ecosystem.  This is a conversation I know you'll enjoy if you are intrigued by any of these topics.

Mitch Zamara is a veteran game designer who started in the free-to-play (F2P) mobile industry at Zynga and is now with Million on Mars, where he's designing an NFT-based game experience based on the colonization of Mars. The past year has seen enormous growth in the play-to-earn (P2E) segment of blockchain gaming. Since there's been plenty of coverage of games like Axie Infinity and NFT-based games in general, Mitch and I decided to the use the hour to dig deeper into issues of game design, as well as opportunities for growth. We talked about the non-blockchain origins of earning money from game-playing--which includes areas like esports, modding, etc.--and discussed how this may inform the future of P2E as it starts to blend the distinction between players and designers. We also discussed some of the game design issues for game designers thinking of making the transition from F2P (or other business models) to P2E.

Diversity & Inclusion--making a more equitable future for everyone to contribute to the metaverse--is the focus of this conversation with Trinidad Hermida. She was the head of Diversity and Inclusion for Niantic, the creators of Pokemon Go. Trinidad is now running her own consulting firm that advises game and technology companies how to bring "equity through empathy" into their organizations. We discussed many of the barriers that underrepresented people find when getting involved in the game industry (as well as technology, more broadly) as well as the reasons why companies ought to rise to the challenge and build bridges to these communities. We also get into Web3 and DAOs and why the newness, community-orientation and lack of institutional barriers may open up more opportunities to people of all backgrounds.

Nicole Lazzaro is a neuroscientist, a leading expert on emotion in games, gamification, and augmented reality. In this episode Jon and Nicole discuss how we can apply the learnings from emotion in games (such as Nicole's "4 Keys 2 Fun" model) to the metaverse, as well as Nicole's work within augmented reality.

James Zhang is the CEO of Concept Art House, which recently sold a Frank Miller NFT for $840K and raised over $25M to expand their business. In this episode of Building the Metaverse, Jon and James talk about the creator economy for NFTs, dispel some common misconceptions, and review the opportunity for artists and collectors.

Evo Heyning talk about intelligent ecosystems, virtual beings, artificial intelligence, AIs--as well as avatars, creativity and decentralized autonomous organizations (DAOs).

In this episode of Building the Metaverse, Jon has a conversation with game designer, entrepreneur and neuroscientist Amy Jo Kim.  This episode focused on game development; this is a key episode to listen to if you're a game developer, but also helpful if you're building any metaverse application inspired by gaming. We discuss the solo-to-social trend; the "games as economies" trend, the technologists-to-artists trend, and the "community first" game development trend. A few games we talk about include Axie Infinity, Roblox and Undertale.

Raph Koster is a legendary game designer behind Ultima Online and Star Wars Galaxies. In this episode, we discuss the metaverse, virtual worlds, governance, game economies, creativity and everything in between.

I discuss the creative platforms like Unity, Roblox and Unreal that are being used to construct the metaverse--and delve into the history of creative technologies overall.

What will you do in the metaverse? I discuss applications including games, esports, social activities, shopping, real estate, travel, education, training, fitness, live music, volumetric live-streaming and immersive theater.

This episode covers the major trends that are shaping the metaverse: digital identity (the mainstream adoption of virtual personas and items); the competition between open standards and closed platforms; artificial intelligence; cybernetics (AR, VR, neural, brain-computer interfaces); blockchain; low code and no code platforms; simulating reality; distributed networks.

In this first episode of "Building the Metaverse," Josh and Jon meet up to discuss what the metaverse is all about. Topics include: how it is the next generation of the Internet; the creator economy; the huge influence of games on what the metaverse is becoming; and the categories of companies who are working towards it.