Text-to-world: Generative AI for Worldbuilding

With Kayla Comalli (Lovelace Studio, Nyric)

Lovelace Studios is building Nyric, a world-generation platform that starts with a text prompt and brings entire worlds to life. I spoke with Kayla Comalli (co-founder and CEO) about how they're doing this.

This is a great episode for anyone who is interested in learning how you can combine generative AI technologies like ChatGPT, Polyhive, Sloyd and Convai -- along with Unreal's Nanite and Lumen capabilities to speak entire worlds into existence.

The Show Notes (below) will introduce you to some of the technologies involved, or tune-in to our entire conversation below:

If you prefer to listen to an episode instead, you can find it here on Spotify—but this is one which is watching the video for, because there’s a lot of interesting clips showing the generative technologies used in Nyric. Text-to-world with Kayla Comalli on Spotify.

Show Notes

You can visit Lovelace Studios at their website.

I broke-out the product demo of Nyric into its own short clip: text-to-world with Nyric. Here’s the clip:

Loading video

Technologies used in Nyric:

• Unreal Engine — Nanite and Lumen

• Sloyd - for parametric meshes

• Polyhive - for generative texture maps

• Convai - for conversational AI (characters)

• Beamable (virtual world backend)

Unreal Engine

Even if you can get 3D objects out of a generative system, it is still complicated to automatically integrate them into a world: in the past, this might require a lot of 3D model optimization and setting up (and baking) lights. With Unreal 5’s Lumen and Nanite technology, these steps no longer need to be part of the developers workflow. This software-based raytracing technology is enabling more dynamic 3D object utilization.

The importance of physics-based rendering (i.e., software ray tracing) is something I covered in my article, The Direct from Imagination Era Has Begun.

Parametric Meshes: Sloyd

Sloyd creates 3D objects using a library of meshes that have a huge number of parameters, allowing you to vary and adjust the way they look in a particular setting.

Generative Texture Maps: Polyhive

Polyhive takes text-prompts and generates texture maps that can be applied to 3D meshes, such as this example shirt below:

Conversational AI: Convai for Characters

Convai is a technology that lets you design characters to bring into your game world.

Beamable

Nyric is also using Beamable (my on company) for live ops, virtual world persistence and microservices to orchestrate the various generative AI calls.