Creative Technology and Generative AI with Bilawal Sidhu

Generative AI is accelerating creativity: complicated creative tasks that once took hours can now take only a minute. My conversation with

Bilawal Sidhu

spans the transformative impact of creative tools like Midjourney, Neural Radiance fields, ControlNet, Ebsynth, phone-based photogrammetry, and the potential to "reskin reality" as generative technologies fuse with augmented reality.

We also talk about Bilawal’s journey and success as a creatorpreneur, with words of wisdom for anyone hoping to follow in his footsteps. We finish up with some of the risks and opportunities around generative artificial intelligence, as well as our different viewpoints around centralized vs. decentralized technologies.

If you prefer a podcast to a video, you can also listen to my conversation with Bilawal Sidhu it on Spotify.

Show Notes

This episode covered a wealth of tools, technologies and research paper. I encourage you to watch the video itself to see the highlights, and then dive into the links you see below.

Of course, much of the problem with the current state of creative technologies is the fragmentation of the toolchain:

From Bilawal’s Twitter thread about the fragmentation of the creator technology stack.

Some of the tools and papers we discussed are adding to this complexity in the near-term, but actually significantly reduce the actual time spent from hours to minutes. You’ll learn about some of these generative tools and their background in the notes below.

ControlNet

ControlNet is a tool for use with Stable Diffusion that allows you to pose characters and position objects the way you want in a generated scene.

Example of how you can prompt a dog, and create variations from the same image using ControlNet.

Ebsynth

Ebsynth is a tool for restyling videos according to your input. It isn’t “generative AI,” but it is a powerful tool for creating highly stylized animations that can fit into your creative toolbox.

Reality Capture

We discussed Reality Capture, a photogrammetry tool from Epic that works with your phone.

Light Fields

We discussed Google’s research on Light Fields, and referred to the paper/videos “A System for Acquiring, Processing, and Rendering Panoramic Light Fields Stills for Virtual Reality.”

Neural Radiance Fields

…and then we contrasted that with a discussion of Neural Radiance fields (NeRFs) which is an artificial intelligence technology for creating scenes from a sparse number of images. A few of the topics related to NeRFs included:

• Reference to the Matthew Tancik website on “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.”

• Luma, a tool for doing NeRFs on your phone.

Here’s a NeRF I made from a gargoyle in my backyard:

@metavertNeural radianxe fields (NeRFs). You take photos of the scene, and an AI model os then trained on the images to perform “reverse ray trackng” that can be used to generate geoemtry of a model (the gargoyle and the princess in this example). Then the 3D model and background images can be used to generate scenes from any angle. I used the Luma all for this demo and the siurce inages are from my backyard. #art #sculpture #3d #nerf #neuralradiancefield #ai #artificialintelligence #winter #photomagic

Tiktok failed to load.

Enable 3rd party cookies or use another browser

Performance Capture

We discussed innovations in avatar technology: capturing yourself and entering into virtual reality, as well as the ability to restyle the way you appear using augmented reality.

The Relightables paper from Google (Guo et al) discusses volumetric performance capture with realistic lighting.

More recently, the Codec Avatars from Meta shows how avatar systems could be linked to realtime performance capture:

A natural extension of performance capture is to bring it into platforms like TikTok. Bilawal has over 100,000,000 downloads for his filters for TikTok on Effect House. This is a creator platform that allows you to invent, distribute and monetize filters on TikTok.

Bilawal’s Journey as a Creatorpreneur

• Bilawal’s TikTok channel

• Bilawal’s YouTube channel

• Bilawal have a tip of the hat to Ali Abdaal, a YouTuber who talks about wellbeing—as his source for the term “creatorpreneur.”

Centralized vs. Decentralized AI

Bilawal and I discussed somewhat different perspectives on artificial intelligence—we agree on how exciting these technologies are for creators, but diverge on how rapidly we think things like multimodal AI and generative agents ought to be made available (and whether the weights ought to be open source). I fall more on the decentralized side, and Bilawal advises more restraint.

This will be one of the most important debates of our time, so it’s something I’d simply encourage you to learn about—and listen to contrasting views on. Here are a few perspectives that we brought up:

Bilawal mentioned Bilaji Srinivasan, an advocate of decentralization:

And we discussed some of the open-source models such as those from Stability. The CEO of Stability, Emad Mosque, advocates for democratized, open-source AI. He sees a need for governance and transparency by all the actors in the AI ecosystem. The following thread touches on this, as well as recent calls regarding the “6 month pause” which was brought up during our conversation:

We also discussed generative agents such as AutoGPT, and the potential impact (good and bad) they may have, as well as risks from Deepfakes.

Direct from Imagination

If you enjoyed the discussion around holodecks, reskinning reality and accelerating creativity—you may also enjoy my article: The Direct from Imagination Era Has Begun.

You may also enjoy the recent talk I gave at MIT: I touch on many topics Bilawal and I covered, such as avatar systems, reskinning reality and the future of generative agents: