Data Fabric for the Metaverse
Interoperability starts at home
Interoperability has the potential unlock massive value within games, online simulations, immersive media—anything we might call “the metaverse.”
Interoperability is a tough problem to solve because it involves a diverse set of intentions behind the protocols and data objects within a system. Here is a diagram that describes the domains that are possible targets of interoperability:
However, it is worth noting that before one can aspire to interoperability between a set of disconnected organizations, it is necessary to figure out how to achieve interoperability simply within parts an organization.
Data Fabric
I want to focus this article on the concept of a data fabric, which is primarily intended to address the “persistence” layer of interoperability within the above chart: basic properties like accounts, ownership of items, the representation of items in an economy, etc.
“Data Fabric” is a newfangled term, albeit one that isn’t completely novel to those who have spent their careers on building large-scale, cloud-native infrastructure for an organization. If you want an extremely detailed exploration of data fabrics, Hitachi has an excellent e-book on the topic, but in summary:
A data fabric is a unified means of interacting with structured and unstructured data across the different applications that needs to utilize data
Policies and access controls that govern who is allowed to do what with the data
Extensible and modular, so you can add to it easily
In the field of game development, few companies have adopted modernized systems for managing data (there are some notable exceptions—e.g., Zynga is well known for having a very comprehensive and robust approach to data science). And although this article is about “the metaverse,” this all begins with game development because it is the game industry that is creating the technology stack the metaverse is being constructed in.
The main purpose of this article is to familiarize people who are building online games and other metaverse applications with the concept of a data fabric, because it will be important to think at this level to achieve any amount of far-flung interoperability.
Benefits of a Data Fabric
In Cloud Native Worlds, I brought up data fabrics in the context of game development. The immediate benefits of a data fabric is realized internally.
Imagine the item database that is the heart of your online game. By the time your game enters live operations, you’ll gain a lot of benefits if you’re able to share item-data—down to the player-account level—across a range of applications.
Examples include:
Customer service (see history of items received, add/remove as needed, investigate cheating)
Analytics and segmentation (find out who owns own, separate according to player populations, see what you can learn about player behaviors from the items they’ve acquired)
Messaging campaigns (send items in messages, target players with messages according to items they own, send notifications to people based on ownership status)
Quality assurance (investigate bugs, set up test scripts)
You could even think of many bleeding-edge applications, such as links into generative AI systems that make new item art for each new item drop, as it is added to the economy.
And naturally, you’ll need to make sure you have policies in place that determine who is able to interact with the different types of data in a secure and reliable manner.
Unfortunately, I’ve seen too many cases where internalized interoperability is an afterthought, resulting in massive technical debt as each add-on system becomes a new ETL process, a new data warehouse, a new offline processing system.
While prototyping a game is too early to think about these things, as soon as a game system and its economy are reasonably settled—it will save a lot of time and energy—and add powerful extensibility—to consider how you structure these objects from the beginning. In a follow-up post, I’ll share some thoughts on how we’ve thought about these things at Beamable.
What about Blockchain?
Web3 advocates often point to blockchain as one of the solutions to interoperability. And indeed, it may be. Blockchain is an effective way to achieve data consensus between multiple organizations who wouldn’t ordinarily trust each other. But it isn’t a panacea either.
The reality is that even the fastest blockchains are not performant enough to handle all of the data flow within a single game, let alone all the games across the internet that choose to cooperate with each other (for one interesting comparison, consider that LinkedIn was doing 2 million data writes/second in their Kafka implementation—back in 2014!—and that big games may need to do a lot more than that).
Game Servers
Game servers are important. Things like player inventories, movement vectors, social structures, tournaments, etc. are better implemented using high-performance databases within the game environment. And metaverse will be built on game servers.
This is where a data fabric strategy comes in again: with the right approach to managing your data, you can link blockchain to publish/subscribe mechanisms with a blockchain that enables you to treat the blockchain as the source of truth for ownership information—yet serve moment-to-moment data in the game in a highly performant manner. Likewise, all of the other subsystems described above (customer service, QA, GenAI, etc.) are easier to connect to a data fabric than direct-to-chain interactions. The chain should be reserved for recording transactions between wallets (and potentially other interesting non-financial use-cases where public provenance and authenticity are essential, like tournament match records or digital identity—which I hope to write about in the future).
And even if you don’t really care about linkage to blockchain, the same principle remains true: many secondary systems will benefit from event-based or publish/subscribe mechanisms that are triggered upon important changes in your data fabric.
Further Reading
I previously wrote Cloud Native Worlds to outline some of the technologies to consider (containers, data fabric, etc.) when building a modern, cloud-native game.
I wrote about interoperability back in Metaverse Interoperability, Part 1: Challenges.