Introducing Omnimodal Intelligence: The Future of Data Management

Organizations have never generated more valuable data than they do today. From genomic sequences to medical imaging, from sensor data to complex simulations, modern enterprises produce massive volumes of information that hold the key to competitive advantage, scientific breakthroughs, and operational excellence.

Yet despite this wealth of data, organizations struggle to extract its full value. The problem isn't a lack of data, it's that these data are heavily siloed across countless systems, and the vast majority exist in non-tabular formats that current data infrastructures simply cannot handle effectively.

The Trap of Multimodal

A new term has emerged in the market: multimodal databases. But there's a problem. The industry is using multimodal to describe systems that handle only a tiny slice of organizational data, typically just text, images, audio, and video. These are often vector databases optimized for retrieval augmented generation (RAG) with large language models.

While RAG and vector databases serve an important purpose, they address only a fraction of enterprise data needs. What about the genomic variant files (VCF) that pharmaceutical companies rely on? The single-cell experiments (h5ad) driving personalized medicine? The point cloud data (LAS/LAZ) essential for autonomous vehicles and defense applications? The time series data powering financial models? The geospatial datasets informing climate research?

These critical data types and countless others are conspicuously absent from multimodal solutions. They're dismissed as unstructured and relegated to object storage as mere blobs, stripped of context and metadata. When you call data unstructured, you essentially give up from the start and declare you can't efficiently extract meaningful information from it.

This is why we need a more ambitious vision: omnimodal intelligence.

What Is Omnimodal Intelligence?

An omnimodal data platform treats all data — every format, every type, every modality — as a first-class citizen with semantic meaning. It makes all organizational data discoverable, securely accessible, and analyzable in a performant way.

The omnimodal approach rejects the concept of unstructured data entirely. Instead, it introduces the modality concept: a data specification that includes metadata, description, schema, visualization capabilities, and APIs for compliance and sophisticated access and analysis.

In an omnimodal system:

• An image modality exposes width, height, resolution levels, and compression information

• A genomic variant modality reveals sample populations, variant types, and genomic coordinates

• A Postgres table is a modality with its own schema and access methods

• A Snowflake table is a modality with analytical query capabilities

• Even code (UDFs, task graphs, notebooks) and apps (custom dashboards) are modalities

Everything has context. Everything has meaning. Everything can be governed, cataloged, and analyzed.

Why Now? The Agentic AI Revolution

The urgency for omnimodal intelligence has never been greater, and the catalyst is agentic AI.

AI agents are transforming how organizations interact with data. Unlike large language models (LLMs) that can only answer questions about their training data, agents can execute tasks on private organizational data to query databases, search filesystems, run analyses, and more.

But here's the challenge: without an omnimodal platform, building AI agents and making them discoverable and secure is a formidable challenge.

To build an effective agent, you need:

• The underlying data infrastructure to be deployed (Postgres for transactional data, Snowflake for analytics, specialized systems for genomics, imaging, etc.)

• Proper metadata and indexing so agents can find relevant data

• Authentication and access control for each system

• Performance optimization for large, complex datasets

For unstructured data stored as blobs in object stores, building useful agents is nearly impossible. Files appear as blobs without context. Without proper metadata and indexing, a "filesystem agent" would be futile. And certain data types, such as genomics, transcriptomics, and imaging, are so large and complex that leaving them in original formats creates dramatic scaling issues that no agent can resolve.

Organizations can't fully leverage AI's capabilities, not because of the quality of LLMs and agents, but because their fragmented data infrastructure prevents them from building agents that efficiently and economically access all organizational data.

Organizations don't have an AI strategy problem. They have a serious data management problem.

The Challenges of Building an Omnimodal Platform

Creating a true omnimodal platform requires solving several fundamental challenges:

Expressing everything as a modality: Every data type must be elevated from blob to modality with semantic context, metadata, and APIs. This gives meaning to data and enables sophisticated processing.

A unified, all-encompassing catalog: With data treated as modalities, cataloging becomes possible through keyword search, domain-specific filters on metadata, and searches of actual asset contents. The catalog must span all modalities across the organization.

A unified governance layer: A single authentication and access control plane must work across all data types and sources. All activity must be logged for auditing. When connecting to external data sources, the platform must ensure security compliance of the entire offering.

Special care for non-tabular data: This is where omnimodal platforms diverge most dramatically from traditional approaches:

• "OK" formats: Some formats, like certain genomic or imaging standards, are acceptable but require their specialized tools to be used securely within a trusted computational environment

• "Not OK" formats: Other formats have severe performance and scale limitations when you accumulate millions of files. These require conversion to more efficient representations that maintain semantic meaning while enabling performance at scale

Enter TileDB Carrara: The Omnimodal Intelligence Platform

We started TileDB as a research project at MIT and Intel Labs from 2014 to 2017, and spun it out as a company in 2017. Since its inception, our vision has been holistic data management, or what we now call omnimodal intelligence. It took several years for the market to come around, which happened only after the recent AI explosion.

When we started TileDB, the market leaned toward vertical databases; for every new exciting data type, someone would start a GitHub project, get funded by VCs, and create yet another database flavor. This led to trends like the "modern data stack" and "data fabric," which attempt to reconcile the incredible fragmentation resulting from the insatiable appetite for incremental data solutions tailored to bespoke use cases.

Our initial approach consolidated diverse data types under a unified data model more general and powerful than tables: the multi-dimensional array. This worked exceptionally well for complex, large-scale data, like genomics, single-cell, point clouds, imaging, and time series. We worked with amazing customers like Amgen, Johnson & Johnson, Quest Diagnostics, Boehringer Ingelheim, Takeda, NTT Docomo, and Lockheed Martin.

To fully realize our omnimodal vision, this year we introduced Carrara, our new product version that implements omnimodal intelligence:

Everything centered around modalities: In Carrara, all data types, including tabular, non-tabular, code, apps, and even AI agents, are modalities carrying semantic context through metadata and easily searchable in the system.

Unified catalog and governance: Carrara provides workspaces and teamspaces with a single authentication system, comprehensive access control, and complete activity logging across all modalities.

Tabular data via partner connections: Carrara integrates with Postgres, Snowflake, Databricks, and other systems, treating external tables as modalities within the unified platform.

Non-tabular data handling:

• "OK" formats: Carrara's virtual filesystem welcomes all formats, allowing users to continue using their favorite domain-specific tools while enjoying advanced governance, catalog, and computational framework capabilities.

• "Not OK" formats: For data with severe performance and scale limitations, Carrara converts data to TileDB arrays, which offer efficient representations optimized for performance while preserving full semantic meaning.

Extensible platform: Features like a "modality builder" and "AI agent builder" are coming soon, providing extreme platform extensibility for any industry while leveraging AI's full power.

The Future Is Omnimodal

The unified architecture and extreme extensibility of TileDB's omnimodal platform make it ideal for the new AI era. We've recently partnered with leading hyperscalers (AWS, Microsoft), technology providers (Snowflake, Databricks), and global systems integrators (EPAM, Cognizant, Accenture) to showcase and demonstrate this value.

But the future of omnimodal intelligence isn't just about TileDB. It's about a new paradigm where customers and partners play crucial roles in this digital transformation. With modality builders, organizations can create custom modalities for their unique data types. Partners can extend the platform with specialized capabilities. Together, we're building an ecosystem that truly encompasses all data.

The future of AI isn't just about smarter models—it's about building the data infrastructure that can truly support them. Organizations that recognize this and invest in omnimodal architecture today will be the ones leading tomorrow's AI revolution.

The choice is clear: continue struggling with fragmented data infrastructure and limited multimodal solutions, or embrace the omnimodal future and unlock the full potential of all your data.

Ready to prepare your organization for the omnimodal future? Contact us to discover how TileDB's unified data platform can revolutionize your data architecture and allow your organization to access the full potential of data.