Hi there,

We wrapped up 2022 and have a range of TileDB Cloud features and open-source updates to share. Let's get straight to the good stuff!

UX enhancements, including storage options and in-app onboarding.

Storage paths per asset type

You can now point each TileDB Cloud asset type to a specific object storage path. Check your TileDB Cloud profile page to toggle the "Show granular storage paths" option. For existing users, the previously set storage path remains unchanged, but we also provide a quick migration option to update the locations of your pre-existing assets.

Namespaces UI

To make it easier to switch between multiple TileDB Cloud organizations, we introduced a new UI for namespaces. You now have a clear snapshot of your currently active org, along with the respective permissions for each organization.

Custom AWS endpoints

For enterprises with special object storage requirements, TileDB Cloud now saves an optional endpoint field to make it easier to connect to non-AWS storage like Wasabi, Cloudflare R2, and others.

In-app onboarding

For individual accounts, TileDB Cloud features a new onboarding experience. You can view your status and return to pending quickstart guides through the progress bar at the bottom of the left navigation menu.

Highlights from versions 2.10, 2.11, 2.12, 2.13, 2.14.

OR clause in query filters

We introduced support for OR clauses in TileDB query conditions in the 2.10 release. Previously, TileDB only supported AND conditional logic when filtering attributes via query conditions.

Object storage performance

The 2.11 release includes performance improvements for TileDB arrays on AWS and Google Cloud object storage, as well as query conditions on dimensions in sparse arrays. Please note that slicing array dimensions will always be more performant, but the option to include dimensions in query conditions alongside attributes is useful for certain complex queries with OR clauses.

DELETEs

We introduced DELETE query support in the 2.12 release. DELETE queries provide the capability to non-destructively remove data from query timestamp-forward, preserving time traveling capabilities until running array consolidation using the "sm.consolidation.purgedeletedcells" parameter for irrevocable deletion.

WebP support

In the 2.13 release we added support for lossless and lossy RGB(A) and BGR(A) image compression using WebP. Popular in geospatial applications for efficient map tiling, WebP compression plays an important role in upcoming TileDB capabilities for biomedical imaging. (More below!) We also added dictionary and RLE compression filtering for strings, useful in life sciences analyses that often use string dimensions like sample ID, chromosome, cell type, etc.

Query conditions on UTF-8 attributes

In addition to fixes and performance improvements, the 2.14 release contains new features related to query conditions: QCs on UTF-8 attributes are now supported with binary comparisons, as well as QCs against boolean attributes in sparse arrays.

News from the TileDB open-source ecosystem.

Allele frequency on data import

No more retraversal of large genomics datasets to compute common statistics. The TileDB-VCF library recently added optional ingestion tasks to efficiently compute allele count and allele frequency upon data import.

Introducing TileDB-BioImaging

TileDB-BioImaging is a Python library for converting images stored in popular biomedical imaging formats to groups of TileDB arrays (and for exporting them back out). It also exposes an expressive API for efficiently querying this data. This library is in pre-release and is currently under heavy development. Early adopters should expect breaking changes. Be on the lookout for more updates soon, including a napari plugin!

Our customer Geoscience Australia recently presented about how they are managing customized datasets and enabling collaborative analysis of global seabed maps. Learn how TileDB Cloud handles the complexities of storage and compute, freeing up geospatial scientists to address core data interoperability problems and expand research efforts.

Check out this refresher on TileDB. In it Stavros Papadopoulos, the creator of TileDB Embedded and CEO of TileDB, Inc., reflects on the technology's history and how far it has come since 2017.

What if you were analyzing tons of data, without using a database? This is happening today in demanding industries like genomics and earth sciences thanks to complicated file formats. Join us on The Data Stack Show to learn about TileDB's approach as a universal database for complex datasets.

We have a lot in store for 2023, so watch your inbox for more frequent updates and news from TileDB, including an overhaul of our documentation and technical examples.

If you'd like to share product feedback, simply reply to this email, join our Slack community, or follow us on Twitter and LinkedIn. We'd love to hear about your TileDB experience and future requirements.

Thank you,

— The TileDB Team