[RELEASE] v2.0.0-beta7 is out!

[jarohen]

We're really proud to announce the release of XTDB 2.0.0-beta7 - this is a significant change to our underlying indices which expands the variety of temporal datasets that XT can performantly support.

Our primary focus in beta7 has been to improve XTDB's support for frequently changing entities, in order to better handle time-series shaped data, and there are also some removals to tell you about.

Frequently-changing entities (aka 'time series') support - migration required

From the beginning of the XT 2.x line, we have largely prioritised update patterns where each individual entity typically only undergoes a handful of updates through their lifetime - e.g. trades/orders progressing through a small state machine, or user profiles where the user may infrequently update their details - optimising for large numbers of those entities.

In this release, we've now significantly improved support for entities where this assumption doesn't hold.

Particularly:

• sensor readings - where we'd model each sensor with an ID; each reading as a version of that entity with a small validity window.
• trade pricing feeds - where we'd model each ticker with an ID; each individual update would be valid from now until corrected, but expecting the next correction to come very shortly afterwards.

To do this, we've made significant changes to our underlying indices.

XTDB nodes collaborate to maintain a log-structured merge tree (LSM tree) in the shared object storage, in order to balance write and read speed. Previously, we sharded deeper levels of this tree by the primary key (PK) of each entity - the deeper the level, the more shards. This doesn't work for these use cases, though: by only sharding by PK, we coalesce together all the versions of each entity - fine when each entity has (say) fewer than twenty versions; not so great when each entity has hundreds, thousands, millions of versions.

In beta7, at level 1 of the LSM tree, we've swapped the PK sharding for partitioning by the 'recency' of the entity versions. This takes inspiration from generational garbage collection, which posits that new data is likely to be quickly superseded; older data that's still current is likely to remain so into the future as well - newer data is collected quickly; older data doesn't have to be collected so frequently. (Level 2 and deeper continue to be sharded by PK, as before.)

The impact of this is that the LSM tree is effectively split into two halves:

• A historical half, predominantly sharded by the recency of the version. This is likely to be relatively shallow but wide, with lots of time buckets that are seldom written to after their window expires.
• A current half, sharded by PK. This is likely to grow deeper rather than wide, and is likely to be non-existent in the above use cases - no individual version ever remains current for long enough to make it into this half of the tree.

As a result, sensor reading and trade pricing feed data should now find that we can very quickly prune the majority of the historical versions, focusing on the specific window of interest to each query.

Migration

Given this is a change to XT's storage format, users upgrading to beta7 are required to migrate their existing clusters. First, ensure your nodes are upgraded to beta6.6 for forwards compatibility.

To migrate, there are two options:

1. If you have an infinite-retention log, you can simply stand beta7 nodes up in the usual green/blue manner. These will then re-index the log from scratch, which may take a while on large datasets.

   This is the easiest approach, assuming you have a relatively small dataset.

2. If you have a larger dataset, or a finite-retention log, you'll need to run our migration tool. This is executed as a one time task - you can leave your beta6 nodes running and run the migration tool in the background.

   To run the migration tool:

   • If you run XTDB through its CLI, run the beta7 artifact supplying your usual configuration arguments, but additionally add --migrate-from 5.
   • If you run XTDB through a Docker container orchestration platform (e.g. Kubernetes), run the beta7 version of your usual Docker image as a one-shot task, and override the Docker command to add the --migrate-from 5 flag.

   Once the migration tool has finished (it'll notify you, and then exit), you can deploy your beta7 nodes and decommission your beta6 nodes in the usual green/blue manner.

The rollback procedure is as follows:

1. Turn off any beta7 nodes if you've started them
2. Turn on beta6.6 nodes
3. Delete the v06 directory in your object store.

Arbitrary precision decimals

We've also added initial support for arbitrary-precision decimals (BigDecimals, in JVM parlance) - these can be inserted as BigDecimal/BigInteger values in your prepared statements.

This was prioritised thanks to requests from our Design Partners - if you have features/functionality you'd like to see, please do consider joining our Design Partner programme to help us shape the future of XT.

Removals

There are a couple of removals in beta7 - these were experimental features for which we haven't seen any take-up from our design partners or community, so we've elected to reduce our API surface area ahead of GA.

• The XTQL DML operators (:insert-into, :delete, :erase, :assert-exists, :assert-not-exists) have been removed due to lack of uptake - we've received barely any (if any) questions and issues related to these. :put-docs, :delete-docs and :erase-docs, as well as query-side XTQL will remain into GA.

  Longer term, we currently intend to inline XTQL within our SQL language (e.g. FROM XTQL (-> (unify ...) (return ...) (limit 10))) to allow it to be queried through the various Postgres drivers, rather than requiring a separate Clojure API.

• Transaction function support has also been removed - of the users who've gotten in touch with us, people who previously had no choice but to use transaction functions (in XTDB 1.x, because we didn't have anything else) now prefer to use standard SQL DML (e.g. UPDATE, ASSERT).

• We no longer support Java <21 in the in-process XTDB nodes nor our client libraries - if you use Java <21, you are still able to use our (Postgres-compatible) JDBC driver against an XTDB Docker image/dev-container.

We have very much been guided by our Design Partners and active community members in making these decisions - if you would like to get involved in shaping the future direction of XTDB, please do get in touch through the below channels!

Elsewhere

As always, for a complete list of the issues resolved in this release (all 84 of them!), see the associated milestone.

We'd love to hear what you think - please do get in touch:

• via email: hello@xtdb.com
• on the web: https://discuss.xtdb.com
• on our new Discord server

As always, a massive thanks for your support throughout this 2.x line - we're turning onto the home straight now!

James & the XT Team

---

This discussion was created from the release v2.0.0-beta7.