Keep going!

[theronic]

Not an issue - just wanted to say this is great and please keep going!

The hot ticket will be to use S3 as a store for batches of 10k+ datoms with some % overheads to slice batches in various ways that are cache-optimized ways, kind of fractal trees. E.g. a client who just joins cares more about the newest 500 datoms and 1000 initial datoms than the newest 50,000 datoms.

If Crux has any plans to build a differential dataflow DB as a service backed by S3 + Firebase (or something like it for hot datoms) & Cloud functions, please get in touch.

[csm]

I did some work simplifying my ideas, and this is now is master.

I adapted hitchhiker-tree for use as the TxLog, and this means any konserve implementation can be used as the underlying data store. konserve-ddb-s3 stores trees across DynamoDB (the tree roots) and S3 (everything else).

I also wrote a kv implementation on hitchhiker-tree, so that can even be used as the kv-store, but I'm having problems getting that to work reliably. The hh-tree TxLog works fairly well, though.

A big missing piece from all of this is some kind of segment GC -- hh-tree will currently leave orphaned nodes in storage, since when nodes are modified they're stored under a different name.