Our current LSM tree db (badger) does not support enough metadata for CDC/snapshots and its tx handling is a bit incompatible with how knox tx work (badgers keeps all writes in memory, whereas knox uses journals which are similar to memtables, but may contain uncommitted and aborted data)
implement
- an own LSM tree engine using badger skiplist (keep values in-place with keys)
- enhance values with xmin/xmax/rid metadata
- keep per field metadata for memtables and sstables
- enhance memtables with tx status handling (can write to disk once memtable is full), but merge only when all tx are complete (skip data from aborted txn)
- use central knox wal for durability
Our current LSM tree db (badger) does not support enough metadata for CDC/snapshots and its tx handling is a bit incompatible with how knox tx work (badgers keeps all writes in memory, whereas knox uses journals which are similar to memtables, but may contain uncommitted and aborted data)
implement