Right now we don't prune the sqlite database very agressively. I want the user to be able to search historically (i think?) so we want some historical data but we dont want gigabytes and gigabytes of embedding data. The question then is this: How do we keep historical data without compromising quality or bloating the sqlite db?
- Compression maybe? : compress old data and only read it when it's needed?
I'm really unsure here.
Right now we don't prune the sqlite database very agressively. I want the user to be able to search historically (i think?) so we want some historical data but we dont want gigabytes and gigabytes of embedding data. The question then is this: How do we keep historical data without compromising quality or bloating the sqlite db?
I'm really unsure here.