Despite the amount of memory for each data type currently being 512 MB, ingesting a data set into modelardbd using Apache Parquet Loader can easily use tens of gigabytes of memory. For example, ingesting a 6.6 GB dataset stored in Apache Parquet uses approximately 38 GB of memory when using commit 8f0dc9b, however, after #420 has been merged, the memory usage seems to have increased. I can no longer ingest the dataset on a laptop with 40 GB of memory without the Linux kernel triggering an OOM kill due to high memory pressure.
Despite the amount of memory for each data type currently being
512 MB, ingesting a data set intomodelardbdusing Apache Parquet Loader can easily use tens of gigabytes of memory. For example, ingesting a 6.6 GB dataset stored in Apache Parquet uses approximately 38 GB of memory when using commit 8f0dc9b, however, after #420 has been merged, the memory usage seems to have increased. I can no longer ingest the dataset on a laptop with 40 GB of memory without the Linux kernel triggering an OOM kill due to high memory pressure.