Repository navigation
[R] Substantial RAM use increase in 9.0.0 release on write_dataset() #32795
Description
Consider the following example of opening a remote dataset (a single 4 GB parquet file) and streaming it to disk. Consider this reprex:
s3 <- arrow::s3_bucket("data", endpoint_override = "minio3.ecoforecast.org", anonymous=TRUE)
df <- arrow::open_dataset(s3$path("waq_test"))
arrow::write_dataset(df, tempfile())
In 8.0.0, this operation peaks at about ~10 GB RAM use, which is already surprisingly high (when the whole file is 4 GB when on disk), but on arrow 9.0.0 RAM use for the same operation approximately doubles, which is large enough to trigger the OOM killer on the task in several of our active production workflows.
Can this large RAM use increase introduced in 9.0 be avoided? Is it possible for this operation to use even less RAM than it does in 8.0 release? Is there something about this particular parquet file that should be responsible for the large RAM use?
Arrow's impressively fast performance on large data on remote hosts is really game-changing for us. Still, the OOM errors are a bit unexpected at this scale (i.e. single 4GB parquet file), as R users we really depend on arrow's out-of-band operations to work with larger-than-RAM data.
Reporter: Carl Boettiger / @cboettig
Related issues:
Original Issue Attachments:
Note: This issue was originally created as ARROW-17541. Please see the migration documentation for further details.
Activity
asfimport commentedon Aug 26, 2022
Weston Pace / @westonpace:
Are you writing the file locally? How are you measuring RAM usage?
Is that 8-10GB in the process' RSS space? If so, we probably need to expose controls in R to lower the readahead amount (there are some other long-term options but this would be a short-term fix that would let you tradeoff RAM vs. I/O throughput)
On the other hand, if the process' RSS space is lower, but the system available memory is still low (e.g. output from the
freecommand) then it's probably because your input throughput (e.g. downloading from S3) is faster than your disk's write speed (not too surprising with an HDD or even some SSDs). What happens in this case is the writes are simply memcpy'ing data from RSS into the kernel page cache and marking the pages dirty. The write doesn't actually persist to the disk until sometime later (even possibly after the process has ended). If your write is slower than your read then the kernel's page cache will fill up and clobber all other memory, pushing it to swap or even potentially invoking the oom killer. On the bright side, we do have a PR in progress which should alleviate this problem, at least on Linux.Can you do some investigation to try and figure out which of these possibilities we are encountering?
asfimport commentedon Aug 26, 2022
Weston Pace / @westonpace:
{blockquote}
Can this large RAM use increase introduced in 9.0 be avoided? Is it possible for this operation to use even less RAM than it does in 8.0 release? Is there something about this particular parquet file that should be responsible for the large RAM use?
{blockquote}
Hmm...I'm not sure why this would increase from 8.0 to 9.0 either. I'll try your reprex and see if I can reproduce this on Monday.
asfimport commentedon Aug 26, 2022
Carl Boettiger / @cboettig:
Thanks Weston for the pointers here!
Yes, tempfile() in the example above returns a local path, though I see the unexpectedly high RAM use in a variety of settings:
In the example described above, tempfile() for me is a local NVMe disk, while the reading is from a remote server, so I would not expect the read-in to be faster than the write-out in this case.
in our original case, we are streaming out to another S3 bucket where we see the crash, rather than writing to a local tempfile()
In both settings I'm testing on a Linux platform. On Arrow 9.0 I see peak RAM use at nearly 30 GB in "RES" category (e.g. in glances/top) or as "used" if I run
free -hfrom bash. The RStudio interface shows this same number (though benchmarking software like bench::bench_memory() does not see this RAM use, reporting it as only a few hundred KB).Sorry I don't know my way around the situations better. Can you successfully run the code above? (It doesn't require any authentication, should be a reprex, I've run it in a lot of places). In every platform in which I run those three lines, I see RAM use 2-3x higher in arrow 9.0 and always way higher than the 4GB file size. Is that expected?
asfimport commentedon Aug 26, 2022
Weston Pace / @westonpace:
Thanks, this should be enough for me to investigate further.
asfimport commentedon Aug 30, 2022
Weston Pace / @westonpace:
I looked at this a bit today but RAM usage didn't get past 6GB and I saw similar behavior between 8 and 9. Even that seems high and I still have a few theories (e.g. I was only using debug builds) so I plan to keep exploring this but wanted to give an update. If it's not too much trouble can you explain how you are obtaining or building the arrow package?
asfimport commentedon Aug 30, 2022
Carl Boettiger / @cboettig:
Weird. I'm running on ubuntu linux (rocker/geospatial container), installing arrow as a prebuilt binary from RStudio package manager
asfimport commentedon Aug 30, 2022
Weston Pace / @westonpace:
Ok. There is definitely something strange going on here. I plotted memory usage today during a run via R and then made the same run via python. Notable observations:
R does not appear to be releasing batches that have been written but the python version is doing so :(
S3 download speeds, when run through python, are half as fast as the R version.
Naively I want to say backpressure is the culprit in both cases. A lack of backpressure leading to excess memory in R and an excess of backpressure throttling downloads in python. However, the R behavior is not what I would expect from a lack of backpressure. Even without backpressure we should be freeing data as it is written.
asfimport commentedon Aug 31, 2022
Weston Pace / @westonpace:
So I'm pretty sure that the real problem is that R's garbage collector is somehow holding onto pool memory. This wasn't new in 9.0.0. As you said yourself, it was already using a pretty excessive amount of RAM in 8.0.0. Notice that, in python, downloading this file uses less than 1GB of RAM. I'm not going to worry too much about the fact that RAM rate doubled between 8.0.0 and 9.0.0. I suspect that may just be that we are more aggressive with readahead on these sorts of files in 9.0.0 (in 8.0.0 we always read ahead 8 batches, in 9.0.0 the readahead is based on # of rows which leads to 20 batches on this file).
R's garbage collector is not running for two reasons:
R is not aware there is any memory pressure, because it doesn't see the RAM used by Arrow pool memory. As far as it is concerned it is only holding onto 80MB and in reality it is holding onto multiple GB of RAM.
We are executing a single (admittedly long running) C statement with
write_dataset. R's garbage collector will not (I think) run mid-execution.We could investigate the above two issues but there is a third, more concerning problem:
The allocations we are making in Arrow come from the memory pool, they are owned by record batch objects, and those record batch objects are never (as far as I know) converted to R. Perhaps they are being converted to R somewhere (we are scanning and then writing, do we scan into R before we send to the write node? I wouldn't think so but I could be wrong). Or perhaps R's memory allocator works in some strange way I'm not aware of.
I'm going to have to step back from this investigation as I've hit my limit for the week and there is other work I am on the hook for. So if any R aficionados want to investigate I'd be grateful.
asfimport commentedon Aug 31, 2022
Dewey Dunnington / @paleolimbot:
I'm working on two PRs that touch some of those parts of the code and will investigate at some point in the next two weeks. A few thoughts:
I believe that's true, although in theory R is also not allocating any memory either. I don't know of any way to allocate R memory without (potentially) running the garbage collector.
Would a more precise way to say this be that there is some shared pointer (potentially held by an R6 object that is still in scope and not being destroyed) that is keeping the record batches from being freed? We do have an R reference to the exec plan and the final node of the exec plan (which would be the penultimate node in the dataset write, which is probably the scan node). (It still makes no sense to me why the batches aren't getting released).
asfimport commentedon Aug 31, 2022
Weston Pace / @westonpace:
Yes, I think that is a more precise way. Holding onto the ExecPlan (which owns the nodes too) should be ok (indeed, desirable).
asfimport commentedon Oct 27, 2022
Dewey Dunnington / @paleolimbot:
This may or may not be related, but we have a report of "leaked memory" from a dataset collect here:
https://stackoverflow.com/questions/74221492/r-arrow-open-dataset-selectmyvars-collect-causing-memory-leak
asfimport commentedon Oct 28, 2022
Dewey Dunnington / @paleolimbot:
Also cataloging @assignUser's observation that this only seems to happen with datasets with a particular schema!
thisisnic commentedon Sep 24, 2026
This comment was written by Claude Code, an AI assistant, which ran the investigation below under my direction. I've reviewed the results before posting.
I revisited this with Arrow 25.0.0. The original MinIO endpoint is no longer reachable, so I reproduced locally with a single large parquet file (40M and 80M rows, 8 columns, 100k-row row groups), running open_dataset() into write_dataset() and sampling process RSS every 0.5s.
What I found:
Peak memory grows with file size, and grows steadily through the run rather than plateauing, which matches the original report.
R and pyarrow now have identical memory pool peaks, so the R-specific retention seen in 2022 (R holding batches Python released) no longer happens. Allocator choice (mimalloc, jemalloc, system) makes no difference, and nothing is retained after write_dataset() returns.
The remaining growth is parquet pre-buffering. It is on by default and, for a single large file, caches the raw bytes of every row group without evicting the ones already consumed. Disabling it flattens the curve with no slowdown on local disk:
open_dataset(path, format = FileFormat$create("parquet", pre_buffer = FALSE))
This is the same mechanism as [Python] Dataset.to_batches accumulates memory usage and leaks #39808, and GH-39808: [C++][Parquet] Evict pre-buffered row-group bytes after decode #49855 is the C++ fix (evict pre-buffered row-group bytes after decode).
The 9.0.0 doubling was most likely the readahead change Weston mentioned, amplified by this retention. The dataset writer backpressure fix in 16.0.0 (#40224) addressed the writer side separately.
Proposal: close this as a duplicate of #39808 once #49855 lands, and link #33366 which looks like the same cause.