Cannot mount large files

[lczech]

Hello,

when trying to mount a large FASTA file (~30GB) with mmap, Rust is giving me an OOM panic:

fusta --no-daemon --max-cache 10000 SILVA_123.1_SSURef_Nr99_tax_silva_full_align_trunc.fasta 
 [INFO] Caching method:  Mmap
 [INFO] Reading SILVA_123.1_SSURef_Nr99_tax_silva_full_align_trunc.fasta...
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }', src/fs.rs:601:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'm not sure if that can be solved, or is a general limitation of mmap for such large files. Maybe at least the exception could be caught and a suggestion could be printed to use --cache file instead? It works fine with that option, and is surprisingly fast!

Cheers
Lucas

[delehef]

I'm not sure if that can be solved

It should be if you are admin on the machine and are OK to allow memory overcommitment. You can try the workaround mentioned here, please tell me if it works for you.

I was personally able to mount ~80GB files on a 16GB machine with vm.overcommit_memory enabled, as the memmap in the kernel will consume the memory address space, but not actually use all the underlying memory at once.