LA4Falcon -m runs out of memory: Move to Nim

[pb-cdunn]

  82 FILE *Fopen(char *name, char *mode)
  83 { FILE *f;
  84   char newmode[3];
  85
  86   if (name == NULL || mode == NULL)
  87     return (NULL);
  88   if (strcmp(mode, "r") == 0) {
  89       strcpy(newmode,"rm");
  90   } else {
  91       strcpy(newmode,mode);
  92   }
  93   if ((f = fopen(name,newmode)) == NULL)
  94     EPRINTF(EPLACE,"%s: Cannot open %s for '%s'\n",Prog_Name,name,mode);
  95   return (f);
  96 }

For the DB, we use GNU fopen(filename, "rm"), which uses "mmap" for the file.

That avoids problems of filesystem-latency, but unfortunately I don't think GNU fopen() uses the MAP_SHARED flag, so separate procs will not share their read-only mmapped DB.

One solution is to use r instead of rm. However, then we are back to filesystem latency problems.

A better solution is to run LA4Falcon as a function, multi-threaded in a program with its consumer. A Nim version might help with that.

[pb-cdunn]

The problem could actually be in the consumer of LA4Falcon. But I combined program could help with both.

[pb-cdunn]

There was a big problem of running too many LA4Falcon jobs and too much memory consumed by the Python multiproc program. Solved in FALCON_unzip_private.

• https://jira.pacificbiosciences.com/browse/SE-697
• https://jira.pacificbiosciences.com/browse/TAG-1448

Still, moving LA4Falcon to Nim will have benefits some day.