File copies using `dd` for cgofuse (memfs/ custom filesystem) are super slow on Windows 10 using PowerShell/MINGW64 terminal

[marius-enlock]

Hello,

I suspect that file writes are super slow on cgofuse windows.

I am running the following experiment on a Windows10 virtual machine with 16GB memory allocated.

I have run the memfs example and tried to issue a dd command for 1 GiB file from outside the filesystem inside the filesystem.
Note: bs=4096 has a similar speed, I just used 131072 as cp on linux deals in 128 KiB block size

dd if=~/Desktop/1GFile of=1GFile2 status=progress bs=131072                   119406592 bytes (119 MB, 114 MiB) copied, 13 s, 9.2 MB/s
926+0 records in
926+0 records out
121372672 bytes (121 MB, 116 MiB) copied, 13.4542 s, 9.0 MB/s

The dd command for files on the native filesystem has a copy speed of around 400MB/s.

A bit of context on how I stumbled on this issue:
I have a custom filesystem using the hanwen/go-fuse library, and on Ubuntu (fusermount3)/ MacOS(macFUSE) I achieve around 400-500MB/s speeds for copying a large file from outside of the filesystem to my custom filesystem.

However after I added windows support via winfsp (only using cgofuse when compiling on windows - the underlying logic is the same, it's just adapted to implement the cgofuse interfaces), I noticed that the speed for copying a large file from outside of the filesystem inside the filesystem is around 28-40 MB/s.

Edit:
I have tried the same experiment using memfs from WinFSP:

 net use Z: \\memfs64\test
 The command completed successfully.

ThrowAway@DESKTOP-L3GCCNG MINGW64 /z
$ dd if=~/Desktop/1GFile of=1GFile2 status=progress
31198720 bytes (31 MB, 30 MiB) copied, 5 s, 6.2 MB/s
68918+0 records in
68918+0 records out
35286016 bytes (35 MB, 34 MiB) copied, 5.92756 s, 6.0 MB/s

and the dd copy speed is very close to the one for cgofuse

Could it be because I am on a virtual machine? Could it be because of windows 10?

[marius-enlock]

Ah, the problem is only present in the MINGW64 terminal that came installed with git.
On cmd/ powershell/ visual studio code terminal the speed is high for memfs

Edit: I am wrong in this comment. The problem still persists on powershell/ visual studio code terminal with memfs and with my custom file system.

The problem seems to be with dd as cp on memfs is fast on visual studio code terminal/ powershell

[marius-enlock]

I can see that on windows the cp block size is 1 MiB, dd is faster using 1MiB block size with memfs, but still slower than cp

PS C:\Users\ThrowAway\go\bin\t1> dd if=C:\Users\ThrowAway\Desktop\1GFile of=1GFile status=progress bs=1048576
318767104 bytes (319 MB, 304 MiB) copied, 12 s, 26.5 MB/s
309+0 records in
309+0 records out
324009984 bytes (324 MB, 309 MiB) copied, 12.496 s, 25.9 MB/s

[billziss-gh]

My first recommendation in such cases is to ensure that you are using -o FileInfoTimeout=-1 which ensures that kernel caching is enabled. But since you are trying your tests against the WinFsp MEMFS (which enables file caching by default), the problem must be something else.

I am not certain why dd would be slower than cp. I note that both programs use the POSIX API presented by Cygwin rather than the native Windows file API. There may be some issue in Cygwin's interpretation of POSIX APIs as Windows file APIs that causes the discrepancy. Or it might simply be that dd does something that is cheap on POSIX but expensive on Windows.

One way to explore the differences is to use FileSpy. This can be used to track all file system operations as posted to the WinFsp FSD (File System Driver). So you could run your experiment with dd and FileSpy enabled, and you could also run your experiment with cp and FileSpy enabled. We might then be able to tell why dd is slower by seeing how it is accessing the file system.

[marius-enlock]

A bit unrelated with memfs, but optimizing my custom filesystem for a file block size of 16777216 bytes, yields a copy speed of 500MB/s on windows (compared to 40 MB/s for 128 KiB file block size optimization), which I am satisfied with for my use case.

Note that for linux (ubuntu) the optimium size from my experiments is 128 KiB, using higher sizes will downgrade the cp speed (the above one would yield 117 MB/s compared to 500 MB/s when I use 128 KiB), and on macOS it's the same as the above one from windows which yields 1400MB/s speeds.

Using: -o FileInfoTimeout=-1 just gave me permission denied at the start of the dd command - file got created but 0 bytes copied; but I wall clocked my Getattr and most of the time it takes 0 ns to return, and sometimes a few hundred nano seconds from time to time, totaling less than 0.2 seconds to copy the 1GFile.

For memfs:

PS C:\Users\ThrowAway\go\bin\t1>  Measure-Command { cp C:\Users\ThrowAway\Desktop\1GFile 1GFile}


Days              : 0
Hours             : 0
Seconds           : 0
Milliseconds      : 483

There is also an increase in speed for dd with this blocksize.

PS C:\Users\ThrowAway\go\bin\t1> dd if=C:\Users\ThrowAway\Desktop\1GFile of=1GFile status=progress bs=16777216
1056964608 bytes (1.1 GB, 1008 MiB) copied, 8 s, 132 MB/s
64+0 records in
64+0 records out

I find the discrepancy between dd and cp interesting, but I will take a break in investigating the issue, I didn't take a look at FileSpy yet.