The problem for running modflow6-parallel

[verkaik]

Li Jian wrote:

Sorry for asking the parallelized modflow6 problem in this repo, but I cannot find the issue submitting in Verkaik's forked repo. And thank you Verkaik for you suggestion. I have two questions as following:
(1) I have compiled modflow6-parallel-6.2.1 with the option MPI_PAEALLEL, and modify the mfsim.nam as you suggested. But when I run mf6 using mpirun -np 2 mf6. The simulation time is longer than the serial mode. Could you please check my .nam file? It's better to send me an example can be run in parallel mode.
(2) Should I conduct the domain decomposition manually? I cannot find any tutorial for running parallelized modflow6 simulation.
Thanks !
I use the modflow6-largetestmodels/test1000_751x751

BEGIN OPTIONS
MEMORY_PRINT_OPTION SUMMARY
DOMAIN_DECOMPOSITION 2
END OPTIONS

BEGIN TIMING
TDIS6 mfusg_C.tdis
END TIMING

BEGIN MODELS
#modeltype namefile modelname
#GWF6 mfusg_C.nam GWF_1

GWF6 mfusg_C.nam GWF_1 1
GWF6 mfusg_C.nam GWF_1 2
END MODELS

BEGIN EXCHANGES
END EXCHANGES

BEGIN SOLUTIONGROUP 1
MXITER 1
IMS6 mfusg_C.ims GWF_1
END SOLUTIONGROUP

[verkaik]

Hi,
To answer your questions:
(1) Your mfsim.nam looks good. You should be aware that performance on non-cluster machines can be limited and strongly depends on your processors (#cores, and more important memory bandwidth). The MPI parallelization is coarse-grained, meaning that the submodels shoulds large (say >1 million cells). This test should run when using 9 MPI processes:
https://github.com/verkaik/modflow6-examples/tree/master/mf6/test031_many_gwf
(2) Yes, you should do that manually! Basically, running in parallel is straighforward like I explained and there is no manual. There are some additional options, see the included powerpoint.

Best, Jarno

Using_parallel_MODFLOW_6.pptx

[lj-cug]

Thanks for your timly and patient reply. The test model you supplied is very helpful to run the parallelized modflow6. At present I just run the parallelized modflow6 on my personal computer, the real case study will be conducted on HPC cluster after I get familiar with the whole workflow to configure modflow6-parallel model. But I still have some questions about the model setup workflow right now.
(1) The already setup test model you supplied has been domain decompositioned into 9 sub-domains and can be run in parallel mode with coasrse grained parallelization. If many processes should be used, for example 1000 subdomains, how can I implement the domaim decomposition？ Still do it manually? It's unpractical. How I can implement the domain decomposition for the modflow6 input files obtained from Flopy tool? If there's any Python scripts, the workflow will be smooth.
(2) I found the iMOD and iMOD-Python tools were supplied in PCR-GLOBWB 2 project, which is a hydrology-groundwater coupled system. The Python script can implement the domain decomposition pre-processing using METIS. But the input data file is huge and the computational scale is big, I have no computational resource and storage to conduct the global scaled simulation. I just want to run the stand-alone parallelized modflow6. If there's Python scripts to pre-process the modflow6-parallel, it's better. But the Python script was for the coupled system modeling, I only want to conduct the regional grounwater simulation.
(3) It's noticed that the input and output files are read and writen in parallel using iMOD or iMOD-python tools in your paper EMS(J. Verkaik, et al. 2021), different format files can be processed including raster and IDF files. The output files from parallel running should be merged after completion for the visualization post-processing. I don't know the workflow for the pre- and post-processing right now, could you please give some hints?
At last, the parallelized modflow model is a powerful tool! Thank you for your great contribution.
Thanks!
Best Regards,
Li Jian
China University of Geosciences

[verkaik]

Dear Li Jian,

Sorry for the late response. Good to hear that the code now runs on your machine! :) To comment on your questions:

(1) Yes, you still need to do everything manually similar to standard MODFLOW 6 using multi-models. Unfortunately. You can use a graph partitioner, like METIS to balance the grid cells. I don't think FloPy already supports settting up coupled multi-models yet, or automatic partitioning using METIS.

(2) I don't understand: do you also want tu use PCR-GLOBWB model data for your regional model?

(3) If you are planning to use parallel MODFLOW 6, there is no standard workflow you can use at the moment. At some point effort will be made to support this with for FloPy and iMOD-Python, but at the time being there is nothing for you can use out of the box. Can you perhaps tell me a little more about the application you have in mind? Resolution and number of cells, number of cores, etc.?

Best, Jarno

[lj-cug]

Dear Jarno:
Thank you very much for your reply.
As for the three questions, I think about them these days and get some hints from Langevin-usgs.
(1) The splitter model is developing in Flopy told by Langevin-usgs, which will be release in a couple of monthes. The splitter model can split the whole domain into several smaller domains used in modflow6-parallel. Before this, I have to create GWF-GWF exchange models manually, which is in flopy-3.3.6/examples/Tutorials/modflow6data/tutorial09_data.py, this tutorial creates two GWF models can be exchanged with each other. But I don't know how to create more than 2 models, such as 3x3 models domain.
Meanwhile, METIS partitioning library was used in PCR-GLOBWB, this confused me, because METIS is for unstructured mesh model and the GWF-GWF exchanged coarse-grained parallelizetion was used in modflow6-parallel as I know.
(2) Actually I don't run PCR-GLOBWB model (it's huge!), I just want to know the running mechanism of parallel-modflow6 in the coupled system. I want to use parallel modflow6 to simulate my regional model - Tarim Basin in China.
(3) The Tarim Basin has great groundwater resource in China, which is the region (0.4 million squared meters) I want to study. The cell size is about 500 meters. Firstly, I want to run modflow6-parallel in my workstation (CPU has 16 cores).