Skip to content

Remove multiprocessing code #63

@gcapes

Description

@gcapes

Copied from https://github.com/UoMResearchIT/hydra-mpi-issues/issues/28

Now that we're using MPI and setting the multiprocessing number of processes to 1, should we just simplify the code and remove the multiprocessing map?

That issue "Why is random seed set on each call of gcr_fgmodes_1d" (if you don't have access to it) says:

I think this is probably a historical/platform specific/package version thing.

The current behaviour creates different random numbers each time gcr_fgmodes_1d is called, because this function sets the seed using a combination of the idx (baseline slice index) and also the pid (but doesn't actually use the value from the config file). If we just use a fixed seed value, every slice gets the same random numbers, and the function sample_S hangs in the while loop. So that explains why it hangs when I use the same fixed seed for every call.

But, why do we need to set the random seed (differently) on each call? I would have thought we could set the seed once before calling gcr_fgmodes_1d (indeed this is already done in gibbs_sample_with_fg using the value from the config file), then each slice would get the next chunk of random numbers from that seed starting point. This would give the reproducibility of the whole analysis when needed (by setting a seed in gibbs_sample_with_fg but not in gcr_fgmodes_1d), or we could just not set a seed anywhere, and get different results each time we run the code. After testing both scenarios, that's what looks to happen on my machine when I using the serial version of the code, using nproc > 1.

I see now that this is probably due to the multiprocessing map code behaviour that the processes spawned inherit the parent process' random state #11, even though I don't look to be able to reproduce this.

In any case, I think we can probably remove the multiprocessing code (#28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions