Analyzing running Multinest chain

[subhajitghosh-phy]

Hello,

This may be a silly question. Is there a way to analyze currently running Nested Sampling (NS) chains in Montepython?

We can analyze running MH chains by the standard 'montepython info'. However, for Multinest ( -m NS) we have to wait till the runs are finished. Otherwise 'clean_conversion' function throws out an error which I guess fails to execute the command 'from_NS_output_to_chains'. The error message does not give any useful info at which stage of 'from_NS_output_to_chains' it's failing.

Can this feature (analyze currently running NS chains) be introduced in Montepython? Or is it a question for MultiNest/PyMultiNest people?

If anybody has any idea please let me know.

Best,
Subhajit

[brinckmann]

Hi Subhajit,

Once your NS chains have reached a large enough amount of accepted points it should be able to be analyzed, is this not so? In my experience, it can take quite a while to reach a stage where it can be analyzed, since MultiNest is slow. In the past this has worked for me, so I wonder if it broke with a recent python version change or something. How many points has it collected and what is the error?

To take a moment to promote something new, I'm personally excited to try out UltraNest (see the pull request #315 by Johannes Buchner) for my next project involving nested sampling, as I think it should be faster than the other existing options, but I haven't had a chance to try it yet.

Best,
Thejs

[subhajitghosh-phy]

Hi Thejs,

Thanks for the reply. Sorry, I made a silly mistake that's why the NS analysis was failing.

I was not very comfortable running info on a running NS chain (worried that it might mess up the sampling which is running for a long time. Just being extra cautious). I made a copy of the output directory and the error I was getting in the filename mismatch.

---

Traceback (most recent call last):
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/analyze.py", line 1527, in clean_conversion
getattr(module, 'from_%s_output_to_chains' % tag)(folder)
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/MultiNest.py", line 326, in from_NS_output_to_chains
with open(base_name+name_arguments, 'r') as afile:
FileNotFoundError: [Errno 2] No such file or directory: 'NS_nnoutN20_c/NS/NS_nnoutN20_c.arguments'

During the handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/MontePython.py", line 40, in
sys.exit(run())
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/run.py", line 31, in run
cosmo, data, command_line, success = safe_initialisation(
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/run.py", line 191, in safe_initialisation
cosmo, data, command_line, success = initialise(custom_command)
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/initialise.py", line 59, in initialise
analyze(command_line)
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/analyze.py", line 101, in analyze
status = prepare(item, info)
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/analyze.py", line 218, in prepare
action_done = clean_conversion(module_name, tag, files[0])
File "/afs/crc.nd.edu/user/s/sghosh5/nnat/mcmc/montepython_public-3.5/montepython/analyze.py", line 1529, in clean_conversion
raise io_mp.AnalyzeError(
io_mp.AnalyzeError:

Analyze Error:
/|\ You asked to analyze a NS folder which seems to come from an unfinished
/o\ run, or to be empty or corrupt. Please make sure the run went smoothly
enough.

---

I overlooked the top portion of the error. My apologies. I have fixed that and I can now analyse the chains. Thanks.

Regarding UltraNest, it would be great to have it implemented in Montepython. Will keep an eye out for that.

Since we are talking about new features, I would like to put one other request. One of the reasons why MultiNest is slow is that it cannot be run across nodes via MPI since CLASS itself does not support MPI. Typically I just run one MultiNest chain. Going forward, if CLASS is made MPI enabled that would also help with the speed. (This is also more regarding to CLASS than MontePython)

Thanks.

Best,
Subhajit