Neuron's mpiexec fails on Windows 10

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Neuron's mpiexec fails on Windows 10

Post by MBeining » Tue Oct 24, 2017 2:13 pm

When I use the mpi exec provided by NEURON (7.3 and 7.5, others not tested) on Windows 10 64 bit using this code

Code: Select all

C:\nrn\bin\mpiexec -n 4 nrniv -nobanner -nogui -mpi "neuron_runthis.hoc" -c quit()
I get this error:
ERROR: Failed to post close command error 1726
ERROR: unable to tear down the job tree. exiting...
Interestingly, when I install Microsoft MPI from here
https://msdn.microsoft.com/en-us/library/bb524831.aspx
and run the same code using

Code: Select all

mpiexec -n 4 nrniv -nobanner -nogui -mpi "neuron_runthis.hoc" -c quit()
it suddenly works on the machine!
Also, on a different Windows 10 machine, this code is working without the need of installing MPI!

Any idea what the problem might be? Would be more convenient if the mpiexec of NEURON could be used for my purposes ;-)

ted
Site Admin
Posts: 5334
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Neuron's mpiexec fails on Windows 10

Post by ted » Tue Oct 24, 2017 11:52 pm

I had no idea that NEURON installers for MSWin came with mpiexec bundled in. Always installed mpich2 from a 3rd party site. Fortunately it hasn't been necessary to use a 3rd party mpich2 for a while, since Microsoft has begun to provide a solid implementation of mpi that works beautifully with MSWin. Suggest you stick with the Microsoft package.

MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Re: Neuron's mpiexec fails on Windows 10

Post by MBeining » Fri Oct 27, 2017 8:23 am

ok thanks!

MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Re: Neuron's mpiexec fails on Windows 10

Post by MBeining » Sun Oct 29, 2017 1:54 pm

Unfortunately, I have to withdraw my latest post, as the problems continue with Microsoft MPI and using neuron.
Actually using Microsoft MPI works with other programs (test batch scripts etc) and works when running neuron without the -mpi argument, but as soon as I use

Code: Select all

"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 4 "C:/nrn/bin/nrniv.exe" -mpi "somesimplescript.hoc"
a few seconds nothing happens (actually I see that 4 nrniv.exe are started but they do not use CPU ressources) and then it returns to command line without anything happened (nrniv.exe are shut down again).

Using NEURON's mpiexec.exe works fine on my machine (unfortunately not on the other Win10 machine, as said before). I also tried to install mpich2 but gave up as they do not compile for Windows anymore and the old versions need some .NET Framework version that refuses to get installed on my machine...
So I tried, if maybe everything works with older NEURON versions, but when using NEURON 7.3 the following way:

Code: Select all

"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 4 "C:/nrn73/bin/nrniv.exe" -mpi "somesimplescript.hoc"
I get 4 times the following error
[unset]: unable to decode hostport from a862cb30-5a7a-498c-8b6b-ce9c61071579
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(433):
MPID_Init(139).......: channel initialization failed
MPID_Init(403).......: PMI_Init returned -1
I also tried NEURON 7.0 with nearly the same error code as result:
[unset]: unable to decode hostport from b890666b-35b5-4081-91a2-97d90c5921ec
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(294): Initialization failed
MPID_Init(82)........: channel initialization failed
MPID_Init(333).......: PMI_Init returned -1

MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Re: Neuron's mpiexec fails on Windows 10

Post by MBeining » Thu Nov 02, 2017 7:41 am

Ok I simplified the problem even more by just using the test0.hoc
The following section shows you the input/output from the NEURON bash shell when
1) starting parallel Neuron as described in the Neuron tutorial
2) checking which mpiexec the bash shell used
3) testing the two other mpiexec executables that are available (one gives no output at all, the other an error)

Code: Select all

bash.exe"-3.1$ mpiexec -n 2 nrniv -mpi /c/nrn/test0.hoc
NEURON -- Release 7.4 (1370:16a7055d4a86) 2015-11-09
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2015
See http://www.neuron.yale.edu/neuron/credits

numprocs=2
I am 1 of 2
I am 0 of 2
bash.exe"-3.1$
bash.exe"-3.1$
bash.exe"-3.1$ where mpiexec
c:\nrn\bin\mpiexec.exe
c:\Program Files\Microsoft MPI\Bin\mpiexec.exe
c:\Program Files\MATLAB\R2016b\bin\win64\mpiexec.exe
bash.exe"-3.1$
bash.exe"-3.1$
bash.exe"-3.1$ "/c/Program Files/Microsoft MPI/Bin/mpiexec" -n 2 nrniv -mpi /c/nrn/test0.hoc
bash.exe"-3.1$
bash.exe"-3.1$
bash.exe"-3.1$ "/c/Program Files/Matlab/R2016b/bin/win64/mpiexec" -n 2 nrniv -mpi /c/nrn/test0.hoc
Error while connecting to host, No connection could be made because the target machine actively refused it. (10061)
Connect on sock (host=XXXX, port=8676) failed, exhaused all end points
Unable to connect to 'XXXXX:8676',
sock error: Error = -1
It really seems the case that NEURON is not working with any other mpiexec implementation than from its own package. But what to do if that mpiexec is not working on a Windows machine?

ted
Site Admin
Posts: 5334
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Neuron's mpiexec fails on Windows 10

Post by ted » Thu Nov 02, 2017 4:37 pm

I had no idea that NEURON installers for MSWin came with mpiexec bundled in.
Actually, the mpiexec in /nrn/bin simply looks for msmpi, and doesn't do anything useful if msmpi doesn't exist. So you have to install the Microsoft mpi package if you want to take advantage of parallel execution.

ted
Site Admin
Posts: 5334
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Neuron's mpiexec fails on Windows 10

Post by ted » Thu Nov 02, 2017 5:25 pm

Under Win 7 with NEURON 7.5 master (2bae509) 2017-08-11, everything works well--
bash.exe"-3.1$ mpiexec -n 4 nrniv -mpi test0.hoc

NEURON -- VERSION 7.5 master (2bae509) 2017-08-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

I am 1 of 4
I am 2 of 4
numprocs=4
I am 0 of 4
I am 3 of 4
bash.exe"-3.1$

(that little " in the prompt is suggests a bug in the specification of the prompt string).

Under Win 10 with NEURON 7.5 master (6b4c19f) 2017-09-25 but using the same version of Microsoft mpi, I get this:
ERROR: Failed to post close command error 1726
ERROR: unable to tear down the job tree. exiting...

and the " is still in the prompt string, not that it necessarily means anything.

ted
Site Admin
Posts: 5334
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Neuron's mpiexec fails on Windows 10

Post by ted » Thu Nov 02, 2017 5:55 pm

I just now tried (5ac5339) 2017-10-17 under both Win 7 and Win 10, and sure enough, mpiexec . . . works properly under Win 7 but not under Win 10.

MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Re: Neuron's mpiexec fails on Windows 10

Post by MBeining » Fri Nov 03, 2017 3:18 am

Thank you for testing!
I had now the opportunity to run the test0.hoc with different NEURON versions under two different Windows 10 machines and the outcome was that it is not the Windows version but the combination of mpiexec and Neuron version:

Windows 10 Pro 64 bit, Version 1703 (OS Build 15063.674), NEURON Release 7.5 master (6b4c19f) 2017-09-25
The mpiexec from Neuron: fails with the post close command error
The mpiexec from Microsoft Open MPI: works fine.

Windows 10 Pro 64 bit, Version 1703 (OS Build 15063.674), NEURON Release 7.4 (1370:16a7055d4a86) 2015-11-09
The mpiexec from Neuron: works fine
The mpiexec from Microsoft Open MPI: does, as mentioned, not display any error but also does not run test0.hoc properly as no output is returned.

Windows 10 Enterprise 64 bit, Version 1607 (OS Build 14393.1593), NEURON Release 7.5 master (6b4c19f) 2017-09-25
The mpiexec from Neuron: fails with the post close command error
The mpiexec from Microsoft Open MPI: works fine. (with some "could not load python warning)

Windows 10 Enterprise 64 bit, Version 1607 (OS Build 14393.1593), NEURON Release 7.4 (1370:16a7055d4a86) 2015-11-09
The mpiexec from Neuron: works fine
The mpiexec from Microsoft Open MPI: does, as mentioned, not display any error but also does not run test0.hoc properly as no output is returned.

And concerning the latter: The mpiexec from Neuron 7.4 definitely also works WITHOUT Microsoft Open MPI as I can deinstall it and it still works. The only other mpiexec I have on my machine is the one from Matlab which has this connection error reported above.


Short note on NEURON 7.4 1370:
When installing it new and try the mpiexec from NEURON I get:
Aborting: mpi appplication on XXXXX is unable to connect to the smpd manager on (null):52017 error 1722
But with the installation I already have, this does not occur. I think it is because the version on your server only comes with mingw32 but my version has a mingw64 implementation, which, I think, is necessary on a 64bit machine. I guess you had a mingw64 available before on the server.

hines
Site Admin
Posts: 1517
Joined: Wed May 18, 2005 3:32 pm

Re: Neuron's mpiexec fails on Windows 10

Post by hines » Fri Nov 03, 2017 8:17 am

The last few installers for mswin inadvertently left out the msmpi.dll file. Please uninstall neuron and install the
nrn-7.5.master-1661.w64-mingw-py-36-35-27-setup.exe
at https://neuron.yale.edu/ftp/neuron/versions/alpha/
I don't think you have to uninstall the mpi you got from microsoft but it should not be needed.

MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Re: Neuron's mpiexec fails on Windows 10

Post by MBeining » Fri Nov 03, 2017 9:52 am

Thank you so much, it works now with Neurons mpiexec again! :-)
I still get the "post close command" error when trying to run this version with Microsofts MPI. I don't mind, but maybe it is of interest for you.

Last question:
mpiexec is not part of the Linux/Mac distributions, right? So I cannot assume that nrnfolder/bin/mpiexec will be there on Linux/Mac?

ted
Site Admin
Posts: 5334
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Neuron's mpiexec fails on Windows 10

Post by ted » Fri Nov 03, 2017 10:16 am

If you install from source code, you will not find mpiexec, test0.hoc, or test0.py in NEURON's installation tree. The former will be in wherever you installed mpi, and the latter two will be present in the source tree.

hines
Site Admin
Posts: 1517
Joined: Wed May 18, 2005 3:32 pm

Re: Neuron's mpiexec fails on Windows 10

Post by hines » Fri Nov 03, 2017 12:36 pm

mpiexec is not part of the Linux/Mac distributions, right?
I am not exactly sure. Certainly not the files needed for building NEURON from sources.
For linux, if there is no /usr/bin/mpiexec, use the appropriate package manager. E.g. for ubuntu one should

Code: Select all

sudo apt-get install openmpi-bin
For OSX, I have /usr/local/bin/mpiexec which suggests I might have built it from open-mpi.org tar.gz sources. But it might also be available with
an anaconda install or you can certainly install with macports or homebrew

MBeining
Posts: 33
Joined: Thu Apr 03, 2014 8:18 am

Re: Neuron's mpiexec fails on Windows 10

Post by MBeining » Sun Nov 05, 2017 6:22 am

ok thanks!

Post Reply