usable mindelay is 0

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
dave_arthurs
Posts: 6
Joined: Fri Jun 25, 2010 6:52 pm

usable mindelay is 0

Post by dave_arthurs » Tue May 20, 2014 1:03 am

Hi folks,

I'm trying to get the model by Brette et al. 2007 "Simulation of networks of spiking neurons: a review of tools and strategies" (accession 83319) working in Neuron-7.3 on OS X Mavericks 10.9.2.

I have done the following:
1) Fresh install of Neuron from the DMG, set up path in .bash_profile
2) Install openmpi via homebrew ($ brew install openmpi)
3) Run the test scripts in ~/neuron/nrn/src/parallel ($ mpirun -n 4 nrniv -mpi test0.hoc, etc)

The test scripts test0.hoc to test7.hoc, and also testsw1.hoc, all run without error.

However, when I do a fresh download of the Brette model (http://senselab.med.yale.edu/modeldb/Sh ... odel=83319) and run it with the following, I get errors in both parallel and serial mode.

Serial error:

Code: Select all

$ cd ~/src/delta/destexhe_benchmarks/NEURON
$ nrnivmodl coba cuba cubadv
... (output, no errors) ...
$ cd cobahh
$ nrngui init.hoc 
NEURON -- VERSION 7.3 ansi (1078:2b0c984183df) 2b0c984183df
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2014
See http://www.neuron.yale.edu/neuron/credits

Created 4000 cells; 4000 on host 0
Created 320000 connections to targets on host 0
SetupTime: 0.75999999 
/Applications/NEURON-7.3/nrn/x86_64/bin/nrniv: Segmentation violation
 in init.hoc near line 27
 prun()  // in common/perfrun.hoc
                                 ^
        ParallelContext[0].psolve(1000)
      ParallelNetManager[0].psolve(1000)
    prun()


Parallel error

Code: Select all

$ mpirun -n 4 nrniv -mpi init.hoc
numprocs=4
NEURON -- VERSION 7.3 ansi (1078:2b0c984183df) 2b0c984183df
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2014
See http://www.neuron.yale.edu/neuron/credits

Created 4000 cells; 1000 on host 0
Created 80000 connections to targets on host 0
SetupTime: 0.21000004 
0 nrniv: usable mindelay is 0 (or less than dt for fixed step method)
0  in init.hoc near line 27
0  prun()  // in common/perfrun.hoc
                                 ^
        0 finitialize(-60)
      0 init()
    0 stdinit()
  0 prun()
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
A couple of things I've tried so far:
- The intrinsic mode (i.e. running intrinsic.hoc instead of init.hoc) works fine
- I tried also manually downloading and compiling openmpi from source (from directions here http://www.neuron.yale.edu/neuron/downl ... ilestd_osx) and setting the paths (LD_LIBRARY_PATH). The same errors still occur

I am running out of things to try, short of trying on a different machine. Any ideas?

Also, Ted, many thanks for your help with my prior questions. Based on your reply to my "Best Practices" thread (viewtopic.php?f=15&t=3076), I've decided that the best way to produce clean code is to start with this example network and add my mechanisms from there.

ted
Site Admin
Posts: 5332
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: usable mindelay is 0

Post by ted » Wed May 21, 2014 10:47 am

Using a somewhat older version of NEURON (7.3 (999:cdec6759122c) 2014-01-10) under CentOS 6.5, serial execution of the cobahh network produces no error, but parallel execution gives me the same error message you saw. Will look into this further.

hines
Site Admin
Posts: 1517
Joined: Wed May 18, 2005 3:32 pm

Re: usable mindelay is 0

Post by hines » Wed May 21, 2014 11:28 am

Remember that parallel only works if the minimum interprocessor NetCon delay is greater than dt (or > 0 for the variable step method)

hines
Site Admin
Posts: 1517
Joined: Wed May 18, 2005 3:32 pm

Re: usable mindelay is 0

Post by hines » Wed May 21, 2014 11:50 am

I was unable to reproduce the serial segmentation violation using ansi 1085 om my mac osx 10.9
Also on ubuntu 12.04 using ansi 1078.

can you run with
gdb nrniv
run init.hoc
and when the segmentation error occurs, type
where
and send me the stack trace. Perhaps we should take this offline to email as it may take a few rounds of communication to resolve the problem.
email to michael dot hines at yale dot edu

dave_arthurs
Posts: 6
Joined: Fri Jun 25, 2010 6:52 pm

Re: usable mindelay is 0

Post by dave_arthurs » Wed May 21, 2014 3:47 pm

Thank you both for your feedback. Michael, I have sent you an email with the trace.

ted
Site Admin
Posts: 5332
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: usable mindelay is 0

Post by ted » Wed May 21, 2014 7:48 pm

hines wrote:Remember that parallel only works if the minimum interprocessor NetCon delay is greater than dt (or > 0 for the variable step method)
True. Also, I need to remember to read the readme etc. that comes with source code, even if I was involved in writing it, especially when it has been more than a couple of years since I last looked at it.

The NetCon delays are specified by the value assigned to the symbolic constant DELAY in common/net.hoc. As distributed from ModelDB, this value is 0. Changing that assignment statement to
DELAY = 1 // (ms)
eliminates that error message.

dave_arthurs
Posts: 6
Joined: Fri Jun 25, 2010 6:52 pm

Re: usable mindelay is 0

Post by dave_arthurs » Wed May 21, 2014 10:01 pm

I appreciate your feedback and your offline help. Your solutions worked. To summarize:

- As per your suggestions, the parallel error (usable mindelay is 0) was fixed by setting "DELAY = 1" in destexhe_benchmarks/NEURON/common/net.hoc
- The segmentation fault when running in serial mode was fixed by compiling from sources, using the instructs here (http://www.neuron.yale.edu/neuron/downl ... ilestd_osx) and making sure to include Michael's addendum:
if you use the --with-paranrn=dynamic in the configure of part D, then before the 'make' step
you should
(cd src/nrnmpi ; sh mkdynam.sh )
A few other notes:
- I verified that the segmentation fault error also occurs in 64-bit Ubuntu 13.10, running the deb version. I presume compiling from source here would solve the problem as well
- The segmentation error did not occur with Neuron installed on our server (Cent OS 6.4)

Thanks again for your help!

Post Reply