Using Multisplit on NSG

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
agmccrei
Posts: 14
Joined: Thu Sep 10, 2015 1:34 pm

Using Multisplit on NSG

Post by agmccrei » Mon Jan 02, 2017 6:51 pm

Hello,

I am new to parallel processing in NEURON but it has become necessary to implement due to the increasing complexity of my model and the size of parameter searches that I am running. I am running repeated simulations of a morphologically detailed multi-compartment model with 1903 Exp2Syn synapses spread across all of the compartments of my model. I first started exploring the parallel processing options offered in the GUI to reduce simulation times. Here are some of my findings from that:

Model complexity before adding synapses: 1897
Model complexity after adding synapses: 7606
Multisplit over 8 processors = splits cell into 43 pieces (in this case) that are distributed over multiple processors
10s simulation duration with synapses, without multisplit = ~53 s
10s simulation duration with synapses, with multisplit = ~15 s
10s simulation duration without synapses, without multisplit = ~16 s
10s simulation duration without synapses, with multisplit = 7.1 s

With these reduced simulation times I decided to implement the following code, based on the code found in the session file when you save a session that includes the parallel processing GUI. For simplicity, I omitted the parameter search function code - essentially, the run() function is just called on many iterations of a parameter search.

SynParamSearch.hoc:

Code: Select all

load_file("DetailedModel.hoc")
func f() {
       // Some Parameter Search Function
}

// Set up parallel context
objectvar pct
load_file("parcom.hoc", "ParallelComputeTool")
pct = ParallelComputeTool[0]
object_push(pct)
change_nthread(8, 1)
multisplit(1)
busywait(1)
And then in a separate file:

Code: Select all

load_file("SynParamSearch.hoc")
f()
While this seems to work and generate reduced simulation times on both my laptop (Macbook with 4 processors) and a my work desktop (Ubuntu with 8 processors), I immediately receive an error when I try to run this code on the neuroscience gateway (NSG):

Code: Select all

NEURON -- Release 7.4 (1341:2ccabed20677) 2015-05-31
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2015
See http://www.neuron.yale.edu/neuron/credits

Additional mechanisms from files
 mod_nsgportal/IKa.mod mod_nsgportal/ingauss.mod mod_nsgportal/Ksoma.mod mod_nsgportal/Nap.mod mod_nsgportal/Nasoma.mod mod_nsgportal/vecevent.mod
Assertion failed: file multisplit.cpp, line 2633
0 /projects/ps-nsg/home/nsguser/applications/neuron7.4/installdir/x86_64/bin/nrniv: ie < ne
0  in SynParamSearch.hoc near line 342
0  multisplit(1)
              ^
        0 ParallelContext[0].multisplit()
      0 ParallelComputeTool[0].use_thread()
    0 ParallelComputeTool[0].multisplit(1)
  0 xopen("SynParamSe...")
and others
0 /projects/ps-nsg/home/nsguser/applications/neuron7.4/installdir/x86_64/bin/nrniv: Segmentation violation
0  in SynParamSearch.hoc near line 3
0  f()
    ^
        0 fadvance()
      0 advance()
    0 step()
  0 continuerun(10000)
and others
Unfortunately, I am unsure of how to proceed since I am not very knowledgeable in high-performance computing and parallel processing. It does seem clear that the error is due to "multisplit(1)", since I do not receive an error immediately after submitting the job to NSG once I remove that line of code.

Thanks for your time,

Alex GM

ted
Site Admin
Posts: 5266
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Using Multisplit on NSG

Post by ted » Thu Jan 05, 2017 5:48 pm

This is one case in which I wouldn't try extracting code from a ses file and incorporating it into a hoc file.
Just use the GUI's Parallel Simulation tool ON YOUR OWN LOCAL MACHINE to specify the number of threads,
save that tool to a ses file all by itself, and then put a
load_file("foo.ses")
statement into the hoc code that you want to run on the NSG, after model setup is complete but before calling run().
The ses file's gui-specific code will be ignored, but all the "administrative statements" that are required to configure multithreaded execution properly will be executed, and your simulation should run without a problem.

agmccrei
Posts: 14
Joined: Thu Sep 10, 2015 1:34 pm

Re: Using Multisplit on NSG

Post by agmccrei » Mon Jan 16, 2017 10:43 pm

Thank you for your reply Dr. Carnevale. I tried doing as you suggested, but I only seem to be receiving a slightly more detailed version of the previous error:

stderr.txt:

Code: Select all

Use of uninitialized value in multiplication (*) at /opt/sdsc/bin/ibrun line 99.
Use of uninitialized value in multiplication (*) at /opt/sdsc/bin/ibrun line 232, <NODEFILE> line 24.
Use of uninitialized value in division (/) at /opt/sdsc/bin/ibrun line 287, <NODEFILE> line 24.
NEURON -- Release 7.4 (1341:2ccabed20677) 2015-05-31
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2015
See http://www.neuron.yale.edu/neuron/credits

Additional mechanisms from files
 mod_nsgportal/IKa.mod mod_nsgportal/ingauss.mod mod_nsgportal/Ksoma.mod mod_nsgportal/Nap.mod mod_nsgportal/Nasoma.mod mod_nsgportal/vecevent.mod
Assertion failed: file multisplit.cpp, line 2633
0 /projects/ps-nsg/home/nsguser/applications/neuron7.4/installdir/x86_64/bin/nrniv: ie < ne
0  in foo.ses near line 15
0  {multisplit(1)}
                ^
        0 ParallelContext[0].multisplit()
      0 ParallelComputeTool[0].use_thread()
    0 ParallelComputeTool[0].multisplit(1)
  0 xopen("foo.ses")
and others
0 /projects/ps-nsg/home/nsguser/applications/neuron7.4/installdir/x86_64/bin/nrniv: Segmentation violation
0  in foo.ses near line 3
0  ^
        0 fadvance()
      0 advance()
    0 step()
  0 continuerun(10000)
and others
And here is the end of the stdout.txt file:

Code: Select all

nthread walltime (count to 1e8 on each thread)
nthread walltime (count to 1e8 on each thread)
nthread walltime (count to 1e8 on each thread)
nthread walltime (count to 1e8 on each thread)
nthread walltime (count to 1e8 on each thread)
nthread walltime (count to 1e8 on each thread)
nthread walltime (count to 1e8 on each thread)
   1	 0.0180399
   1	 0.0172892
   1	 0.0181921
   1	 0.018167
   1	 0.0347099
   1	 0.0347118
   1	 0.0339501
   2	 0.0179939
   2	 0.0183091
   2	 0.0179341
   2	 0.0181119
   2	 0.018188
   4	 0.018043
   4	 0.0179501
   4	 0.0180528
   2	 0.034668
   2	 0.0390191
   4	 0.038744
   4	 0.0271981
   8	 0.0349479
   4	 0.0244889
   8	 0.032109
   8	 0.0456769
   4	 0.039685
   8	 0.0347021
   8	 0.0409641
   8	 0.0432718
  16	 0.0449109
  16	 0.0482471
  32	 0.0461781
  16	 0.047076
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
I get this error when loading the session file within the parameter search function, right before the run() call:

Code: Select all

load_file("DetailedModel.hoc")
func f() {
       // Some Parameter Search Function
       load_file("foo.ses")
       run()
}
I also get a similar error when I try loading the session file in the init.hoc file:

Code: Select all

load_file("SynParamSearch.hoc")
load_file("foo.ses")
f()
I will continue to troubleshoot. I'm wondering if it is just the order in which I have structured my code. Or possibly that I need to disable parallel processing somehow in between each iteration of the parameter search, since I adjust the inputs to the synapses through manipulation of VecStim objects on each iteration of the parameter search. Could that be leading to a segmentation violation, specifically when run on NSG (since I do not receive this error when running on my local machines)? Then again, if that were to case, I would expect to receive at least one output file when I put the "load_file("foo.ses")" line right before "run()", which is not the case.

Alexandre Guet-McCreight

Post Reply