Pause/Stop NEURON simulation to call run another program

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
catubc
Posts: 21
Joined: Mon Apr 07, 2014 6:46 pm

Pause/Stop NEURON simulation to call run another program

Post by catubc »

Hi all. I've finally got my extracellular potential model to work properly, and now I'd like to be able to compute the potential every second or so of NEURON simulation.

I'd like to be able to "pause" or somehow hang NEURON up every second of simulation and call another program (in FORTRAN or perhaps C) to compute the potentials. Is it possible to get NEURON to dump the memory to a swap and for the nodes to take up another task for a while and then return to NEURON?

I see there is this procedure called "savestate", is this what I should be looking at?

Thanks in advance.
catubc
ted
Site Admin
Posts: 5784
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Pause/Stop NEURON simulation to call run another program

Post by ted »

Simplest would be just write the membrane currents to a file every second or so, then after the end of the simulation feed that file to your program that computes the field. If you really must interleave simulation execution and field calculation, I'd suggest using an FInitializeHandler, cvode.event(), and system(). Something like this:

Code: Select all

START = 0 // first time at which field is calculated
INTERVAL = 1000 // between successive calculations of field
objref fih
fih = new FInitializeHandler("initcalc()")
proc initcalc() {
  cvode.event(START, "docalc()")
// print "launched event to trigger field calc at ", START
}
proc docalc() {
// print "calculating field at t = ", t
  system("command_that_computes_field")
  cvode.event(t + INTERVAL, "docalc()")
// print "launched event to trigger next calc"
}
If your "field calculating program" allows command line specification of the output file name, your proc docalc() could use an sprint statement like this

Code: Select all

strdef cmdstr
proc docalc() {
// print "calculating field at t = ", t
  sprint(cmdstr, "calcfield field%d", t) // assumes program name is calcfield
    // and argument is name of output file
  system(cmdstr)
  cvode.event(t + INTERVAL, "docalc()")
// print "launched event to trigger next calc"
}
catubc
Posts: 21
Joined: Mon Apr 07, 2014 6:46 pm

Re: Pause/Stop NEURON simulation to call run another program

Post by catubc »

Thanks Ted. I'll take a look at your code... it makes sense already. I'm assuming that the CPU cores will be offloaded on system call? That's my main concern as I'd like to be able to use NEURON on clusters and the lsa subroutines also are optimized for multithreading. catubc
ted
Site Admin
Posts: 5784
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Pause/Stop NEURON simulation to call run another program

Post by ted »

catubc wrote:I'm assuming that the CPU cores will be offloaded on system call?
system() works when called during a single threaded simulation so the answer is "yes". I'd expect it to work regardless of whether the "external" program is single or multithreaded.
catubc
Posts: 21
Joined: Mon Apr 07, 2014 6:46 pm

Re: Pause/Stop NEURON simulation to call run another program

Post by catubc »

Ted

I've used the equivalent os.system call in python inside Michael's pysave_imembrane() function which as far as I researched is the only way for me to access the time value "t" in a parallel implementation.

Code: Select all

def pysave_imembrane(): #Hines' pybs.mod method calls this procedure at every time step
  
  for gid in cells:
    ... #This code saves all the currents for all the cells to binary files
 
     #Want to call executable  "a.out" every 1000 ms  
     if ((index1 % 1000) == 0 and index1>0 and gid==0):
       os.system("/home/cat/neuron/eaps/a.out")
I solved one problem (the executable was being called multiple times) but my executable is not running multithreaded... it seems that on execution of the "a.out" all but one of the cores remain loaded with previous NEURON jobs and they all wait for the single core running the "a.out" to finish. However, when I run the "a.out" directly from a shell, it runs about 8-10 x faster and loads 100% of cpus (on a multi-core machine).

Is there any way to unload the other cores/threads from NEURON during this system call? Basically I need to drop out of the parallel context into single core context for the system call... Is this possible?

I guess the alternative is to let NEURON/python manage the multi-threading and run X instances of the "a.out" executable... I'm going to try that as well just to see if it's possible... though I might run out of memory.

Thanks very much,
catubc
catubc
Posts: 21
Joined: Mon Apr 07, 2014 6:46 pm

Re: Pause/Stop NEURON simulation to call run another program

Post by catubc »

Ted

So it seems NEURON can run multithreaded system calls... The only issue is about a 25%-30% loss in efficiency (when comparing running the executables from NEURON vs. running them after NEURON terminates).

I will work on this for a few days, but if you have any other suggestions, they are much appreciated.

Thanks very much,
catubc
Post Reply