Multisplit simulation of a single neuron

[jackfolla]

Hi Ted,
I solved thanks to your suggestions...

Now, I'm working on single neuron multisplit,
but, from the material that I found on the net, I did not understand how I could assign different sections to different CPUs efficiently.

Furthermore, what happens to sections for which I don't express the multisplit?

Thanks, Pasquale.

[ted]

My first suggestion is to read this paper
Hines, M.L., Markram, H. and Schuermann, F.
Fully implicit parallel simulation of single neurons.
Journal of Computational Neuroscience 25:439-448, 2008
(preprint available from http://www.neuron.yale.edu/neuron/nrnpubs)

and examine the source code
http://senselab.med.yale.edu/modeldb/Sh ... odel=97985

Most of the magic is in proc multisplit(), which is defined in multisplit.hoc. init.hoc contains several examples of how to use this with different single cell models from ModelDB.

[jackfolla]

Hi Ted.

I tried to run the examples (to understand how they work), but they give errors.

However, now I'm trying to understand the proc multisplit().
If I were to have difficulties, I hope I can count on your suggestions.

Thanks, Pasquale.

[jackfolla]

Hi Ted.

I'm having some problems in understanding what parts of proc multisplit() are actually needed,
and the manner in which I can integrate this procedure in my model, which is very similar to poirazi (model = 1).

Can I safely delete proc read_mcomplex()?

Thanks.

[jackfolla]

Hi Ted,
I am having several problems with the Multisplit.

This is part of my code:

Code: Select all

objref pc
pc = new ParallelContext()
{load_file("loadbal.hoc")}
{load_file("binfo.hoc")}

if (name_declared("maxfactor") == 0) {execute("maxfactor = .3")}

proc multisplit() {local c, cm  localobj b, ms, vs, bi, cb, nc, nil
	b = new LoadBalance()
	//read_mcomplex(b)
    if (split > 1 || pc.nhost > 1) {
	ms = new Vector(100)
	if (pc.id == 0) {
		c = b.cell_complexity()
		cm = c * maxfactor / pc.nhost
		print "c = ", c, "  maxfactor = ", maxfactor, "  cm = ", cm
		b.multisplit(0, cm, ms)
		//bprint(ms)
	}
	pc.broadcast(ms, 0)
	vs = new VectorStream(ms)
	bi = new BalanceInfo()
	bi.msgid = 1e6
	bi.nhost = pc.nhost
	bi.ihost = pc.id
	bi.bilist.append(new CellBalanceInfo(vs))
	bi.mymetis2(pc.nhost)
	if (pc.id == 0) { bi.stat() }
	cb = bi.bilist.object(0)
	nc = new NetCon(&v(.5), nil)
	cb.multisplit(nc, 100, pc, pc.id)
    }
	//printf("%d cpu complexity %g\n", pc.id, b.cpu_complexity())
	cplx = b.cpu_complexity()
}

. . .

multisplit()

{pc.multisplit()}
{pc.set_maxstep(100)}
setuptime = startsw() - setuptime
if (pc.id == 0) {printf("setuptime = %g\n", setuptime)}

proc batchrun() { local i
  for i=0, n_simul-1 {
   . . .
   place_syn_GLU()
   place_syn_GABAa()
   place_syn_GABAb()
   place_syn_NMDA()
   runtime = startsw()
   stdinit(tstop)
   runtime = startsw() - runtime
   if (pc.id == 0) {printf("runtime = %g\n", runtime)}
   printf("%5d time %10g %10g %10g %5g\n", pc.id, pc.step_time(), pc.vtransfer_time(1), pc.vtransfer_time(2), cplx)
   pc.runworker()
   //getstat()
   pc.done()
   //perf2file()
   quit()
}  //end proc run

batchrun()

If I run myfile.hoc with

Code: Select all

nrngui -c "split=2" myfile.hoc

the behavior of the cell changes than the serial version.

If I run myfile.hoc with

Code: Select all

nrngui -c "split=1" myfile.hoc

the behavior of the cell is identical to the serial.

If I run myfile.hoc with

Code: Select all

mpiexec -np 2 nrniv -mpi -c "split=1" myfile.hoc"

or

Code: Select all

mpiexec -np 2 nrniv -mpi -c "split=2" myfile.hoc"

the program returns the following error:

Code: Select all

nrniv: apical_dendrite : section was deleted
0  in mysimctl2_Multisplit.hoc near line 273
0  batchrun()
           ^
        0 place_syn_GLU()
      0 batchrun()
    0 xopen("mysimctl2_Multisplit.hoc")
  0 execute1("{xopen("mysimctl2_Multisplit.hoc")}")
and others

nrniv: dendrite : section was deleted
1  in mysimctl2_Multisplit.hoc near line 273
1  batchrun()
           ^
        1 place_syn_GLU()
      1 batchrun()
    1 xopen("mysimctl2_Multisplit.hoc")
  1 execute1("{xopen("mysimctl2_Multisplit.hoc")}")
and others

Furthermore if I try to run http://senselab.med.yale.edu/modeldb/Sh ... odel=97985 with mpiexec
the program returns the following error:

Code: Select all

nrniv: subscript out of range x
0  in init.hoc near line 23
0  xrun()
       ^
        0 LoadBalance[0].ExperimentalMechComplex()
      0 mcomplex()
    0 chk_mcomplex()
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
rank 0 in job 4  pasquale-laptop_36704   caused collective abort of all ranks
  exit status of rank 0: return code 255 

Bye, Pasquale.

[ted]

I downloaded model 97985's zip file from ModelDB, expanded it, cd'd into its directory. Then I had to create a subdirectory called modeldb before I could get modelfix.sh to work. But what stopped me was that
mpiexec -n 4 nrniv -mpi init.hoc
fails with a chain of errors that start with

Code: Select all

1 nrniv: syntax error
1  in multisplit.hoc near line 11
1      if (split > 1 || pc.nhost > 1) {
                        ^

and sure enough, there isn't anything called "pc" when this line of code is parsed. I have to dig into this more . . .

[jackfolla]

fails with a chain of errors that start with

Code: Select all

1 nrniv: syntax error
1  in multisplit.hoc near line 11
1      if (split > 1 || pc.nhost > 1) {
                        ^

and sure enough, there isn't anything called "pc" when this line of code is parsed. I have to dig into this more . . .

Dear Ted,
I solved this error changing file init.hoc:

Code: Select all

{setuptime = startsw()}

if (!name_declared("split")) { execute("split = 0") }
if (!name_declared("model")) { execute("model = 2") }

{load_file("nrngui.hoc")}
{load_file("netparmpi.hoc")}
{load_file("trajec.hoc")}

objref pc, pnm
pnm = new ParallelNetManager(1)
pc = pnm.pc

{load_file("multisplit.hoc")}


{load_file("perf.hoc")}

In practice, I moved up

Code: Select all

objref pc, pnm
pnm = new ParallelNetManager(1)
pc = pnm.pc

over

Code: Select all

{load_file("multisplit.hoc")}

because pc is used in multisplit.hoc:

Code: Select all

if (split > 1 || pc.nhost > 1) {

Solved this error, occurs what I described to you.

[ted]

That does work. Good suggestion. It turns out that it is only necessary to ensure that an objref called pc exists, before loading multisplit.hoc. So in init.hoc I changed
{load_file("multisplit.hoc")}
to
objref pc
{load_file("multisplit.hoc")}

After making that change, I did the following (based on the contents of perfrun.sh and further suggestions from Michael Hines) in order to run a multisplit simulation of attenuation of a backpropagating spike using the Poirazi model:

1. Compiled the mechanisms
nrnivmodl modeldb/CA1_multi/mechanism
This produced an i686 subdirectory in the same directory as where multisplit.hoc, pltsplit.hoc etc. were located.

2. Ran
i686/special -nogui -c model=1 -c split=2 init.hoc
which created a file called mcomplex.dat. This file lists the names of the compiled mechanisms and their "complexity measures", which are used for load balancing in the next step. (Note: in this step, NEURON exits with an
execerror("stop", "")
message. This is deliberate, not a bug.)

3. Ran a multisplit simulation
mpiexec -np 4 nrniv -mpi -NSTACK 2000 -c model=1 init.hoc
NEURON printed numerous lines, the last of which were

Code: Select all

c = 10852   maxfactor = 0.3   cm = 813.9 
total complexity is 10921.4
1 cells
24 pieces
maximum complexity is 2739.6 for host 2
load imbalance is 0.3%
maximum of 6 pieces on host 0
at least one cell is broken into 24 pieces (bilist[0], gid 0)
at least one cell has 13 sids (bilist[0], gid 0)
 . . . 
setuptime = 0.35
runtime = 3.78
    1 time     3.3291   0.450758 0.00203443  2725
    3 time    3.41018    0.36975 0.00211477  2714
    2 time    3.38026   0.399674 0.00203133  2721
    0 time     3.7389  0.0414011 0.00833654  2715

The simulation also generated some files named trajec_*.dat that contained recordings of membrane potential at the middle of the soma and a couple of dendrites.

So to do multisplit simulation with a cell of your own choice, there are many steps.
1. The model files must be placed in a subdirectory of modeldb, and it may be necessary to make some changes to the hoc code so that the model loads but does not execute (see comments in modelfix.sh). Some mod files may also need to be revised (see modelfix.sh). Then the mod files must be compiled so that they are available when NEURON executes the init.hoc file in the multisplit directory.
2. Next, make your own copy of init.hoc and edit it so that the statements pertain to your own model. I'd just comment out the stuff that pertains to proc poirazi(), proc lazar(), and proc miyasho(), and make and execute my own proc mymodel(), which would look something like this:

Code: Select all

proc mymodel() {
  load_file("my_path/init.hoc")
  // tstop = whatever, if necessary
  multisplit()
  add_trajec("soma")
  add_trajec("somesection", 4) // or whatever section name and "index number"
  . . . etc. . . .
  // cvode_active(0) if necessary
}

mymodel()

3. Next, generate the mcomplex.dat file.
4. Finally, run the multisplit simulation.

[jackfolla]

Dear Ted,
I followed your suggestions and it runs.

Now I try to run my model.

[jackfolla]

Dear Ted,
it RUNS!!!

But I do not know if it is normal that the cell has 29 spikes, while in the serial version has 9 spikes.
Perhaps it depends on the number of splits?

[ted]

I do not know if it is normal that the cell has 29 spikes, while in the serial version has 9 spikes.
Perhaps it depends on the number of splits?

Model source code should generate the same result regardless of whether the cell is intact or split into pieces, how many pieces there are, where the splits were made, how many processors are running the simulation, or how the pieces are distributed over the processors. If it doesn't, there's a bug--the question is where. To diagnose and fix the problem we will have to be able to reproduce the symptom. If you are using a model from ModelDB, tell me which one it is; if it is your own "private" source code, you can zip it up (see
What to include in a zip file, and what to leave out
http://www.neuron.yale.edu/phpBB/viewto ... f=28&t=560
)
and email it to
ted dot carnevale at yale dot edu
and I will tell you what I find.

[jackfolla]

Dear Ted,
afeter several tests, I realized that there were small problems at compile-time.

Solved these problems, now I'm trying to run the program.

It runs, but there are a problem: not changing my model, in setuptime phase
my proc batchrun() is invoked, and so this runs all my model.

After, in runtime phase, proc dorun() (defined in init.hoc) is invoked, which calls proc statrun()...
In other words, in this phase, does not run my proc batchrun().

Maybe it is more difficult to explain that to see, so now I prepare the directory to be sent to your e-mail.

Thanks, Pasquale.

[jackfolla]

Dear Ted,
I solved the problem.

I have one last question:
If I wanted to run a family of simulations (with a loop for: RUN 1, RUN2 ...) , how can I modify proc dorun() in file init.hoc?

I'm trying following suggestions that you gave me for another program (in http://www.neuron.yale.edu/phpBB/viewto ... 2004#p7363), but, in this case, I can't do it.

I've also written on your e-mail.

[ted]

I solved the problem.

Excellent. How?

I have one last question:
If I wanted to run a family of simulations (with a loop for: RUN 1, RUN2 ...) , how can I modify proc dorun() in file init.hoc?

The code that implements the parameter changes from run to run must only be executed on those processors that have the mechanisms that are to be affected, and the particular parameter instances that are to differ from run to run. With a parallelized network in which the cells are NOT split, this is easy because each cell has a unique identifier (gid). When it comes time to change a parameter associated with any cell, each host just checks for the existence of the cell's gid, and if it is found, executes the code necessary to make the parameter changes.

Multisplit is implemented by a model setup strategy in which each host does the following:
1. Create a complete instance of the entire model cell on this host.
2. Delete all sections except those that belong to one of the pieces of the cell that are on this host.

I don't think there's anything for sections that is equivalent to a gid. I wonder if the right approach would be to build a SectionList on each host whose elements are the sections on that host whose parameters will be changed. The only other alternative would be to do a complete model cell setup before each run, followed by a teardown after the run is finished
REPEAT
select a new set of parameter values
create the entire model on each host, using these parameter values
on each host, delete all sectons except those that belong on this host
run a simulation
save results
destroy the entire model cell
UNTIL done
but that seems too inefficient.

[hines]

For split cells on a cluster machine, to change parameters in specific sections (after the cell has been split) it is necessary to use the function section_exists.
http://www.neuron.yale.edu/neuron/stati ... ion_exists
after obtaining the cell object, (eg. with cell = pc.gid2cell(gid)) on each machine where the gid exists.
(Though if you only have one cell at the top level that is unnecessary and no gid's are needed)

If you are only changing parameters between runs and not the topology, there is no need to
setup and destroy the cell on each run.