Multisplit simulations of a network (multiple neurons)

[jackfolla]

Dear All,
I would be interested in understanding how to make Multisplit simulations of a network (obviously with multiple neurons).

In the article "Fully Implicit Parallel Simulation of Single Neurons", paragraph "Network simulations" (page 14), there is a mention of this technique, but do not specify the differences in deployment than the version "single-cell" which are provided with examples.

In paragraph "Methods" is said: "The first network model used to test the multisplit algorithm in conjunction with load balancing is that of Traub et al (2005) using code modified from the ModelDB section of the Senselab database (http://senselab.med.yale.edu) and is the same as described in the previous paper (gap junctions turned off)."

I suppose that reference is relative to the network model which are available from ModelDB with accession number 45539 (Traub et al, 2005), but if the code has been changed, little may use this model to understand how to apply the multisplit to a network of neurons.

So, I ask you if I can find this modified code somewhere, or, if not, at least if it was possible to see a more detailed article about multisplit of network of neurons.

Thanks.

Pasquale.

[hines]

The multisplit version of the Traub model is in the repository
http://www.neuron.yale.edu/hg/z/models/nrntraub/
This model is very
complex in ways having nothing to do with multisplit and I'm not sure
the transformation from parallel whole cell to multisplit used in this
case is the best way to introduce the concept of multisplit. However
I'll assume you are now familiar with the whole cell parallel version
and try to descibe the major flow of control for setup of a multisplit
simulation.

Translate, compile, and link the mod files using
nrnivmodl mod

Suppose we want to run a balanced 356 cell simulation on 512 cores.

The first thing to do is decide on a proxy for guessing how long a
computation takes. A rough idea is to count the number of states for
each mechanism. More precisely, but of course not perfect, is to
experimentally determine the time it takes to compute each mechanism in
isolation as well as an empty compartment. The former (number of
states) is used by default. So it is optional but you can create the
experimental time measurement proxy for traub with:
nrniv -c load_balance_phase=1 init.hoc
which creates an mcomplex.dat file. If mcomplex.dat does not exist then
the proxy is the number of states.

Next, we need to decide how to distribute the cells and pieces of cells
among the nhost processes used for actual simulation so the expected computation
time on each process is as similar as possible. We do this in two steps via

mpiexec -np 4 nrniv -mpi -c load_balance_phase=6 -c multisplit_nhost=512 init.hoc

nrniv
load_file("binfo.hoc")
mymetis3("cx", 512)

The first creates a cx.dat file that contains information about dividing
the cells into pieces so that each piece has mxoptfactor less complexity than
"total network complexity"/nhost. The second reads this file and
creates a cx.512.dat file that contains the piece distribution on the ranks.

the last step is to do a simulation with
mpiexec -np 512 nrniv -mpi -c load_balance_phase=7 init.hoc

The substantive flow of control for load_balance_phase=7 is to:

read_multisplit_info("cx") in hoc/parlib.hoc which uses
a standard library binfo.hoc file to read the cx.512.dat file.

execute groucho.hoc which calls cell_append to create a cell and
nc_append to create a NetCon between source cell and target synapse.

The first occurrence of cell_append in groucho.hoc is on line 2703.
Note for parallel we call par_create in hoc/parlib.hoc
passing the cell_append statement as a string. There are 14 such
statements, one for each cell type. Note that the cell gid
is defined as the order (incrementing par_ncell on every call)
in which all cells are created in a serial model.
par_create calls hoc/multisplit_create(par_ncell, statement) in hoc/mscreate.hoc.
That looks in binfo to see if the gid is supposed to be created on this rank
(true if any piece is supposed to be on the rank). If so then the
statement is executed to create the whole cell (the whole cell may exist
on several ranks at this point). Then a standard library function, multisplit,
in binfo.hoc is called to use the binfo information to split the cell into
pieces, destroy all the pieces that don't belong on this rank, and
associate what is left of the cell with a gid which is the whole cell gid
if this host contains the cell piece that generates an output spike, and otherwise
gives it a spgid value which is unique to that portion of the cell on
this rank. There are functions in hoc/parlib.hoc which make it easy to get
the whole-cell gid if one knows the spgid on this host via
base_gid(spgid) and if one knows the wholecell gid, thishost_gid(gid) will
return spgid (or -1 if there is no piece associated with the cell on
this rank).

Now groucho continues on to create the connections by calling nc_append. The
first occurrence is on line 5732 and there are 229 of them in the 138 projections
among cell types. (The projection for loops also create synapses, often
as a pair of ampa/nmda.). For multisplit we added

if (!section_exists("comp", k, postsyncell_)) { continue }

in each of the 138 projection loops just before the synapses are created.
The section_exists is a built-in function in NEURON and will return true if
comp[k] exists in the postsyncell_ (note that at least a portion of
postsyncell_ exists because the first statement of a projection loop is,
e.g., for postsyn_cell_num = 1, num_suppyrRS \
if (gidexist(suppyrRS_base+postsyn_cell_num)) {
(see gidexist in net/serial_or_par_wrapper.hoc)

Note that load_balance_phase=6 creates a round-robin distribution of whole
cells on the available ranks and calls
print_multisplit_info(multisplit_prefix, multisplit_nhost)
in hoc/parlib.hoc. That iterates over all the cells on each rank and
calls functions in the standard library loadbal.hoc to compute the
total network complexity, and decide on an appropriate set of pieces consistent
with the multisplit method.