How to transfer Calcium concentrations between two tasks

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
nancyShu
Posts: 7
Joined: Tue May 24, 2016 2:27 am

How to transfer Calcium concentrations between two tasks

Post by nancyShu »

Dear all,

I'm simulating the information transfer (i.e., transferring the Calcium concentration at each time step) from a cell to a ribbon synapse, and the cell and the ribbon synapse are assigned to two cpus respectively. Is there any advice? Thanks!

PS: According to the parallelcontext tutorial, the function pc.source_var() and pc.target_var() can transfer source voltages to the target variables. I have compelled them to transfer Calcium concentration and failed (The error is: Pointer to v is not in the currently accessed section). My neuron version is 7.4 (1370:16a7055d4a86). How can I fix it?

ted
Site Admin
Posts: 5772
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: How to transfer Calcium concentrations between two tasks

Post by ted »

nancyShu wrote:I'm simulating the information transfer (i.e., transferring the Calcium concentration at each time step) from a cell to a ribbon synapse, and the cell and the ribbon synapse are assigned to two cpus respectively. Is there any advice?
First, a question: are the model cell and synapse both implemented with NEURON?

nancyShu
Posts: 7
Joined: Tue May 24, 2016 2:27 am

Re: How to transfer Calcium concentrations between two tasks

Post by nancyShu »

Thanks for your fast reply!
First, a question: are the model cell and synapse both implemented with NEURON?
Yes, both of them are implemented with NEURON.
Actually, I use the ribbon synapse model (accession:50997) provided by ModelDB.
And, the cell is defined by myself. It mainly has some calcium channels.

ted
Site Admin
Posts: 5772
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: How to transfer Calcium concentrations between two tasks

Post by ted »

Consider these two cases:

Case 1: neither the model cell nor the model synapse has a very high degree of complexity. Interprocessor communication overhead will penalize your simulations far more than any benefit you might get by trying to distribute your model over two or more processors.

Case 2: one of the models presents a high computational burden but the other does not. The processor with the light load will finish early and sit idle during each time step while the other is still grinding away, and you still won't see any benefit from creating a distributed implementation.

Case 2 may be amenable to multithreaded execution, which requires little or no effort on your part--just make sure that your mod files are threadsafe. Depending on model complexity, multithreaded execution on a shared memory architecture machine (e.g. one with a multicore CPU) can produce speedups that are nearly proportional to the number of processors that share a common memory, but the model must be of sufficient complexity to keep all allocated cores roughly equally busy.

There is a third case, in which both models present significant computational burdens. While the model cell can be distributed over multiple processors (this is called multisplitting), considerable program revision is required and this may require significant effort. I am unfamiliar with the ribbon synapse model but doubt that it is amenable to being split in a similar manner, so you could end up having difficulty achieving good load balance--and that can be a real performance killer.

The parsimonious approach is to start by combining both models into a single model, make sure it works, then try multithreaded execution (assuming that the mod files can be made threadsafe).

nancyShu
Posts: 7
Joined: Tue May 24, 2016 2:27 am

Re: How to transfer Calcium concentrations between two tasks

Post by nancyShu »

Thanks for your reply! Your advise is so professional!

I started by combining a pair of cell and synapse into a single model, and it works well.

Next, I would build a network composed by many cells and synapses with some predefined connections.
In this situation, I have to consider how to assign these models on a cluster for speedup.
Since I have no idea of the communication load, I planned to try the distribution way (i.e., for testing, put a single cell and single synapse on two CPUs respectively). Final decision will depends on the experimental results.

Thus, I have to try transferring Calcium concentrations from the cell to the synapse.
I wonder if there are any solution to transfer the analog signal between two tasks at each time step (e.g., If I use the pc.source_var and target_var, any tips?).
There is my pseudocode, it failed to transfer Calcium concentrations.

Code: Select all

load_file("nrngui.hoc")

create soma_pre, preSyn, preSyn2
create soma[1], dendrite[23]
objectvar stimulation1
objectvar synapse[6]

tbegin=0
tstop=16000
cvode.active(0)
dt=0.1

objref pc
pc = new ParallelContext()

st = pc.time()
print "nhost=", pc.nhost

index = 0
proc do() {
        pc.take("task", &index)
        print "working task index=", index, "on CPU ",pc.id()

	if ($1==1) {
		// task 1: soma_pre + presyn
		create soma_pre
		create preSyn,preSyn2
		// ... some initialization.

		objectvar stimulation1
		soma_pre stimulation1= new initStim(0.5)
		
		soma_pre connect preSyn(0), 0
		soma_pre connect preSyn2(0), 0	
		
		pc.source_var(&preSyn.Cai(0.5),128)	 // Calcium concentrations transfer 1
		pc.source_var(&preSyn2.Cai(0.5),256) // Calcium concentrations transfer 2
		pc.setup_transfer()
		pc.set_maxstep(5)
		stdinit()
		forall {finitialize(-65)}
		while(tbegin<tstop) { fadvance() }
	} else {
		// task 2: soma_post
		create soma_post[1]
		create dendrite[23]
		// ... some initialization.
		
		objectvar synapse[6]
		access dendrite[22]
		for i=0,5{
			synapse[i] = new ribbon_tiger(0.5)
			pc.target_var(&synapse[i].preCA1,128)	// Calcium concentrations transfer 1
			pc.target_var(&synapse[i].preCA2,256)	// Calcium concentrations transfer 2
		}
		
		pc.setup_transfer()
		pc.set_maxstep(5)
		stdinit()
		forall {finitialize(-65)}
		while(tbegin<tstop) { fadvance() }
	}
     
    pc.post("done", index)
}

pc.runworker()

for i=1,2 {
        pc.post("task", i)
        pc.submit("do",i)
}

while (pc.working()) {
        pc.take("done", &index)
        print "                        Data center: done index = ", index
}

pc.done()
print "Cost time:",pc.time()-st
quit()
PS: Guided by the parallelcontext tutorial, I consider the communication between two tasks only.
Your reply reminds me to check the multi-threads implementation. Thanks!

ted
Site Admin
Posts: 5772
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: How to transfer Calcium concentrations between two tasks

Post by ted »

Thanks. User support is an important part of the NEURON project.

Examining the code used by Sikora et al., I see two things that prevent the use of multithreaded simulation of models that use their synaptic mechanism.

First, several of the mechanisms used by their model are not threadsafe. In some cases this is because they contain GLOBAL variables. The files and the variables are
caconc.mod: GLOBAL caconc
modulate.mod: GLOBAL mod
ribbon_tiger.mod: GLOBAL total_count
ribbon_tiger.mod: GLOBAL vmin, vmax
rib_ufp.mod: GLOBAL total_count, Orate
rib_ufp.mod: GLOBAL vmin, vmax
At least a few of these variables appear on the left hand side of assignment statements. This is probably fixable, but judgement and effort will be required, including testing to make sure the revised mechanisms still work properly.

There are also VERBATIM blocks in
caconc.mod
cagf.mod
modulate.mod
spike.mod
tsbp.mod
vclmp.mod (3 separate VERBATIM blocks)
Unfortunately, some of these VERBATIM blocks perform nontrivial tasks, e.g. in
caconc.mod
modulate.mod
vclmp.mod (3 separate VERBATIM blocks)
Fixing these may be possible but it will not be trivial--judgement, effort, and time will be required.
nancyShu wrote:Next, I would build a network composed by many cells and synapses with some predefined connections.
In this situation, I have to consider how to assign these models on a cluster for speedup.
Not the way to go. First build a model designed for serial execution on a single processor. It is much easier to verify correct implementation and operation of such a model. A serial implementation is particularly important if this is your first foray into modeling with NEURON. Even if you are already an experienced NEURON user, a serial implementation is essential because it provides "standard model output" and "baseline performance measures" against which you can check your parallelization attempts. How else are you going to know that the parallelized model is working properly, and that it is indeed faster than the serial implementation?
Thus, I have to try transferring Calcium concentrations from the cell to the synapse.
Your parallelized implementation will involve "multisplitting" of model cells. Where to split, and the nature of the analog variable that must be communicated from the presynaptic cell to the synapses that are attached to postsynaptic targets, depends on the answers to these questions about the synapses that a presynaptic cell makes onto its targets.

1. Does each synapse see the same presynaptic calcium concentration (all are anatomically very close to each other), or must every synapse keep track of its own local calcium concentration?

2. Does each synapse see the same presynaptic membrane potential (all are electrically close to each other), or might they see significantly different membrane potentials (perhaps because they are located at very different places in the presynaptic cell)?

nancyShu
Posts: 7
Joined: Tue May 24, 2016 2:27 am

Re: How to transfer Calcium concentrations between two tasks

Post by nancyShu »

Many thanks for your checking on the Ribbon Synapse Model! I will take care of them.
"First build a model designed for serial execution on a single processor."
I have tried the serial implementation, and it works well.
The pseudocode is:

Code: Select all

load_file("nrngui.hoc")

create soma_pre
create preSyn,preSyn2
// ... some initialization.
objectvar stimulation1
soma_pre stimulation1= new initStim(0.5)
soma_pre connect preSyn(0), 0
soma_pre connect preSyn2(0), 0	

create soma_post[1]
create dendrite[23]
// ... some initialization.		
objectvar synapse[6]
access dendrite[22]
for i=0,5{
	synapse[i] = new ribbon_tiger(0.5)
	setpointer synapse[i].preCA1,preSyn.Cai(0.5)
	setpointer synapse[i].preCA2,preSyn2.Cai(0.5)
}

tbegin=0
tstop=16000

forall{ finitialize() }
secondorder = 2
cvode.active(0)
objref file 
file = new File()
file.wopen("v.dat")
while (tbegin < tstop){
    fadvance()
	file.printf("%g\n",dendrite[13].v)
}
file.close()
1. Does each synapse see the same presynaptic calcium concentration (all are anatomically very close to each other), or must every synapse keep track of its own local calcium concentration?

2. Does each synapse see the same presynaptic membrane potential (all are electrically close to each other), or might they see significantly different membrane potentials (perhaps because they are located at very different places in the presynaptic cell)?
We can see from the serial implementation code that each synapse see the same presynaptic calcium concentration and the same presynaptic membrane potential. How can I transfer this kind of analog variables?

ted
Site Admin
Posts: 5772
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: How to transfer Calcium concentrations between two tasks

Post by ted »

nancyShu wrote:each synapse see the same presynaptic calcium concentration and the same presynaptic membrane potential.
Actually, according to the NMODL code in ribbon_tiger.mod, this synaptic mechanism doesn't use presynaptic membrane potential at all--it only needs to know calcium concentration at two depths in the presynaptic terminal.

Documentation of parallel transfer contains this cautionary message:
"Communication overhead for such models is far greater than when the only communication between cells is with discrete events." (see https://www.neuron.yale.edu/neuron/stat ... l-transfer)
Are you quite sure that it is necessary to parallelize the model? Do you already have performance metrics for simulation of a serial (single-processor) implementation of the model network?
How can I transfer this kind of analog variables?
The necessary ParallelContext methods are source_var(), target_var(), and setup_transfer(). You won't need splitcell() or multisplit(), because you only need to set up one-way communication of calcium concentrations from the presynaptic segment of the presynaptic neuron to a synaptic mechanism instance attached to the postsynaptic cell.

These methods were used in models reported in these papers:
Hines M, Eichner H, Schuermann F (2008) Neuron splitting in compute-bound parallel network simulations enables runtime scaling with twice as many processors J Comput Neurosci 25(1):203-210, ModelDB entry 97917
and
Hines ML, Markram H, Schuermann F (2008) Fully Implicit Parallel Simulation of Single Neurons J Comp Neurosci 25:439-448. ModelDB entry 97985
Download the source code for these and search their contents for the method names. Of course it would also be a good idea to see if clear examples are provided in the papers, which are available from links at http://www.neuron.yale.edu/neuron/nrnpubs

Post Reply