How to make VecStim (vecevent.mod) thread safe?

NMODL and the Channel Builder.
Post Reply
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Fri Feb 13, 2015 5:00 pm

I find VecStim very useful for driving spikes with a list of spike times from python. However, I'm trying to make all my .mod files thread safe and cvode-compatible. Compiling vecevent.mod results in:

Code: Select all

Translating vecevent.mod into vecevent.c
Notice: VERBATIM blocks are not thread safe
Any ideas? Here is the entire .mod file for convenient reference:

Code: Select all

:  Vector stream of events

NEURON {
	ARTIFICIAL_CELL VecStim
}

ASSIGNED {
	index
	etime (ms)
	space
}

INITIAL {
	index = 0
	element()
	if (index > 0) {
		net_send(etime - t, 1)
	}
}

NET_RECEIVE (w) {
	if (flag == 1) {
		net_event(t)
		element()
		if (index > 0) {
			net_send(etime - t, 1)
		}
	}
}

VERBATIM
extern double* vector_vec();
extern int vector_capacity();
extern void* vector_arg();
ENDVERBATIM

PROCEDURE element() {
VERBATIM	
  { void* vv; int i, size; double* px;
	i = (int)index;
	if (i >= 0) {
		vv = *((void**)(&space));
		if (vv) {
			size = vector_capacity(vv);
			px = vector_vec(vv);
			if (i < size) {
				etime = px[i];
				index += 1.;
			}else{
				index = -1.;
			}
		}else{
			index = -1.;
		}
	}
  }
ENDVERBATIM
}

PROCEDURE play() {
VERBATIM
	void** vv;
	vv = (void**)(&space);
	*vv = (void*)0;
	if (ifarg(1)) {
		*vv = vector_arg(1);
	}
ENDVERBATIM
}

ted
Site Admin
Posts: 5623
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: How to make VecStim (vecevent.mod) thread safe?

Post by ted » Sun Feb 15, 2015 2:50 pm

VecStim is not going to be thread safe. It uses VERBATIM blocks, and as per the message emitted by nrnivmodl (or mknrndll), VERBATIM blocks are not thread safe. Which isn't really much of a loss because multithreaded parallel execution is not as efficient as mpi-based parallel execution.

Strong suggestion: use the latest implementation of VecStim--see
viewtopic.php?f=2&t=2147&p=13614#p13614

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Sun Feb 15, 2015 6:09 pm

Thanks for the updated vecevent.mod

So even if the VecStim object is not threadsafe, if a single cell model was split into pieces to run in multiple threads, could VecStim objects still be used as sources for NetCon objects targeting point processes in the multisplit cell, or not? Artificial cells aren't even associated with sections, so maybe?

Also, I have another .mod file that uses VERBATIM blocks that compiles as thread safe. What makes the difference?

Code: Select all

NEURON {
	POINT_PROCESS Pr
	RANGE P, P0, random, f, tau_F, d1, tau_D1, d2, tau_D2, F, D1, D2, tlast
    THREADSAFE
    POINTER randObjPtr
}

PARAMETER {
    : the (1) is needed for the range limits to be effective
	P0 = 0.5 (1) < 0, 1 >               : basal release probability
    : these values are from Fig.3 in Varela et al. 1997
    f = 0.917 (1) < 0, 1e9 >            : additive facilitation per spike
    tau_F = 94 (ms) < 1e-9, 1e9 >       : rate of decay back to baseline following facilitation
    d1 = 0.416 (1) < 0, 1 >             : multiplicative fast depression per spike
    tau_D1 = 380 (ms) < 1e-9, 1e9 >     : rate of decay back to baseline following fast depression
    d2 = 0.975 (1) < 0, 1 >             : multiplicative slow depression per spike
    tau_D2 = 9200 (ms) < 1e-9, 1e9 >    : rate of decay back to baseline following slow depression
}

ASSIGNED {
	P				        : instantaneous release probability
    randObjPtr              : pointer to a hoc random number generator Random.uniform(0,1)
    random                  : individual instance of random number
    F                       : current level of facilitation
    D1                      : current level of fast depression
    D2                      : current level of slow depression
    tlast (ms)              : time of last spike
}

INITIAL {
	P = P0
    random = 1
    F = 1
    D1 = 1
    D2 = 1
}

NET_RECEIVE(weight) {
    INITIAL {
        tlast = t
    }
    F = 1 + (F-1)*exp(-(t - tlast)/tau_F)
    D1 = 1 - (1-D1)*exp(-(t - tlast)/tau_D1)
:    D2 = 1 - (1-D2)*exp(-(t - tlast)/tau_D2)
:    if (P0*F*D1*D2 > 1) {
    if (P0*F*D1 > 1) {
        P = 1
    } else {
:        P = P0*F*D1*D2
       P = P0*F*D1
    }
    random = randGen()
    if (random <= P) {
        net_event(t)
    }
    tlast = t
    F = F + f
    D1 = D1 * d1
:    D2 = D2 * d2
}

VERBATIM
double nrn_random_pick(void* r);
void* nrn_random_arg(int argpos);
ENDVERBATIM

FUNCTION randGen() {
VERBATIM
   if (_p_randObjPtr) {
      /*
      :Supports separate independent but reproducible streams for
      : each instance. However, the corresponding hoc Random
      : distribution MUST be set to Random.uniform(0,1)
      */
      _lrandGen = nrn_random_pick(_p_randObjPtr);
   }else{
      hoc_execerror("Random object ref not set correctly for randObjPtr"," only via hoc Random");
   }
ENDVERBATIM
}

PROCEDURE setRandObjRef() {
VERBATIM
   void** pv4 = (void**)(&_p_randObjPtr);
   if (ifarg(1)) {
      *pv4 = nrn_random_arg(1);
   }else{
      *pv4 = (void*)0;
   }
ENDVERBATIM
}

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: How to make VecStim (vecevent.mod) thread safe?

Post by hines » Mon Feb 16, 2015 1:54 am

Adding the keyword, THREADSAFE, to the NEURON block in a mod file, is a claim by the author that the model is, in fact, thead safe. The only automatic help that the nocmodl translator
will provide in making this statement true is to promote all GLOBAL variables that are on the left hand side of assignment statements, into threads specific instance variables. I.e. instead
of thread shareing one memory location, there will be pc.nthread() memory locations devoted to the variable so a thread reading and writing to the variable will not cause a race condition.
The translator cannot determine if POINTER variables or the code in VERBATIM blocks is thread safe or not and so, conservatively assumes it is NOT thread safe and refuses to allow
an instance of the model into any thread other than thread 0. Upon a review of the code as well as the intended use of any POINTER variable, the author may determine that the
model is, in fact thread safe and so force that interpretation by adding the THREADSAFE keyword.

It is an accident that the THREADSAFE keyword does not turn off the "Notice: Use of POINTER is not thread safe." unless it appear before the POINTER declaration. So merely
put the THREADSAFE at the beginning of the NEURON block.

It turns out that vecevent.mod is, in fact, threadsafe if the POINTER is only used to reference a Vector object from Hoc. So THREADSAFE should be added to the beginning of the NEURON
block in vecevent.mod.

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Mon Feb 16, 2015 12:02 pm

hines wrote: It turns out that vecevent.mod is, in fact, threadsafe if the POINTER is only used to reference a Vector object from Hoc. So THREADSAFE should be added to the beginning of the NEURON
block in vecevent.mod.
This actually results in a series of errors:

Code: Select all

vecevent.c:282:23: error: parameter named '_p' is missing
        Prop *_prop; double* _p; Datum* _ppvar; Datum* _thread;
                             ^
vecevent.c:282:34: error: parameter named '_ppvar' is missing
        Prop *_prop; double* _p; Datum* _ppvar; Datum* _thread;
                                        ^
vecevent.c:282:49: error: parameter named '_thread' is missing
        Prop *_prop; double* _p; Datum* _ppvar; Datum* _thread;
                                                       ^
vecevent.c:281:8: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
static _destructor(_prop)
~~~~~~ ^
vecevent.c:281:8: error: conflicting types for '_destructor'
vecevent.c:129:14: note: previous declaration is here
 static void _destructor(Prop*);
             ^
vecevent.c:284:2: error: use of undeclared identifier '_thread'
        _thread = (Datum*)0;
        ^
vecevent.c:285:2: error: use of undeclared identifier '_p'
        _p = prop->param; _ppvar = _prop->dparam;
        ^
vecevent.c:285:7: error: use of undeclared identifier 'prop'
        _p = prop->param; _ppvar = _prop->dparam;
             ^
vecevent.c:285:20: error: use of undeclared identifier '_ppvar'
        _p = prop->param; _ppvar = _prop->dparam;
                          ^
vecevent.c:290:21: error: use of undeclared identifier '_ppvar'
        void* vv = (void*)(_p_ptr);  
                           ^
vecevent.c:44:16: note: expanded from macro '_p_ptr'
#define _p_ptr  _ppvar[2]._pval
                ^
1 warning and 9 errors generated.

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: How to make VecStim (vecevent.mod) thread safe?

Post by hines » Mon Feb 16, 2015 1:00 pm

is your version of the mod file the same as at
http://www.neuron.yale.edu/hg/neuron/nr ... cevent.mod
If so, what is your version of NEURON?

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Mon Feb 16, 2015 1:36 pm

hines wrote:is your version of the mod file the same as at
http://www.neuron.yale.edu/hg/neuron/nr ... cevent.mod
If so, what is your version of NEURON?
Yes.
NEURON -- VERSION 7.3 ansi (975:433096832316) 433096832316

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: How to make VecStim (vecevent.mod) thread safe?

Post by hines » Mon Feb 16, 2015 2:08 pm

The THREADSAFE keyword did not cause compiler problems when I checked it today with the latest mercurial trunk version. I'm guessing the
problem will disappear if you use that or one of the 7.4 alpha distribution files at
http://www.neuron.yale.edu/ftp/neuron/versions/alpha/
Let me know if that is not the case.

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Tue Feb 17, 2015 11:29 am

hines wrote:The THREADSAFE keyword did not cause compiler problems when I checked it today with the latest mercurial trunk version. I'm guessing the
problem will disappear if you use that or one of the 7.4 alpha distribution files at
http://www.neuron.yale.edu/ftp/neuron/versions/alpha/
Let me know if that is not the case.
Hmm. Is it true that if I want to use anaconda python or any python other than cython that I am stuck building from the ansi branch? Doesn't look like that has been updated recently.

I followed your advice and built from the latest trunk. It generated quite a few warnings and had a long pause during 'make install', but it completed and appears to be functioning properly. I still get the same compile errors when adding 'THREADSAFE' to the NEURON block in vecevent.mod

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: How to make VecStim (vecevent.mod) thread safe?

Post by hines » Tue Feb 17, 2015 11:50 am

I assume you want to launch anaconda's version of python and import neuron. What machine are we talking about?
If mswin then use:
http://www.neuron.yale.edu/ftp/neuron/v ... -setup.exe
if mac os x then use:
http://www.neuron.yale.edu/ftp/neuron/v ... 10.7.5.dmg
if linux then people often build from sources, but
http://www.neuron.yale.edu/ftp/neuron/v ... x86_64.deb
is supposed to work. In any case, for launching python, it should be sufficient to
export PYTHONPATH=/where/you/installed/neuron/lib/python # but check to see if lib is where the python folder exists or if it is lib64

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Tue Feb 17, 2015 12:27 pm

hines wrote:I assume you want to launch anaconda's version of python and import neuron. What machine are we talking about?
Mac. Has that image been built --with-paranrn to work with mpi?

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: How to make VecStim (vecevent.mod) thread safe?

Post by hines » Tue Feb 17, 2015 2:38 pm

Yes. All the distributions are built using --with-paranrn=dynamic --with-nrnpython=dynamic .
On the mac, you can use mpi if you install openmpi .

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: How to make VecStim (vecevent.mod) thread safe?

Post by aaronmil » Tue Feb 17, 2015 2:48 pm

hines wrote: if mac os x then use:
http://www.neuron.yale.edu/ftp/neuron/v ... 10.7.5.dmg
Thanks for all your help, Michael. I installed from this image, but still get the same compile errors when attempting to make vecevent.mod thread safe.

Going back to the original question, does it matter? If vecevent is an artifical cell used to drive a netcon, won't it be agnostic to whether or not the target of the netcon has been split into more than one thread?

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: How to make VecStim (vecevent.mod) thread safe?

Post by hines » Thu Feb 19, 2015 7:47 am

My apologies. There was a bug in the nmodl translator which produced uncompilable code for the combination of DESTRUCTOR and THREADSAFE.
http://www.neuron.yale.edu/hg/neuron/nr ... 534c11dd4b
A new distribution for the mac is at
http://www.neuron.yale.edu/ftp/neuron/v ... 10.7.5.dmg
I must earlier had only looked at the elimination of the translator warning when THREADSAFE was prepended to the NEURON block.
Going back to the original question, does it matter? If vecevent is an artifical cell used to drive a netcon, won't it be agnostic to whether or not the target of the netcon has been split into more than one thread?
It likely does not matter in practical terms. The number of VecStims would have to be huge to make a performance difference as to whether they were all handled by thread 0 or were carefully placed
on the threads that contained the cells that received the input from VecStim. The vecevent is agnostic to whether it is the source of many netcons that have targets on other threads or other mpi
processes. It would be an experimental performance queston whether one would ever see a performance improvement in the case where many VecEvents each send spikes to many targets on
different threads.

Post Reply