Page 1 of 1

VecStim play leading to Segmentation violation

Posted: Fri Mar 01, 2013 11:39 am
by Malbolge
I'm currently having some issues using the play method to provide spike times to drive Exp2Syns as input to my model(s). I get the following error message (under certain circumstances)...

Code: Select all

2 nrniv: Segmentation violation
2  in RChiBuild3.hoc near line 224
2  pc.runworker()
               ^
        2 fadvance()
      2 advance()
    2 step()
  2 continuerun(4200)
and others
I have a BulletinBoard system setup for running large numbers of single cell simulations (each with slightly different parameters, input spike trains, positioning of synapses etc) using ParallelContext and Submit. Each cell has multiple synapses, with a VecStim attached, using NetCons, and then each VecStim is given a vector of times for the simulation to be run. If I submit a number of jobs/simulations less than or equal to the number of processors available then everything works as it should. The problem comes when I submit more jobs to the BBoard than I have available processors... when a processor finishes its first job, and moves on to the second, the Segmentation violation occurs and the program stops. Similarly, when I run the system/code in a serial form, a single simulation works fine, any more and the Segmentation violation occurs as the 2nd run should be starting.

I've been able to isolate the issue to the use of 'MyVecStim.play(MyTimes)'. If I comment that statement out, the code works fine for as many jobs as I request, even if it exceeds the number of available processors. The code is as follows (relevant part is the func DoRun(). It omits further code for setting up the synapses randomly along given sections, and the ParallelContext setup):

Code: Select all

{load_file("nrngui.hoc")}
//Cell Type
{load_file("mvnB_TEMP.hoc")}
//Global Values. Still some used. 
{load_file("globals.hoc")}

objref cell, stim
objref tms

cell = new mvnB()
access cell.somaB
objref rsVec, vek
objref pc
objref initial_Voltage_Stream
objref rate_Stream
objref inp_Stream
objref inp_trains
objref inps, tlist, vect

//Recording Vectors, for results/output. 
objref recV, recI, recG
recV = new Vector()
recI = new Vector()
recG = new Vector()
 
sim_time = 4200
DT = 0.1
dt = DT
steps_per_ms = 1/DT
tstart = TSTART
tstop = sim_time
pop_size = 13
seed = 337
V_INIT = -65
syn_tau1 = 0.5      //Rise time
syn_tau2 = 3        //Fall time 
nsyns = 20          //Number of Synapses 
num_trains = 5000    // Total number of trains given in the .dat file. The total size of the input pool. 
syn_Thresh = 10     //Exp2Syn Threshold
syn_Delay = 1       //Delay in ms
syn_Weight = 1.0    //Weight

objref vs[nsyns], nc[nsyns]
objref syn[nsyns], randomize_location

//Analysis start after X time.
analstart = 1200
{load_file("analysis.hoc")}
pc = new ParallelContext()
st1 = pc.time

//Read in .dat with input SpikeTimes, make vectors of them.
  inps = new File()
  inps.ropen("test_8Hz_A03.dat")
  tlist = new List(num_trains)  
  for kk = 0, num_trains-1 {
    vect = new Vector()
    vect.scantil(inps, 9999)
    tlist.append(vect)
  }

func doRun() {
  
  access cell.somaB
  // recording vectors
	recV = new Vector()
	recV.record(&v(0.5), dt)
  id = hoc_ac_
  
  //Initial voltage, and rate for this particular simulation. And initialise Random Streams used WITHIN the DoRun block.
  v_init = $2 
  F_rate = $3
  syn_rand_seed = $4
  train_seed = $5
  
  inp_trains = new List(nsyns)
  
  //Setup Random stream for choosing Input Trains
  inp_Stream = new Random (train_seed)
  inp_Stream.discunif(0, num_trains-1)
  
  //Create the Synapses
  creatensyn(cell, cell.alldendritesB, nsyns, syn_rand_seed)

  //Set up Inputs
  //Make the VecStims for driving the Synapses
  for iv = 0, nsyns - 1{
    vs[iv] = new VecStim()
  }

  //Attach VecStims to Synapses using NetCons
  for in = 0, nsyns-1{
    nc[in] = new NetCon(vs[in], syn[in], syn_Thresh, syn_Delay, syn_Weight)
  }
  
  //Choose input trains from pool
  for k = 0, nsyns-1{
    r_n = inp_Stream.repick()
    inp_trains.append(tlist.o[k])
    }
  //Play these VecStims
 for k = 0, nsyns-1{
   tms = new Vector()
   tms = inp_trains.o[k]
   vs[k].play(tms)
  }

  finitialize(v_init)
  run()
  analspikes(analstart)
  pc.post(id, spikes)
  return 0
}
I have had a look around the forum for similar problems or discussion of the issue. I have a feeling that it might be along the lines discussed in this topic ( http://www.neuron.yale.edu/phpbb/viewto ... f=2&t=2147 ), however, I'm a little unsure as to the solution for adding the Vectors to a list as they are created. Also, I tried adding the changes M Hines mentions to the vecevent.mod, but it refused to compile with those changes, giving the error...

vecevent.c: In function 'play':
vecevent.c:238: error: invalid type of argument of 'unary *' (have 'int')
(and the same for the _destructor method)
make: *** [vecevent.lo] Error 1

Are there any alternatives to VecStim play method that I could/should be using here? Or is there an easy fix for the issue? I've come across Ted's suggestion for using FInitializeHandler, but each synapse has around 100-200 events during the simulation, and each cell can have up to 60 synapses attached, so I figure that wouldn't be the best way to go about it.

Currently using Neuron 7.2, with openmpi 1.6, using SGE (I think).

Re: VecStim play leading to Segmentation violation

Posted: Fri Mar 01, 2013 12:20 pm
by ted
I've come across Ted's suggestion for using FInitializeHandler . . . I figure that wouldn't be the best way to go about it.
You made the right decision.

Given that your code runs well serially or in parallel except under very specific conditions, my first suspicion is that you may have run into an obscure bug in NEURON 7.2. There have been many performance improvements and bug fixes since 7.2 was released, so I'd suggest updating to the most recent development code, or at least the most recent alpha installer.

Re: VecStim play leading to Segmentation violation

Posted: Wed Mar 06, 2013 10:59 am
by Malbolge
Managed to get the most recent 7.3.alpha installed on the system we're making use of, and ran a very brief test, and it seems to have worked correctly this time (20 jobs submitted on 12 processors, 20 sets of results returned and no segmentation violation). I'm running a more extensive test currently, and will post if there are any further issues.

Just a little further information, in case it's useful at all... I initially tried stepping back to 7.1, as the code was originally written/tested and worked without error under 7.1, on a different (Condor) Cluster here at our University. It didn't work on the new system (ARCHIE WeSt) under 7.1, and gave the same seg violation error when the number of tasks submitted exceeded the number of usable processors, at the same time as it had under 7.2.

Re: VecStim play leading to Segmentation violation

Posted: Mon Dec 08, 2014 11:25 am
by laura
I am running into this same error using NEURON 7.3. I am running single cell simulations in parallel as well. The errors are exactly the same - if the simulation is submitted with x number of processors, then x number of cells will run until it gets to the x+1 cell when the Segmentation violation occurs and the program stops. The program completes if the number of cells is less than the number of processors. Similarly to the previous post, if I comment out the VecStim.play, the code works. An additional problem that I noticed is that the error is frequency dependent. It works for some stimulation frequencies but not all - and there seems to be no pattern. Is this still a bug in 7.3 or are there any other ideas what might be causing this error?

Re: VecStim play leading to Segmentation violation

Posted: Tue Dec 09, 2014 11:52 am
by ted
Exactly what version of NEURON are you using? That is, what is the entire first line that NEURON prints to the terminal when it starts?

Re: VecStim play leading to Segmentation violation

Posted: Tue Dec 09, 2014 12:08 pm
by laura
This is the first line:
NEURON -- VERSION 7.3 ansi (1078:2b0c984183df) 2014-04-04

Re: VecStim play leading to Segmentation violation

Posted: Tue Dec 09, 2014 12:35 pm
by ted
Suggest you uninstall that,
then either
get one of the latest alpha installers from
http://www.neuron.yale.edu/ftp/neuron/versions/alpha
or
get the latest development code from the mercurial repository (see
http://www.neuron.yale.edu/neuron/download/getdevel
) and install that.