Error msg: "Event arrived out of order"

[sec6]

I am trying to do something complicated with the event delivery system, and clearly not doing it right: I'd like to know what I'm doing wrong.

The full error message is:

Code: Select all

/Applications/NEURON-7.0/nrn/umac/bin/nrniv: GABAa_S[599] :Event arrived out of order. Must call ParallelContext.set_maxstep AFTER assigning minimum NetCon.delay

Note that this is not a paralellized model and I'm not using a parallel network manager, or anything like that. Although I'm using NEURON version 7, I'm pretty sure the execution's not multithreaded because some of the NMODL mechanisms are written using very old idioms which prevent that (I've verified that the model uses only one core on my two-core machine).

GABAa_S is a point process in the model, which has nothing directly to do with the problem. The problem's being caused by my use of CVode.event() and FInitializeHandler().

Because the model generates a lot of output, I have to do buffered output to disk. I use Vector.record() to capture spike times, then use CVode.event() and FInitializeHandler() (per suggestion of M. Hines in this forum) to pause execution at defined times, write the Vectors to disk, then Vector.size(0) and Vector.buffer_size(0) and resume execution.

That worked fine. The problem came when I tried to do buffered *input* also. Reading and writing are supposed to be asynchronous, that is, the times at which the output buffer is flushed to disk are different from the times at which a buffer full of input is read off disk.

Let h be the top-level Hoc interpreter
Let dt_o be the interval between buffered writes.
Let dt_i[k] be the kth interval between buffered reads

Note that the intervals between buffered writes are always the same, but the intervals between buffered read may vary. I don't *think* this is central to my problem, though.
A buffered read occurs at t=0, so dt_i[1] is the interval between the zeroth & first buffered read.
No buffered write occurs at t=0, so the first buffered write occurs at dt_o, the second at dt_o * 2, the third at dt_o * 3, and so on.
(dt_o * 3) < dt_i[1] < (dt_o * 4)

I create two objects of type "timer" (call them timer_o and timer_i).
Objects of type timer have an attribute (call it "callback") which is callable.
the timer.__init__ method (which Python calls when a timer is instantiated) creates a type-1 FInitializeHandler associated with that instance (timer_o.fih or timer_i.fih). The callback for timer_o.fih is timer_o.callback; the callback for timer_i.fih is timer_i.callback.
Finally cvode.event() is called, specifying an interval of h.t+dt_o (for timer_o) or h.t_dt_i[k] (for timer_i), and specifying, as the callback, timer_o.callback
(for timer_o) or timer_i.callback (for timer_i).

Here's what happens when I run the model:

Code: Select all

At t=0
	callback_o occurs (correct)
	callback_o occurs *again* (incorrect but harmless)
	buffered read occurs (correct)
	no buffered write occurs (correct)
	cvode.event() call occurrs without error[/list]
at t = dt_0 * 1
	callback_o occurs (correct)
	buffered write occurs (correct)
	no buffered read occurs (correct)
	cvode.event() call occurrs without error[/list]
at t = dt_0 * 2
	callback_o occurs (correct)
	buffered write occurs (correct)
	no buffered read occurs (correct)
	cvode.event() call occurrs without error[/list]
at t = dt_0 * 3
	callback_o occurs (correct)
	buffered write occurs (correct)
	no buffered read occurs (correct)
	cvode.event() call occurrs without error[/list]
at t = dt_i * 1
	callback_i occurs (correct)
	buffered read occurs (correct)
	no buffered write occurs (correct)
	cvode.event() call occurrs without error[/list]

Some time after that, the following error occurs:

Code: Select all

/Applications/NEURON-7.0/nrn/umac/bin/nrniv: GABAa_S[599] :Event arrived out of order. Must call ParallelContext.set_maxstep AFTER assigning minimum NetCon.delay
 near line 0
 ^
        fadvance()
      advance()
    step()
  continuerun(800000)
Traceback (most recent call last):
  File "./model.py", line 165, in <module>
    ses.runModel(runDur,dt=dt,recordWhat=None,recordSpikes=True,bufferedSpikeOutputFilePath=bufferedSpikeOutputFilePath,recordSpikeInterval=10000)
  File "nrnpy.py", line 795, in runModel
    self.h.continuerun(tstop)
RuntimeError: hoc error
>>> 

The actual program is rather complex, and, as yet, I have *not* written a "toy" program to reproduce this effect. I'm posting this in case either

1) I'm guilty of some obvious misunderstanding regarding cvode.event() and/or FInitializeHandler, in which case maybe you can put me straight without needing the toy program.

or

2) Maybe (I don't know) it should be impossible for unthreaded NEURON/Python code to cause an "Event arrived out of order" error, in which case the fact that this did, in fact, occur would be of some general interest.

If neither of those is true, I might just evade the whole issue & take a different approach to implementing the functionality I need: that might get me to my goal quicker than writing the toy program, although I suppose I'd learn less.

[hines]

diagnosing this requires tracing paths through c/c++ code. Please send me all the files needed to exhibit the error
and I'll look into it. Send the zip file to michael.hines@yale.edu.