Running multiple simulations in one process

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

lcampagn
Posts: 22
Joined: Mon Jul 07, 2014 1:12 pm

Running multiple simulations in one process

Post by lcampagn » Tue Oct 28, 2014 2:17 pm

I am looking for a comprehensive set of guidelines on how to correctly run two different simulations (serially) within the same process (or alternatively, whether this is even supported behavior in the first place). Ideally, the results of the second simulation should be in no way affected by any details of the first simulation. What I have found so far:

1. Python must collect all objects from the first simulation before starting the second. This is problematic because reference management is difficult in Python, and when it goes wrong it is very difficult to diagnose. A solution for this is to wrap every NEURON object in a Python object that keeps a private reference to its underlying object, thus ensuring that references are very unlikely to be leaked (this is one of the features I am building into the API wrapper discussed here: viewtopic.php?f=2&t=3212).

2. Ensure that global variables like t, dt, celcius, etc. are explicitly set for each run. This is problematic because of a note in the documentation:
Warning: Not very useful. No way to completely restart neuron exect to quit() and re-load.
(http://www.neuron.yale.edu/neuron/stati ... ml#initnrn)
Is anything missing from this list? Is there anything to be done about (2) ?

ted
Site Admin
Posts: 5652
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Running multiple simulations in one process

Post by ted » Tue Oct 28, 2014 4:26 pm

lcampagn wrote:I am looking for a comprehensive set of guidelines on how to correctly run two different simulations (serially) within the same process (or alternatively, whether this is even supported behavior in the first place).
This is a big deal when using NEURON from Python? From hoc, given properly written code, there's no problem doing this at all. Merely separate model specification code from simulation execution code, and that's it. Period. Execute the model setup code just once. After that has been done, if you need to run simulation after simulation, simply

Code: Select all

REPEAT
  tinker with this or that parameter
  call the standard run system's run()
  do whatever you like with simulation results
UNTIL you're done
If you know what you're doing,
"tinker with this or that parameter"
can even involve adding or deleting new cells, connections, or other objects.

So what's the big deal if the model specification was written in Python? After h.run() completes, does Python break the stuff that was created during the model setup phase?

ramcdougal
Posts: 163
Joined: Fri Nov 28, 2008 3:38 pm
Location: Yale School of Public Health

Re: Running multiple simulations in one process

Post by ramcdougal » Tue Oct 28, 2014 4:58 pm

Is the issue that you have no control over the code that is being run?

i.e Are you trying to write reusable code (if so, follow Ted's advice)? Or are you trying to run arbitrary simulations whose code you do not control?

If it is the latter, are you sure you need it to only be in one process? Why? Is this part of context-switching for pynrn?

My usual approach is to have a controlling NEURON process that does an os.fork(), immediately joins the child, and then lets the child run the simulation. This way, you can set up special logging, etc in the parent, and the child gets it for free without modifying the simulation's code. When the child process finishes, the controlling process automatically resumes and can then os.fork() again for the next simulation. Since the previous simulation was run in a forked process, the new simulation starts with a "pristine" version of NEURON.

lcampagn
Posts: 22
Joined: Mon Jul 07, 2014 1:12 pm

Re: Running multiple simulations in one process

Post by lcampagn » Tue Oct 28, 2014 5:28 pm

This is a big deal when using NEURON from Python?
It has been a big deal for me, but perhaps I have some basic misunderstanding of how the API works. The problem is that there is no way to explicitly remove objects from the NEURON kernel; instead this is done indirectly when Python collects the objects. If I have leaked a reference to any of those objects, Python will not collect them and the underlying NEURON object will leak into subsequent simulations. As I mentioned above, reference management is hard and debugging leaked references is tedious. I've also mentioned elsewhere that, due to the lack of functions for introspection, I also am not able to check for leaked objects before starting a simulation. The wrapper I am working on should resolve the issue by making leaked references unlikely, but to complete that wrapper I need the "comprehensive set of guidelines" for resetting NEURON.

lcampagn
Posts: 22
Joined: Mon Jul 07, 2014 1:12 pm

Re: Running multiple simulations in one process

Post by lcampagn » Tue Oct 28, 2014 6:13 pm

Is the issue that you have no control over the code that is being run?
We are writing a general-purpose model that we hope will be useful to a larger community. So in that sense we have complete control over the code within the model, but we do no control the ways it will be used (except by writing documentation and imposing API limitations).
are you sure you need it to only be in one process?
No, it is just more convenient to have the option of running a single process. My choices are (seemingly) either to wrap all NEURON objects or to move to a multi-process system, and my initial assessment of the cost / benefits suggests that the wrapper is a better plan.
My usual approach is to have a controlling NEURON process that does an os.fork()
Using fork() would be awesome except that it is not portable, and that "pristine" NEURON state is fragile--the users of our model would have to understand the limitations of fork(), which is an invitation for error. There are plenty of other multiprocessing options that would work (although having the single-process option, if possible, would be ideal).

ramcdougal
Posts: 163
Joined: Fri Nov 28, 2008 3:38 pm
Location: Yale School of Public Health

Re: Running multiple simulations in one process

Post by ramcdougal » Tue Oct 28, 2014 6:27 pm

If you wanted to be super-safe, you could only ever have one copy of a NEURON object (e.g. of a section) that was held by Python and have everything else work with a weakref proxy object. For good measure, you could have the gc module force a garbage collection before any NEURON initialization or advances.

That said, NEURON offers a decent amount of introspection capabilities: if your interface forces a Python-only world (no custom HOC), I'd think you'd be reset if you reset globals, removed all sections (check h.allsec), all point processes (get a list of possible types with h.MechanismType and of instances with List), all FInitializeHandlers (h.List), and all LinearMechanism instances (h.List) -- actually, use List on anything that could be a class and remove any instances.

lcampagn
Posts: 22
Joined: Mon Jul 07, 2014 1:12 pm

Re: Running multiple simulations in one process

Post by lcampagn » Wed Oct 29, 2014 8:50 pm

. . . have everything else work with a weakref proxy object.
Unfortunately, weakrefs are not allowed to most (all?) neuron objects.
I'd think you'd be reset if you reset globals, removed all sections (check h.allsec), all point processes (get a list of possible types with h.MechanismType and of instances with List), all FInitializeHandlers (h.List), and all LinearMechanism instances (h.List) -- actually, use List on anything that could be a class and remove any instances.
This is immensely helpful, thank you! I had come across List before, but didn't realize the significance of List(template). I have a couple of follow-up questions:

- The global variables I have identified from the documentation are t, dt, celcius, clamp_resist, secondorder, and stoprun. Anything else I should be aware of? A quick search turned up about 40 writable numerical variables in the "h" namespace, but I don't know if any of these are relevant.. I suppose it doesn't hurt to just reset everything?
- Is there any way to call MechanismType.selection(string) from Python, or should I call into HOC for this?

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: Running multiple simulations in one process

Post by hines » Fri Oct 31, 2014 4:23 pm

It is likely I don't comprehend all requirements of your project. However, if one of the elements is to create and destroy models, take a look at how it is done
by http://www.neuron.yale.edu/hg/z/models/ ... 3d/ring.py
Ie the model is a class which, because it involves gids which are instance independent, mean it can only be instantiated as a singleton object. But an instance
can be repeatedly created and destroyed with the number cells as a parameter. It is used in the context of subworlds which combine the bulletin board style of parallelism with mpi parallel computation of a ring instance. Each subworld has its own ring instance. My point is that if you define a model as a class with
constructor and destructor, you can manage quite well. I would suggest a general tear down order is gid, netcon, cells. As long there is no reference to
section, Pointprocess, netcon, etc, outside of the model object, the model will disappear completely when the last reference to the model is released.
There is no need to consider MechanismType for density mechanisms as all that is destroyed when the sections are destroyed (when a cell is destroyed).

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: Running multiple simulations in one process

Post by hines » Mon Nov 03, 2014 6:43 am

- Is there any way to call MechanismType.selection(string) from Python, or should I call into HOC for this?
Consider:

Code: Select all

from neuron import h
mt = h.MechanismType(0)
sref = h.ref('')
for i in range(int(mt.count())):
  mt.select(i)
  mt.selected(sref)
  print sref[0]
And see in http://neuron.yale.edu/neuron/static/ne ... hlight=ref the paragraph that begins with:
Many hoc functions use call by reference and return information by changing the value of an argument. These are called from the python world by passing a HocObject.ref() object. Here is an example that changes a string.

lcampagn
Posts: 22
Joined: Mon Jul 07, 2014 1:12 pm

Re: Running multiple simulations in one process

Post by lcampagn » Thu Nov 06, 2014 3:49 pm

Thanks again, Michael!

I'd like to return to this issue of object deletion, because there seems to be some doubt about my concern that reference management in python is hard. Let's take the following example:

Code: Select all

from neuron import h

sections = {}

def make_sec(name):
    sec = h.Section(name=name)
    if name in sections:
        raise Exception("Section %s already exists." % name)
    sections[name] = sec

for name in ['soma', 'dend1', 'dend5', 'dend3.1', 'dend3.2', 'dend3.3', 
             'dend5.1', 'dend5.1', 'dend5.2']:
    make_sec(name)

sections.clear()  # no more Section references, right?

assert len(list(h.allsec())) == 0
Of course, the assertion at the end of the script fails, and the reason is not at all obvious unless you happen to have spent a lot of time debugging such issues. Most python programmers never worry at all about leaked references because they do not cause trouble unless you are in danger of running out of memory. This peculiar behavior of NEURON, however, means that there are many real situations where I can generate unreproduceable simulation results from leaked references. This is just one example of many I have run into over the years; I could provide more but I hope I have already convinced you that reference management is not a good way to interact with NEURON. The wrapper I am working on provides a workaround for this, but I wonder if it would be possible for you to add `destroy` methods to all of the relevant python wrappers?

ramcdougal
Posts: 163
Joined: Fri Nov 28, 2008 3:38 pm
Location: Yale School of Public Health

Re: Running multiple simulations in one process

Post by ramcdougal » Thu Nov 06, 2014 5:46 pm

The behavior you describe is a property of Python not of NEURON. Here's a NEURON-free example:

Code: Select all

class A:
    def __del__(self):
        print 'obj deleted'

def test():
    a = A()
    raise Exception()

test()
    
print 'done test'
If you run this interactively, it'll print out the error message and then print "done test"... but it won't print the "obj deleted" message from the destructor.

Why not? The simple answer is that the destructor hasn't been invoked. You could try forcing a garbage collection, but that won't help. That's because there is still a frame somewhere that maintains a reference to the section. Who is it? Let's use your example and ask Python to tell us:

Code: Select all

from neuron import h

sections = {}

def make_sec(name):
    sec = h.Section(name=name)
    if name in sections:
        raise Exception("Section %s already exists." % name)
    sections[name] = sec

for name in ['soma', 'dend1', 'dend5', 'dend3.1', 'dend3.2', 'dend3.3', 
             'dend5.1', 'dend5.1', 'dend5.2']:
    make_sec(name)

sections.clear()  # no more Section references, right?

import gc
print gc.get_referrers(list(h.allsec())[0])[0].f_locals
Python reports that the section is still referred to by a frame with variables:

Code: Select all

{'sec': <nrn.Section object at 0x7fe1b3ae5630>, 'name': 'dend5.1'}
We realize that the stack frame that hosted the last exception is preserved.

All we have to do is eliminate that frame. Fortunately, Python provides a function for this: sys.exc_clear(). This won't work from an interactive session (of course all you have to do then is generate a new exception... I recommend typing ^C)

Here then is my revised version, that includes a safety purge of any stored exception frame. To make this runnable as a script, I've enclosed the exception causing part in a try-catch:

Code: Select all

from neuron import h

sections = {}

def make_sec(name):
    sec = h.Section(name=name)
    if name in sections:
        raise Exception("Section %s already exists." % name)
    sections[name] = sec

try:
    for name in ['soma', 'dend1', 'dend5', 'dend3.1', 'dend3.2', 'dend3.3', 
                 'dend5.1', 'dend5.1', 'dend5.2']:
        make_sec(name)
except:
    pass

sections.clear()  # no more Section references, right?
import sys
sys.exc_clear()

assert(len(list(h.allsec())) == 0)
print 'assert passed'
Without the sys.exc_clear() the assert would fail; with it, it passes.

lcampagn
Posts: 22
Joined: Mon Jul 07, 2014 1:12 pm

Re: Running multiple simulations in one process

Post by lcampagn » Thu Nov 06, 2014 8:22 pm

The behavior you describe is a property of Python not of NEURON.
That's exactly my point. You and I are aware of sys.last_traceback, but the vast majority of NEURON users are not (and even I would have to do a lot of digging to fix this issue). As I have said, there are many different ways references can be leaked and they usually take considerable time and skill to debug, especially for larger models. I am not really interested in fixing these issues one at a time; my intent here is to see how we can make NEURON more robust by eliminating its reliance on implicit reference management. I would like explicit control over which objects NEURON simulates, and which it does not.

hines
Site Admin
Posts: 1580
Joined: Wed May 18, 2005 3:32 pm

Re: Running multiple simulations in one process

Post by hines » Fri Nov 07, 2014 2:16 am

First, when I run your code fragment that has the duplicate name ( 'dend5.1', 'dend5.1') I get

Code: Select all

  File "sec.py", line 8, in make_sec
    raise Exception("Section %s already exists." % name)
Exception: Section dend5.1 already exists.
which, I gather, is what you intended with the fragment:

Code: Select all

    if name in sections:
        raise Exception("Section %s already exists." % name)
I'll assume this aspect just introduces some confusion and the duplicate name issue is irrelevant to your point, so I changed one the of the dend5.1 to dend5.1.1 .
Of course, the assertion at the end of the script fails
But on my machine it does not fail because the the answer to

Code: Select all

sections.clear()  # no more Section references, right?
was yes when I put the fragment in file and executed it or copy pasted to the terminal.

However, I am aware of the possibiltiy of deferred garbage collection and so added the fragment

Code: Select all

for s in h.allsec(): h.delete_section()
Unfortunately, although this would work for HOC sections that are instantiated via h('create ...') statements (and in fact is the ONLY way to destroy such sections), if one tries to do this with a Python created section, you get:

Code: Select all

nrniv: Cannot delete an unnamed section
So this raises the delicate question of whether I should change the behavion of h.delete_section() so that it removes Python created sections from the internal list of sections that are to be
simulated (and causes an error message to be generated if the section is used in any way). It also raises a little flock of questions about alternatives to deleting sections and merely
removing them from the list of sections to be simulated.

Anyway, so far I am unable to reproduce the problem we are discussing and it would be helpful to have a more robust example. Is it really the case that the problem is that a section is
not deleted when the reference count goes to 0, or is the problem that the program code or user or graphical interface holds a reference to the section that no one knows about and therefore
finds it difficult to destroy the section?

ramcdougal
Posts: 163
Joined: Fri Nov 28, 2008 3:38 pm
Location: Yale School of Public Health

Re: Running multiple simulations in one process

Post by ramcdougal » Fri Nov 07, 2014 7:01 am

The exception is critical to the point being discussed. When there is an exception, the local stack frame is not immediately released (and hence variables are not freed), but is preserved in case other code wishes to examine what went wrong.

To reproduce the error, copy and paste the code as is into a python terminal. It'll raise an exception, but that is expected and the next lines will still run (they would of code not run in a script).

Here's someone discussing this issue back in 2006: http://scott.yang.id.au/2006/11/python- ... frame.html

(Deferred garbage collection isn't a huge problem because the user can force garbage collection at any time with gc.collect(2).

aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Running multiple simulations in one process

Post by aaronmil » Thu Jan 01, 2015 2:32 pm

hines wrote: So this raises the delicate question of whether I should change the behavion of h.delete_section() so that it removes Python created sections from the internal list of sections that are to be simulated
I vote YES, please allow Python-created sections to be deleted via h.delete_section(sec=python_name_of_section) so that h.allsec() becomes a relevant and usable list for those of us using Python to build models. Even better would be if h.allsec() could take a cell=python_object parameter, so that it only listed sections that were assigned a cell using h.Section(name=python_str, cell=python_object)

Post Reply