Page 1 of 1

Parallel Neuron: NetCon failed to connect different hosts

Posted: Wed Jan 05, 2011 5:50 pm
by Nin
Hi everybody and happy new year to all of you!

I am trying to create a basic ring network to run in parallel as in Hines & Carnevalle 2007 (see in http://www.neuron.yale.edu/neuron/nrnpubs) together with the help of Thomas McTavish tutorial on https://nn.med.yale.edu:8000 to implement it in Python. To become familiar with the parallel environment of NEURON, I wanted to create a simple ring of cells, with as many neurons as parallel processes. For that I first created a Cell class, with a synapse on the soma, and methods to get the vectors of time and voltage and to create a NetCon.

Code: Select all

"""
 morphology.py
 this file contains a class to create a very basic cell type to work
 in parallel 
"""

from mpi4py import MPI # always load mpi4py before neuron
from neuron import h

import numpy as np

class DummyCell(object):
    """ A dummy cell simply for testing """
    def __init__(self):
        self.soma = h.Section(name ='soma', cell = self)
        self.soma.L = self.soma.diam = 30.
        self.soma.insert('hh')

        # create a synapse in soma 
        self.syn = h.ExpSyn(self.soma(0.5), name='syn', sec=self.soma)
        self.syn.tau = 2.
        
        # assign gid to current thread/rank
        # because we did not start ParallelContext, we have to use
        # the MPI for getting the rank
        self.gid = MPI.COMM_WORLD.Get_rank()
        
        # time and voltage vectors
        self._time = h.Vector()
        self._time.record(h._ref_t)
        self._voltage = h.Vector()
        self._voltage.record(self.soma(0.5)._ref_v)
    
        # a list of the NetCons of this cell
        self.netcon = []

    def connect2target(self, target):
        """ connects the firing of this cell to a target 
        via NetCon and appends the NetCon to the netcon list.
        """
        source = self.soma(0.5)._ref_v
        netcon = h.NetCon(source, target, sec = self.soma)
        netcon.threshold = 10.0
        netcon.delay = 3.1
        netcon.weight[0] = 0.04
        self.netcon.append(netcon)
        return netcon
    
    def get_vectorlist(self, time=True, voltage=True):
        """ return a list of NumPy vectors with time 
        and voltage at the soma """
        vectorlist = []
        
        if time is True:
            vectorlist.append(np.array(self._time))

        if voltage is True:
            vectorlist.append(np.array(self._voltage))
        
        return vectorlist

After that, I defined a Ring class that simply creates a cell with gid=rank of the parallel process and connect it to a next cell (if it exists) via a NetCon. If the cell has the gid=0, then a NetStim will stimulate the cell. After that ,cells are connected like 0-1, 1-2, and so on, until the last one is connected to the cell 0.

Code: Select all

"""
ring.py

A class defining a network of N cells, where N is the number of
process currently called in a parallel environment. Cell N will be
connected to cell with gid =0.
"""

# this will initialize MPI
from morphology import DummyCell as Cell

from neuron import h

class Ring(object):
    def __init__(self):
        """ creates a ring containing as many cells as 
        parallel processes """
        # list of cell objects 
        self.cell = Cell() # create one cell in the current thread

        # lists of netcons on this host
        self.netcon = []

        # external stimulator to this cell only if gid=0
        self.spk_generator = None
        self.nc_generator = None

        # external stimulator
        if self.cell.gid == 0:
            self.spk_generator = h.NetStim()
            self.spk_generator.number = 1
            self.spk_generator.start = 20.
            
            self.nc_generator = h.NetCon(self.spk_generator, self.cell.syn)
            self.nc_generator.delay = 1.
            self.nc_generator.weight[0] = 0.04
            self.nc_generator.threshold = 10.

        self.pc = h.ParallelContext()
        # associate the gid of the cell with host's thread
        self.pc.set_gid2node(self.cell.gid, int(self.pc.id()))

        # attach NetCon source (spike detector) to cell.gid
        nc = self.cell.connect2target(None)
        self.pc.cell(self.cell.gid, nc)
       

        # ** Connection **
        ncells = int(self.pc.nhost())
        targid = (self.cell.gid+1)%ncells

        if self.pc.gid_exists(targid):
            # get the object associated with that gid ???
            target = self.pc.gid2cell(targid)
            # and connect this gid to its synapse
            netcon = self.pc.gid_connect(self.cell.gid, target.syn)
            netcon.weight[0] = 0.04
            netcon.delay = 1.
            self.netcon.append(netcon)

    def get_cell(self, cell_gid=0):
        """ returns the object associated with the gid """
        if self.pc.gid_exists(cell_gid):
            return self.pc.gid2cell(cell_gid)
        else:
            return None

Unfortunately, when I tried to run the simulation in parallel (with, say 5 processes), and observe the voltage vectors obtained in cell 0, I can only see the one spike coming from NetStim. Apparently, the NetCons are not connecting the cells in the different hosts.

Code: Select all

""" test_simulate.py
execute with mpiexec -np 5 python test_simulate.py 
""""
from ring import Ring

h.load_file('stdrun.hoc')
pc = h.ParallelContext()
myring = Ring()

# ** Simulation 
pc.set_maxstep(10)
h.stdinit()
h.dt = 0.025
pc.psolve(120)

import numpy as np
# read voltage
cell = myring.get_cell(0)
if cell is not None:
    t,v = cell.get_vectorlist()
    import matplotlib.pyplot as plt
    plt.plot(t,v)
    plt.show()
    
pc.runworker()
pc.done()
    
I have been working on the parallel version of that program, and I a see a regular series of spikes. Does anybody knows how to track the absence/presence of the NetCon objects within the hosts? . Help here would be greatly appreciated!

Thanks in advance!

Re: Parallel Neuron: NetCon failed to connect different hosts

Posted: Thu Jan 06, 2011 2:49 pm
by hines

Code: Select all

self.pc.set_gid2node(self.cell.gid, int(self.pc.id()))
and

Code: Select all

        targid = (self.cell.gid+1)%ncells

        if self.pc.gid_exists(targid):

so targid does not exist on this rank and the body of the if statement is skipped.

You can get a list of all NetCon objects on a rank with
h.List('NetCon')

Re: Parallel Neuron: NetCon failed to connect different hosts

Posted: Thu Jan 06, 2011 6:53 pm
by Nin
Thanks a lot Hines!

I was erroneously taking the gid source of the NetCon from the current thread, and this is supposed to be taken from other gid/cell/thread. The toy model works now and to show my gratitude I wrote a mini-guide and uploaded as a Sage notebook with the model and an explanation about parallel NEURON. This might be useful for people starting with Parallelization with NEURON. You can find it here:
https://nn.med.yale.edu:8000/home/pub/14

thanks a lot!!!

Re: Parallel Neuron: NetCon failed to connect different hosts

Posted: Fri Jan 07, 2011 3:28 pm
by mctavish
Nice use of Sage, Jose!

A couple of notes: The Sage worksheet is a bit problematic since the permissions do not allow you to write to the "data" folder from the mpiexec call. Even though the simulation.py file is in the data folder and you can typically write new files there in python with Sage script blocks, the worksheet throws a permissions error going through mpiexec. The fix is to change the simulation.py file to include these lines:

Code: Select all

if cell is not None:
    t, v = cell.get_vectorlist(time = True, voltage = True)
    # this only works with Sage
    np.savetxt('time.out',t)
    np.savetxt('voltage.out',v)
During the run, the current working directory is actually something like "/path/to/published/worksheet/cells/8" and not the "data" directory. As such, this will write the output files to this folder, which are then available for subsequent lines of the script block. Therefore, you can also add the plotting lines to that script block:

Code: Select all

os.system("mpiexec -n 6  sage data/simulation.py")
import numpy as np
voltage, time = np.loadtxt('voltage.out'), np.loadtxt('time.out')
list_plot(zip(time,voltage), plotjoined=True, fontsize=14)
Finally, while savetxt/loadtxt files are human readable, you may want to use the save/load methods that may result in smaller file sizes.

Re: Parallel Neuron: NetCon failed to connect different hosts

Posted: Fri Jan 07, 2011 4:59 pm
by Nin
Very nice advices! I have changed the Sage notebook as you suggested, and now everything can be viewed online. Thanks again for your nice Parallel Neuron tutorials, without them, I would not have done it!!!!.