Strange behavior Updating HOC Vectors inside Python+mpi deff

[R_Jarvis]

I have written an analysis to calculate transfer entropy on a large data set. The data set is so large I am using MPI such that I have access to more memory.

I am having trouble updating HOC Vectors inside Python definition, whenever I am using mpi4py as well, which is why I have made tautology statements like: 'h.py.j=j', the same is true for h.py.k=k, h.py.i=i. Strangely a statement like that will make update the python variable j, k, and i and if I did not make such a statement the values of j k i would not be updated. Which is catastrophic when they required to act as iterators in loops.

Sometimes k does get updated and sometimes it doesn't, it seems to be influenced by how many statements come before it. Is it possible that this is caused by blocking behavior of synchronous MPI calls like broadcast? Maybe the HOC object in Python is not designed to work with mpi4py.

Code: Select all

def t_e(SIZE, NCELL, RANK, trentm):
  
    h('storval = new Vector()')
    h('objref vec')
    h('vec=new Vector()')


    for k in xrange(0, SIZE):
        for j in xrange(0, int(NCELL)):
            if RANK == k:
                histogram = []
                test = 0
                h.py.j=j
                h('py.test=py.int(pc.gid_exists(py.int(py.j)))')
            
                if test != 0:
                    h('source=pc.gid2cell(py.int(py.j))')
                
                    h('vec=source.recvec1')

                    h('vec.x[0]=py.j')

                    histogram = h.source.recvec1.to_python()
                if test == 0:
                    histogram = None
            else:

                histogram = None
            histogram = COMM.bcast(histogram, root=k)  # ie root = rank
            h.py.k=k
            
            h('py.gidn=int(vec.x[0])')
            h('vec.x[0]=0')
            gidn = int(gidn)
            h('py.test2=int(vec.sum())')
            test2 = h.vec.sum()


            if int(test2) != 0:
                print 'cell number ', gidn
                for i in xrange(0, int(NCELL - 1)):
                    h.py.i=i
                    test3 = 0
                    h('py.test3=py.int(pc.gid_exists(py.int(py.i)))')
                    if test3 != 0:
                        h('target=pc.gid2cell(py.int(py.i))')
                   
                        h('storval=normte(target.recvec1,vec,20)')
                
                        
                        trentm[i][gidn] =float(h.storval.x[2])
    return trentm           

trentm=t_e(SIZE, NCELL, RANK, trentm,np)

Thanks for any help or suggestions. I think there will definitely be ways of simplify the code, I am open to any suggestions about how to do this. I think the whole thing should be rewritten as a more general python definition that performs a broadcast inside an iterator and returns a matrix.

[aaronmil]

Why are your storval and vec objects created through the "last resort" hoc interpreter? Just declare them as Python objects like:

Code: Select all

storval = h.Vector()
vec = h.Vector()

Then interact with the objects purely in Python, rather than hoc.

[hines]

To further emphasize aaronmil's point, I would also replace all the lines analogous to

Code: Select all

h('py.test3=py.int(pc.gid_exists(py.int(py.i)))')

with

Code: Select all

test3=int(pc.gid_exists(i))

ie. use pc directly in python with perhaps
pc = h.ParallelContext()
rank = int(pc.id())
nhost = int(pc.nhost())

However, you mentioned the problem:

I am having trouble updating HOC Vectors

h.py.i is in a different namespace than your local i used in your def. Circumstances where you need h('...') are very rare.

[R_Jarvis]

Okay great. Thanks for the advice, I will try to write everything in a more pythonic way. I also ran into trouble when I tried that once. I will post what happened here later.