Gap junctions with ParallelContext

mlazarew · Post by **mlazarew** » Wed May 16, 2007 1:28 pm

I am trying to understand how to use a ParallelContext class with a network of neurons connected by gap junctions across nodes.

I have tried to run a simple program (below) on two nodes. It should transfer the value 99 of the variable "a" between nodes 1 and 0.

If my program is correct, then I should see a value 99 assigned to the variable "a" on both nodes 0 and 1. But after running the program the variable "a" has the value 99 only on the node 1, where it was set to this value, and value 1 on the node 0, where it should be transfered. That means that the transfer across nodes did not happened.

The most probably I am not using these methods properly.

Does somebody know how to use them correctly?

Code: Select all

load_file("nrngui.hoc")
objref pc

proc set_maxstep() { pc.set_maxstep(5) }
proc doinit()      { stdinit() }
proc psolve()      { pc.psolve($1) print_a() }
proc print_a()     { print "My id: ", pc.id, " a= ", a, "time= ", t }
proc tr()          { pc.setup_transfer() }
proc set_a()       { if (pc.id == 1) a = 99 }

cvode.active(0)

pc    = new ParallelContext()
tstop = 100
a     = 1

if (pc.id == 0) {
    pc.target_var(&a, 0)
    pc.setup_transfer()
}

if (pc.id == 1) {
    pc.source_var(&a, 0)
    pc.setup_transfer()
}
    
pc.runworker()

pc.context("set_maxstep()\n")
set_maxstep()

pc.context("tr()\n")
tr()

pc.context("doinit()\n")
doinit()

pc.context("set_a()\n")

pc.context("psolve", tstop)
psolve(tstop)

pc.done()
quit()

numprocs=2
NEURON -- VERSION 6.0.870 (1745) 2007-05-16 (1745M)
by John W. Moore, Michael Hines, and Ted Carnevale
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2007

1
1
0
1
1
1
1
1
My id: 1 a= 99 time= 99.975
My id: 0 a= 1 time= 99.975
0

Post by **hines** » Fri May 18, 2007 1:10 pm

Does the following work:

Code: Select all

{load_file("nrngui.hoc")}
objref pc
proc set_maxstep() { pc.set_maxstep(5) }
proc doinit()      { stdinit() }
proc psolve()      { pc.psolve($1) print_a() }
proc print_a()     { printf("My id: %d a=%g time=%g\n", pc.id, a, t) }
proc tr()          { pc.setup_transfer() }
proc set_a()       { if (pc.id == 1) a = 99 }

{cvode.active(0)}

pc    = new ParallelContext()
tstop = 100
a     = 1

if (pc.id == 0) {
    pc.target_var(&a, 0)
}

if (pc.id == 1) {
    pc.source_var(&a, 0)
}

set_maxstep()
tr()
doinit()
set_a()
print_a()
psolve(tstop)

{pc.runworker()}
{pc.done()}
quit()

On my machine it prints:

Code: Select all

[hines@localhost Parallel]$ mpiexec -np 2 /home/hines/neuron/nrnmpi/x86_64/bin/nrniv -mpi maciej.hoc
numprocs=2
NEURON -- VERSION 6.0.870 (1745) 2007-05-16
by John W. Moore, Michael Hines, and Ted Carnevale
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2007

My id: 0 a=1 time=0
My id: 1 a=99 time=0
My id: 0 a=99 time=99.975
My id: 1 a=99 time=99.975
[hines@localhost Parallel]$

Also my machine hanged on your original code in one of the pc.context calls. I'll look further into that. Notice how I enclose each of the function calls that would print 1 or 0 within {}. Also I replaced the print statement in print_a with a printf which prints the line atomically and prevents mixing from different pc.id.

mlazarew · Post by **mlazarew** » Fri May 18, 2007 1:30 pm

On my machine I am getting the following error:

Code: Select all

mpiexec -n 2 /Volumes/MTL/neuron/nrn/powerpc/bin/nrniv -mpi  /Volumes/MTL/CA1/maciej.hoc 

numprocs=2

NEURON -- VERSION 6.0.870 (1745) 2007-05-16 (1745M)
by John W. Moore, Michael Hines, and Ted Carnevale
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2007

Additional mechanisms from files
 Exp2SynNMDA.mod caolmw.mod capr.mod gap.mod icaolmw.mod icappr.mod iholmw.mod imppr.mod kahpppr.mod kcaolmw.mod kcppr.mod kdrbwb.mod kdrmsw.mod kdrppr.mod koppr.mod ksmsw.mod nafbwb.mod nafmsw.mod nafppr.mod naoppr.mod
1 /Volumes/MTL/neuron/nrn/powerpc/bin/nrniv: multiple instances of source gid: 0
1  in /Volumes/MTL/CA1/test_hines.hoc near line 25
1  tr()
     ^
        1 ParallelContext[0].setup_transfer()
      1 tr()
dyld: lazy symbol binding failed: Symbol not found: _MPI_Initialized
  Referenced from: /Volumes/MTL/neuron/nrn/powerpc/lib/libnrnmpi.0.dylib
  Expected in: flat namespace

dyld: Symbol not found: _MPI_Initialized
  Referenced from: /Volumes/MTL/neuron/nrn/powerpc/lib/libnrnmpi.0.dylib
  Expected in: flat namespace

rank 1 in job 46  d221mac4_87651   caused collective abort of all ranks
  exit status of rank 1: killed by signal 5

But after I modified the code to

Code: Select all

{load_file("nrngui.hoc")}
objref pc
proc set_maxstep() { pc.set_maxstep(5) }
proc doinit()      { stdinit() }
proc psolve()      { pc.psolve($1) print_a() }
proc print_a()     { printf("My id: %d a=%g time=%g\n", pc.id, a, t) }
proc tr()          { pc.setup_transfer() }
proc set_a()       { if (pc.id == 1) a = 99 }

{cvode.active(0)}

pc    = new ParallelContext()
tstop = 100
a     = 1

if (pc.id == 0) {
    pc.target_var(&a, 0)
    tr()
}

if (pc.id == 1) {
    pc.source_var(&a, 0)
    tr()
}

set_maxstep()
{printf("ID: %d\n", pc.id)}
//tr()
doinit()
set_a()
print_a()
psolve(tstop)

{pc.runworker()}
{pc.done()}
quit()

then I am getting:

Code: Select all

numprocs=2
NEURON -- VERSION 6.0.870 (1745) 2007-05-16 (1745M)
by John W. Moore, Michael Hines, and Ted Carnevale
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2007

Additional mechanisms from files
 Exp2SynNMDA.mod caolmw.mod capr.mod gap.mod icaolmw.mod icappr.mod iholmw.mod imppr.mod kahpppr.mod kcaolmw.mod kcppr.mod kdrbwb.mod kdrmsw.mod kdrppr.mod koppr.mod ksmsw.mod nafbwb.mod nafmsw.mod nafppr.mod naoppr.mod
ID: 0
ID: 1
My id: 0 a=1 time=0
My id: 1 a=99 time=0
My id: 0 a=1 time=99.975
My id: 1 a=99 time=99.975

That means that there is some problem on my machine that prevents me to use successfully the inter-node gap junction.

Post by **hines** » Fri May 18, 2007 5:14 pm

Code: Select all

1 /Volumes/MTL/neuron/nrn/powerpc/bin/nrniv: multiple instances of source gid: 0

This is exceedingly puzzling. The order of pc.set_maxstep() and pc.setup_transfer() should not matter. The message suggests that pc.source_var(&a, 0) was executed twice but that manifestly is not the case.

Code: Select all

dyld: lazy symbol binding failed: Symbol not found: _MPI_Initialized

Since setup_transfer failed, NEURON called nrnmpi_abort which calls MPI_Abort() or abort() depending on the flag MPI_Initialized returns. The mac requires that all the MPI functions that NEURON calls be listed in the file, nrn/src/oc/ockludge.c . I see that MPI_Initialized is missing from the list and I'll have to add it to make the message go away for the mac.

From my previous message

Also my machine hanged on your original code in one of the pc.context calls.

I've concluded that is due to a race condition between how the bulletin board communicates and the direct MPI_... calls for spike exchange, setup, transfer, etc. The bottom line is do not make a pc.context(statement\n)
call where the statement does something that uses any non-bulletin board MPI call.

Post by **hines** » Sat May 19, 2007 8:47 am

The problem was a missing, reasonable, error message. The svn repository version will now state:

Code: Select all

numprocs=2
NEURON -- VERSION 6.0.872 (1747) 2007-05-19
...
0...: To use ParallelContext.setup_transfer when nhost > 1, NEURON must be configured with --with-paranrn
...