BB: Sorting values and returning multiple values

[nikkip]

I am using a simple bulletin board setup to parallelize my simulations in hoc. However, I'm struggling with 2 things:

A) I would like to return 2 result values (threshold and conduction speed).

B) I would also like to return/read the input arguments of the pc.submit() corresponding to each set of result values so that I know to which set of parameters they belong.
As you can see, I'm not simply looping through a single parameter, and so I cannot simply sort as shown in the initbatpar.hoc example.

I have a couple of ideas, but not sure if they're doable in hoc:

1) Is there a way to have my function (find_thresh) return multiple values? If so, how do I obtain these values with some variant of pc.retval()? This would address issue A.

2) The Parallel Context intro in the online documentation states that: "The arguments to the function executed by the submit call are also available." If so, how can I access the input arguments (in which case I would know where to store the returned threshold value)? This would address issue B.

3) Both issues could be addressed if the following is allowable: I have a matrix that is defined (and allocated memory) before starting pc.runworker(). Then each job would save its results in its correct location in the results matrix. However, I assume that the parallel processors cannot save a matrix in this fashion, correct?

Barring a more elegant solution, my backup plan is to have each job write its own output file (thus not having to return anything to the master via the bulletin board.



Thank you!

Nikki

[ted]

I am using a simple bulletin board setup to parallelize my simulations in hoc. However, I'm struggling with 2 things:

A) I would like to return 2 result values (threshold and conduction speed).

B) I would also like to return/read the input arguments of the pc.submit() corresponding to each set of result values so that I know to which set of parameters they belong.

Good questions. Strangely enough, the documentation of ParallelContext's submit method discusses these very items. This bears careful reading.
http://www.neuron.yale.edu/neuron/stati ... ext.submit

A. How to return multiple result values.
"In the event more than a single scalar return value is required use ParallelContext.post() within the function_name body with a key equal to the id of the task. For example:

Code: Select all

func function_name() {local id
   id = hoc_ac_
   $o1.reverse()
   pc.post(id, $o1)
   return 0
}
...
while( (id = pc.working) != 0) {
   pc.take(id)
   pc.upkvec.printf
}

"
where the example is a toy problem (the worker is given a Vector that it must reverse and return to the master). If more than one thing is to be returned (i.e. a mix of scalars, Vectors, strdefs, and pickleable Python objects), use the pack() method to compose them into a single message body before posting, then take and unpack the message on the master.

B. How to retrieve submit's arguments
"If there is no explicit userid, then the args (after the function name) are saved locally and can be unpacked when the corresponding working call returns. A local userid (unique only for this ParallelContext) is generated and returned by the submit call and is also retrieved with ParallelContext.userid() when the corresponding working call returns."

Both issues could be addressed if the following is allowable: I have a matrix that is defined (and allocated memory) before starting pc.runworker(). Then each job would save its results in its correct location in the results matrix. However, I assume that the parallel processors cannot save a matrix in this fashion, correct?

Each host would have its own local matrix instance, and no host could access any other host's matrix. All communication between processors is via the bulletin board and must use submit (available only to the master), post, and take.

[nikkip]

Thank you Ted! I was just working on understanding pack, post, and retval/upkscalar/upkvec/etc. I now have this working. However, I'm still struggling with retrieving the input arguments. I can obtain the user ID, as suggested in your post, but then to which function to I feed the user ID to unpack the input arguments that were used?

I'm testing this with very simple code:

Code: Select all

objref pc
pc = new ParallelContext()

objref myvec
myvec = new Vector(3,0)

func myobfunc() { local key, myvar1, myvar2, myvar3
	key = $1
	myvar1 = $2
	myvar2 = $3
	myvar3 = $4

	myvec.x[0] = 2*myvar1
	myvec.x[1] = 3*myvar2
	myvec.x[2] = 4*myvar3
	
	pc.pack(myvec.x[0])
	pc.pack(myvec.x[1])
	pc.pack(myvec.x[2])
	pc.pack(myvec)
	
	pc.post(key)
	
	return key
}

{ pc.runworker() }

objref myvec_returned
myvec_returned = new Vector()

proc batchrun() {local K, cntr
	cntr = 0
	
	for K = 0, 4 {
		pc.submit("myobfunc",cntr,K,K+1,K+2)
		cntr = cntr+1
	}
	
	while (pc.working) {
		x = pc.userid()
		print "userID: ", x
		pc.unpack(x)
		
		key = pc.retval()
		pc.look_take(key)
		
		print "key=", key
		var1 = pc.upkscalar()
		print "var1=", var1
		var2 = pc.upkscalar()
		print "var2=", var2
		var3 = pc.upkscalar()
		print "var3=", var3
		
		myvec_returned = pc.upkvec()
		print myvec_returned.x[0], myvec_returned.x[1], myvec_returned.x[2]
	}
}
batchrun()
{ pc.done() }
quit()

[ted]

I can obtain the user ID, as suggested in your post, but then to which function to I feed the user ID to unpack the input arguments that were used?

Apparently there isn't such a function. Here's a very simple test program that can be executed serially, that you can use as a starting point for trying different ways to submit tasks, retrieve return values, and access input arguments (thanks to Michael Hines for providing this).

Code: Select all

objref pc
pc = new ParallelContext()
func f() {
  return $1*$2
}
pc.runworker()
for i=3, 5 pc.submit("f", i, 4)
while ((id = pc.working()) != 0) {
  print "id ", id
  print "arg1 ", pc.upkscalar()
  print "arg2 ", pc.upkscalar()
  print "return value ", pc.retval()
}

Since there is no explicit userid (that is, pc.submit is called without a userid argument), so when pc.working returns
(1) it returns an automatically generated local userid
and
(2) the arguments arg1 arg2 . . . to the pc.submit that placed this particular task on the bulletin board can be retrieved, one at a time and in the same order as in that pc.submit call, via whichever upk* method is appropriate (upkscalar, upkstr, upkvec, or upkpyobj)
(3) the return value can be retrieved via the retval or pyret method, as appropriate.

If you prefer to call pc.submit with an explicit userid, you'll have to manage arguments yourself. For example, if you have three arguments that are a scalar, a vector, and a string, you could

Code: Select all

objref tmpvec, tmpstr
objref arg1vec, arg2list, arg3list
arg1vec = new Vector() // arg1vec.x[i] will be the scalar arg associated with userid1
arg2list = new List() // arg2list.o(i) will be the vector arg associate with userid1
arg3list = new List() // arg3list.o(i) will be a String (defined in nrn/lib/hoc/stdlib.hoc)
for i = 0,2 {
  x = . . . code that returns a scalar . . .
  arg1vec.append(x)
  tmpvec = . . . code that returns a Vector . . .
  arg2list.append(tmpvec)
  tmpstr = new String() // String.s is a strdef
  . . . code that prints a string to tmpstr.s . . .
  arg3list.append(tmpstr)
  pc.submit(i, "f", x, tmpvec, tmpstr)
}
objref tmpvec, tmpstr

Then you could

Code: Select all

while ((id = pc.working()) !=0 {
  // id is the userid for the task that returned
  // use it as an index into arg1vec, arg2list, and arg3list
  // to recall the corresponding arguments
}

But it's easier not to bother with explicit userid, and just let ParallelContext keep track of the args.