Logging parameter and error values during optimization

Using the Multiple Run Fitter, praxis, etc..
Post Reply
ted
Site Admin
Posts: 5330
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Logging parameter and error values during optimization

Post by ted » Sun May 22, 2005 11:41 am

On 3/16/2005 Randy Powers <rkpowers@u.washington.edu> asked:
1. Is there any way to get the MultiRun fitter to save a log of the parameter and error values for all of the runs?
2. Does anyone have an example of a hoc file that uses multiple fitters to simultaneously fit voltage- and current-clamp data?

ted
Site Admin
Posts: 5330
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Post by ted » Sun May 22, 2005 11:43 am

On 3/21/2005 Michael Hines <michael.hines@yale.edu> wrote:
The Optimize panel of the MulRunFitter has a checkbox for
appending the path the fitter takes to "savepath.fit". However
this is not every run but only the best result after each quadratic form.
You can look at nrn/lib/hoc/mulfit/fitparm.hoc and put a call to saveval()
in the praxis_efun() function to get info after each multiplerun.

A multiple run involving both voltage clamp and current clamp data
presents no special difficulties but I don't have an example. The key
would be the protocol statements which would have to do several things,
most importantly turning off one when the other was turned on. For IClamp
turning off is most easily accomplished by setting the pulse amplitude
to 0. For SEClamp, one would have to set dur1, dur2, dur3 so that each
was negative. (The OClamp is just a clone of SEClamp but is a single
pulse with "on" and "off" times along with a "switched_on" flag that
activates the clamp)

shailesh
Posts: 104
Joined: Thu Mar 10, 2011 12:11 am

Re: Logging parameter and error values during optimization

Post by shailesh » Tue Jun 10, 2014 7:39 am

As savepath.fit (by default) 'does not save after every run but only the best result after each quadratic form', I wanted to make changes such that it logged data after each run. I tried the below:
You can look at nrn/lib/hoc/mulfit/fitparm.hoc and put a call to saveval()
in the praxis_efun() function to get info after each multiplerun.
But I suppose some changes have been made to MRF over time as I could not locate the function "praxis_efun()" in fitparm.hoc or any of the other files in 'mulfit' folder.

I tried to work out where the necessary changes to be made, and my best attempt was changing the function "call_opt_efuc()" in 'fitparm.hoc' by adding a call to "after_quad()" as follows:

Code: Select all

func call_opt_efun() {
	after_quad()
	return opt.optwrap_efun($1, &$&2)
}
Also, the earlier call to "after_quad()" in function "prun()" was taken out.
Now I do get logs for almost all the runs, but what is troubling me is that the GUI shows "# mutliple runs" as 54, whereas 'savepath.fit' finishes off with printing the log for the 52nd run. Why is it missing some runs()? Also, is there a better way to go about doing this?

Not sure but maybe relevant: A few of the records intermittently show the same simulation run number but different quadratic equation number. Also, the reason for calling "after_quad()" rather than just "saveval()" was since the latter did not record error values amongst other data.

hines
Site Admin
Posts: 1517
Joined: Wed May 18, 2005 3:32 pm

Re: Logging parameter and error values during optimization

Post by hines » Wed Jun 11, 2014 10:05 am

in optwrap.hoc, optwrap_efun() increments the nefun counter. But optwrap_efun is called also from at the beginning and end of the prun() in optwrap.hoc without going through
call_opt_efun() in fitparm.hoc.

fitparm.hoc is partly praxis specific. i.e MulfitPraxWrap.prun, the optwrap is intended to allow one to provide new optimizer methods.

shailesh
Posts: 104
Joined: Thu Mar 10, 2011 12:11 am

Re: Logging parameter and error values during optimization

Post by shailesh » Thu Jun 12, 2014 2:03 am

I found the two additional calls to "optwrap_efun()" from "prun()" in 'optwrap.hoc'... but as it is part of a different template (OptimizerWrap) I could not invoke "after_quad()". As the latter belongs to template "MulfitPraxWrap", I tried locating an instance/object of that class, but couldn't find one. Apparently it is used to set the Optimizer directly by:

Code: Select all

opt.set_optimizer("MulfitPraxWrap")
So could not figure out how I could call after_quad() from there.
Is there any other (better/cleaner) alternative method to enable logging data of each simulation run?

hines
Site Admin
Posts: 1517
Joined: Wed May 18, 2005 3:32 pm

Re: Logging parameter and error values during optimization

Post by hines » Thu Jun 12, 2014 6:52 am

Perhaps the savepath stuff should also have been factored out of after_quad() in MulFitPraxWrap in fitparm.hoc into a small interface in OptimizerWrap in optwrap.hoc.
Then it would be simpler to implement different choices for how often to record the parameters and return values of the elementarty fitness function. However, one could also imagine wanting
to record the component fitnesses of the global fitness function. after_quad is a praxis specfic concept. Anyway, I believe in your case a reasonable choice is to do the file recording from within
optwrap_efun using your own implementation. Alternatively, the OptimizerWrap has a reference, pf, to the MulFitPraxWrap instance, so you can make after_quad and/or savepath and/or saveval
public in the fitparm.hoc file
in order to call it after the pair of explicit optwrap_efun calls in prun in optwrap.hoc
Actually, I think things would be less confusing if you copied the relevant fragments into optwrap.hoc, and called your new function directly from optwrap_efun.

Note that you can have your own version of the mulrunfitter by copying the multfit.hoc file into your project and also copying the entire mulfit folder. That way, when you upgrade to newer
versions of NEURON that will not affect your modified version of mulfit. (the general problem of a user modified version of NEURON that is easy to upgrade without losing local changes
can also be managed by using mercurial or git.)

shailesh
Posts: 104
Joined: Thu Mar 10, 2011 12:11 am

Re: Logging parameter and error values during optimization

Post by shailesh » Thu Jun 12, 2014 10:08 am

I took your last suggestion and created a custom function for logging data after each run. It appears to be working well after resolving some errors along the way. Posting it here for your confirmation and in case it would be handy for anyone else:

optwrap.hoc needs to be updated to:

Code: Select all

begintemplate OptimizerWrap

external classname, mulfit_optimizers_, mulfit_optimizer_names_

// public functions
public prun, set_optimizer, save_optimizer, optwrap_efun, showopt
// public variables
public saveflag, start, nquad, wfile, optimizer, opt_index, minerr, nefun, time

objref savepath, tl, pf, tobj, opt, this, start, optimizer
strdef tstr

proc init() {
	pf = $o1
	start = new Vector()
	savepath = new File()
	nefun = 0
	st = startsw()
	time = 0
	currenterr = 1e9
	saveflag = 0
	e = 0
}

proc set_optimizer() {
    for i=0, mulfit_optimizers_.count-1 { // need to know index too
		if (!strcmp(mulfit_optimizers_.object(i).s, $s1)) {
			opt_index = i
	    }
    }
	sprint(tstr, "optimizer = new %s(pf)", $s1)
	execute(tstr, this)
}

//--------------------------------------------------------------
objref saveDataPath
//Function for logging data after each run into a temporary file
proc saveTempFile() {
	if (saveflag) {
		saveDataPath = new File()	//Added
		saveDataPath.wopen("savepath_complete.tmp")
	}
}

proc mySaveData() {
	if (saveDataPath.isopen) {		
		saveDataPath.printf("%d %d %-12.8g ", nefun, time, minerr)
		saveval_modified()
	}	
}

//Saves the current parameter set
proc saveval_modified() {
	if (numarg() == 1) saveDataPath.printf("%s", $s1)
	tl = pf.parmlist
	for i=0, tl.count-1 {
		saveDataPath.printf("%-12.8g ", tl.object(i).val)
	}
	saveDataPath.printf("\n")
}

//Creates final output file
proc saveFile() {
	if (saveDataPath.isopen) {
		saveDataPath.close
		saveDataPath.aopen("savepath_complete.fit")
		saveDataPath.printf("start\n")
		saveDataPath.close
		system("cat savepath_complete.tmp >> savepath_complete.fit")
	}
}
//--------------------------------------------------------------

func optwrap_efun() {local i
	nefun += 1
	e =  pf.efun($1, &$&2)
	if (!stoprun) {
		if (minerr == -1 || e < minerr) {
			minerr = e
		}
	}
	
	//--------------------------------------------------------------
	//Function call to save data after each run
	if (saveflag) {
		mySaveData()
	}
	//--------------------------------------------------------------
	
	doNotify()
	time = startsw() - st
	return e
}

proc saveval() {
	if (numarg() == 1) savepath.printf("%s", $s1)
	tl = pf.parmlist
	for i=0, tl.count-1 {
		savepath.printf("%-12.8g ", tl.object(i).val)
	}
	savepath.printf("\n")
}

func prun() {
	//--------------------------------------------------------------
	//Call to create temporary log file
	if (saveflag) {
		saveTempFile()
	}
	//--------------------------------------------------------------

	nefun = 0
	minerr = -1
	pf.doarg_get(start)
	nquad = 0
	if (start.size == 0) {
		minerr = pf.efun(0, &time) // time is dummy here
	}else{
		minerr = optwrap_efun(start.size, &start.x[0])
		if (stoprun) {return minerr}
		st = startsw()
		time = startsw() - st
		minerr = optimizer.prun(this)
		time = startsw() - st
		minerr = optwrap_efun(start.size, &start.x[0])
	}
	
	//--------------------------------------------------------------
	//Call to create final output file
	if (saveflag) {
		saveFile()
	}
	//--------------------------------------------------------------
	
	return minerr
}

proc showopt() {
	sprint(tstr, "%s specific items", mulfit_optimizer_names_.object(opt_index).s)
	xlabel(tstr)
	optimizer.showopt()
}

strdef opt_class
proc save_optimizer() { localobj vbox
	vbox = $o1
	classname(optimizer, opt_class)
	// set the optimizer than let it save itself
	sprint(tstr, "opt.set_optimizer(\"%s\")", mulfit_optimizers_.object(opt_index).s)
	vbox.save(tstr)
	vbox.save("{object_push(opt.optimizer)}")	
	vbox.save("{")	
	optimizer.save_optimizer(vbox) 
	vbox.save("}")	
	vbox.save("{object_pop()}")	
}

endtemplate OptimizerWrap
(The additions have been enclosed within dividers - 4 segments)

With the above changes we would get:
> Default "savepath.fit" that MRF provides
> An additional file named "savepath_complete.fit" with data from each simulation run

The implementation is quite similar to the original. The data is saved in the following format:
Line 1: start (just a label)
Line 2 onwards: (data)
- Column 1: Simulation Run Number
- Column 2: Time (s)
- Column 3: Error Value
- Columns 4+: Parameter Values

Post Reply