Automatically run multiple simulations.

Particularly useful chunks of hoc and/or NMODL code. May be pedestrian or stunningly brilliant, may make you gasp or bring tears to your eyes, but always makes you think "I wish I had written that; I'm sure going to steal it."
Post Reply
OspreyNeurons

Automatically run multiple simulations.

Post by OspreyNeurons »

Hello all,

I am running some simulations for a research project and I would like to automate the process. Currently I have to go into the .hoc file and change the .DAT file name each run and I have hundreds of runs. Each .DAT file consists of information about where to place the synapses. I would like to build my program to start with the first file, say Sh_1.DAT and run through Sh_300.DAT. Also the model I am using was built so you could select from 4 different neurons with the user interface and run the program for that specific cell by typing into the shell terminal, myrun() and hitting enter. Then the simulation occurs and the information is put out to a .csv file.

So basically I would like the program to pull information from a file, select the right cell model, perform the simulation, and produce the output, then open the next file and do the same thing for x amount of files.

Here is the code for the file input:
// Open input and output files, and specify input file type
run_name = "sh2_a_1"
sprint(randnum_input, "shells/%s.dat", run_name)
sprint(randnum_output, "shells/%s.out", run_name)
sprint(vplot_name, "shells/%s.plt", run_name)
f = new File()
f.ropen(randnum_input)
wdata = new File()
wdata.wopen(randnum_output)
csv = new File()
csv.aopen("shells/shells1.csv")
vplot = new File()
vplot.wopen(vplot_name)


Here I believe is the code for the cell user interface menu that pops up when I load the .hoc file and what's behind the buttons (I am running j8 which is the L3 pyramid):
// Instructions
printf("To run simulation, type:\n")
printf("\tmyrun()\n")

// wdata.close()
}

proc fig1a() {
load_3dcell("cells/lcAS3.hoc")
st.amp = 0.05
}

proc fig1b() {
load_3dcell("cells/j7.hoc")
st.amp = 0.07
}

proc fig1c() {
load_3dcell("cells/j8.hoc")
// st.amp = 0.1
}

proc fig1d() {
load_3dcell("cells/j4a.hoc")
st.amp = 0.2
}

proc myrun() {
vrecord() // Set up vector to store soma.v(0.5)
init()
run()
vmax() // find Vmax and tmax
multiples() // find segments with more than one synapse
// End row in csv file and close
csv.printf("\n")
csv.close()
// close output file
wdata.close()
// close vplot file
vplot.close()
}

xpanel("Figure 1")
xbutton("a. L3 Aspiny","fig1a()")
xbutton("b. L4 Stellate","fig1b()")
xbutton("c. L3 Pyramid","fig1c()")
xbutton("d. L5 Pyramid","fig1d()")
xpanel()

nrnmainmenu()
nrncontrolmenu()
newPlotV()



Any advice would be greatly appreciated as it would really help me focus my time on analysis of the results instead of exercising the motor cortex :]

Thanks,
Kevin
ted
Site Admin
Posts: 6287
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Automatically run multiple simulations.

Post by ted »

There are several problems here. One is the obvious issue of how to deal with each run's parameters. The other is how to keep each run from depending on prior runs or interfering with subsequent runs. This is a special concern in your case because different model cells will be used in different simulations. If these model cells are created by code that executes at the top level, i.e. outside of an object, then there are going to be problems when new sections are created--what happens to the sections left over from the previous run? Ditto for any object instances that may be created in the course of model setup or for the purpose of instrumentation.

The only sure-fire approach to this problem is to factor your code into two categories:
1. A "model setup code" category that creates all the cell instances that are needed for a given run, takes care of all parameters and any custom initialization associated with that run, and accumulates whatever data are to be saved from that run. This category should contain anything that is unique from run to run. It might also contain the code that executes a run. Turn this code into a class definition by wrapping it inside a template. If you're project involves independent simulations of 4 different kinds of models, at a minimum you'll have 4 different classes (you might have more if you are two or more different things to each kind of model).
2. A "purely administrative code" category that is responsible for carrying out a sequence of simulations, including writing data to output files. This code will be responsible for overall supervision of your sequence of simulation experiments. It does this:

Code: Select all

while (there is yet another run to be executed) {
  create an instance of the appropriate model class
  call that instance's run() method
  do whatever you must with the data accumulated by the model instance
  kill this model instance // destroys all traces that it existed,
    // so you're ready to create a new, potentially different instance
}
To avoid getting lost in the weeds, it's best to work through this with a couple of toy models that run very short simulations--just enough to generate different results. Once you have that working properly, you'll be ready to deal with your "real" models. (all models are deeply unreal)
OspreyNeurons

Re: Automatically run multiple simulations.

Post by OspreyNeurons »

Hey Ted, thanks for your timely response.
I actually will only be using one cell from the model so that makes things a bit easier. I would really like to run Neuron in a 'batch' mode.

I have found that I can run a hoc file through Neuron by entering in the following code instead of dragging and dropping the file onto nrngui (I am using a Mac):

Code: Select all

cd ; clear; cp .nrn_as2sh /tmp/nrn_as2sh$$; sh /tmp/nrn_as2sh$$;rm -f /tmp/nrn_as2sh$$ ; exit 
I see that the .nrn_as2sh file consists of the following commands:

Code: Select all

#!/bin/sh                                                                       
NRNHOME="/Applications/NEURON-7.3/nrn"
NEURONHOME="${NRNHOME}/share/nrn"
CPU=i386
NRNBIN="${NRNHOME}/i386/bin/"
PATH="${NRNHOME}/i386/bin:${PATH}"
export NRNHOME
export NEURONHOME
export NRNBIN
export PATH
export CPU
nrncarbon=yes
export nrncarbon
cd "${NRNHOME}/i386/bin"
./nrngui.sh "/Applications/NSD 2013/NSD2013(Shells1-4)(debug).hoc"
I was able to code the hoc file to load the appropriate cell and execute the simulation automatically however I can't seem to find a way to exit the Neuron program without shutting down the shell as well. I tried quit() but it quits everything. My intention is to leave the shell open and run the nrn_as2sh commands with the first hoc file then do it again with a new hoc file and so on. I would be ok building a program to run the nrn_as2sh commands repeatedly with different hoc files each time. My question is how to have the shell run through those commands one after the other? It would need to open Neuron, run that simulation, close Neuron, then do it again with a different hoc file.

Is there maybe a code to load a hoc file into Neuron once it is already open? Maybe I could have it load one hoc file after the other following the completion of their runs. Right now it seems that the 'batch' approach with the repeated nrn_as2sh might be the most straight forward solution.

Thanks again,
Kevin
ted
Site Admin
Posts: 6287
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Automatically run multiple simulations.

Post by ted »

Sorry it took so long to get back to you, but we were wrapping up the NEURON course and things were busier than usual.

Open a terminal and see if you can't just launch NEURON by typing
neurondemo
If that works, it should be sufficient to call
nrniv foo.hoc
from any shell script. If not, your .profile (or whatever equivalent is used by OS X) isn't setting up the necessary environment variables.

nrniv foo.hoc
exits after completion.
nrngui foo.hoc
keeps NEURON running until you force an exit, either by typing
^D
at the oc> prompt or executing quit()

Instead of repeated execution supervised by a shell script, you might prefer to use NEURON's bulletin-board style parallelization--would speed up your work if you have a multicore Mac (who doesn't these days?). Download and expand http://www.neuron.yale.edu/ftp/neuron/course.zip,
descend into the directory bulletin_board_parallelization and use your browser to examine the file bulletin_board_parallelization.html
OspreyNeurons

Re: Automatically run multiple simulations.

Post by OspreyNeurons »

No worries.

The series batch won't work for my particular situation as all the synapses will change between each run rather then adjusting a parameter.

My environment variables were not set correctly but I am able to temporarily fix that by using the following code (although once I shut down the shell the changes don't stick):

Code: Select all

PATH=$PATH:$HOME/bin:/Applications/NEURON-7.3/nrn/i386/bin; export PATH
I can deal with that by initializing first with the above code and proceeding with simming out the runs utilizing a shell-hosted program. Now my issue is that Neuron doesn't seem to recognize my hoc file. I receive the following message:

Code: Select all

unknown002332bfc5ba:~ Kmac$ nrniv NSD2013ShellsAuto.hoc
NEURON -- Release 7.3 (849:5be3d097b917) 5be3d097b917
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2013
See http://www.neuron.yale.edu/neuron/credits

0 nrniv: can't open NSD2013ShellsAuto.hoc
First I thought to add a path towards that file which didn't work, then I moved the file into nrn/i386/bin but to no avail. Thoughts?
ted
Site Admin
Posts: 6287
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Automatically run multiple simulations.

Post by ted »

OspreyNeurons wrote:The series batch won't work for my particular situation as all the synapses will change between each run rather then adjusting a parameter.
Easily fixed. Innervate the model with a proc that destroys any pre-existing synapses and NetCons. Example (partly in pseudocode):

Code: Select all

// assumes ns is a NetStim that drives all synapses
objref synlist, nclist

proc innervate() { localobj tobj
  objref synlist // discard pre-existing synapses and NetCons
  objref nclist
  forall for (x,0) {
    if(this_segment_needs_a_new_synapse()) {
      tobj = new Syn(x)
      synlist.append(tobj)
      nclist.append(new NetCon(ns, tobj))
    }
  }
}
This can be called as many times as you like without stepping on any other aspect of your model.
My environment variables were not set correctly but I am able to temporarily fix that by using the following code
If you're going to be doing much with NEURON you may want to fix that. In your home directory you could
cp .profile OLD_PROFILE
cp .profile NEW_PROFILE
add your PATH statement to NEW_PROFILE
cp NEW_PROFILE .profile
Now my issue is that Neuron doesn't seem to recognize my hoc file.
Aren't you running nrniv in the directory that contains your hoc file? If not, why not? If you simply launch nrngui, then at the oc> prompt type
system("ls")
you ought to see whatever hoc file you're trying to run. If not, cd to the appropriate location.
I thought to add a path towards that file which didn't work
Why not put your hoc code in a directory that is easy to reach from an xterm?
I moved the file into nrn/i386/bin
A very bad idea. Don't put anything inside the installation tree, and stay out of the installation tree.
OspreyNeurons

Re: Automatically run multiple simulations.

Post by OspreyNeurons »

For whatever reason my shell doesn't like trying to make a copy of .profile. It clearly states in the UNIX library that cp will copy files and directories but when I try to use cp I get that .profile is a directory (Not copied).

I was able to place the hoc files were I can access them.

We use a program to determine where the synapses will be placed and the output files are utilized in a few different fashions throughout the hoc file. Therefore I think I will have to go with a more archaic process to run each modified hoc file which individually pulls in the different dat files. I have a working shell program right now that should do the trick.

Thank you very much for your advice and help!

-Kevin
Post Reply