Page 1 of 1

NEURON on Ranger

Posted: Thu Apr 28, 2011 9:00 pm
by neuromau
Hello, Michael Hines installed NEURON on Ranger for me and so I'm posting the instructions here for other users of Ranger.

Ranger is a Linux cluster at the University of Texas which, according to the website (1), has "62,976 compute cores, 123 TB of total memory and 1.7 PB of raw global disk space." Everything's bigger in Texas, right? Ranger resources can be used by those not associated with UT by acquiring one of NSF's TeraGrid allocations (2).

Users with an account on Ranger have access to three filesystems: HOME, WORK, and SCRATCH. The setup below guides you through installing NEURON on the HOME filesystem, but jobs should be run out of WORK or SCRATCH.

To install NEURON on your Ranger account:
1. On Ranger, cd to your HOME directory:

Code: Select all

cdh
2. Create a directory named 'neuron':

Code: Select all

mkdir neuron
3. Enter the neuron directory:

Code: Select all

cd neuron
4. Put the desired nrn-nn.tar.gz file (either the current standard distribution (3) or a development version (4)) in the neuron directory. Note that you will not be installing Interviews, so iv-nn.tar.gz is not needed.

5. Unpack the file:

Code: Select all

tar xzf nrn-nn.tar.gz
6. Rename the unpacked directory:

Code: Select all

mv nrn-nn nrn
7. Create the nrninstall.sh script that will install NEURON on Ranger. This file should contain the following text:

Code: Select all

#!/bin/sh
../nrn/configure --prefix=`pwd` --with-nmodl-only --without-x
make
make install

../nrn/configure --prefix=`pwd` '--without-nmodl' '--without-x' \
 '--without-memacs' '--with-paranrn' 'CC=mpicc' 'CXX=mpicxx' \
 '--disable-shared' linux_nrnmech=no
make
make install
There are a few observations to make about this script. First, to paraphrase a previous comment by Ted & Michael (5), it is cross compiling NEURON because the login machine where you submit the job to run your model code is configured differently than the processors on which your model code will actually run. Second, this particular compilation does not include Python, though that is an option. Third, there are several other arguments that you probably didn't include when installing NEURON on your own machine; these arguments are nicely explained elsewhere on the forum (5).

8. In the neuron directory, create a second folder called 'mpi':

Code: Select all

mkdir mpi
This is the directory in which NEURON will be built. The nrn directory will be kept clean, in case you want to build other versions of NEURON on your Ranger account.

9. Enter the mpi directory:

Code: Select all

cd mpi
10. From the mpi directory, run the install script and write the output and errors to a log:

Code: Select all

sh ../nrninstall.sh > & installnotes.log
Testing the installation and running job scripts are covered in the reply to this post.

Links
1. Ranger User Guide: http://services.tacc.utexas.edu/index.p ... user-guide
2. TeraGrid: http://www.teragrid.org
3. Standard Distribution: http://www.neuron.yale.edu/neuron/download/getstd
4. Development Versions: http://www.neuron.yale.edu/neuron/download/getdevel
5. Cross Compiling NEURON: http://www.neuron.yale.edu/phpBB/viewto ... 71&start=0

~ Marianne Case

Re: NEURON on Ranger

Posted: Thu Apr 28, 2011 9:05 pm
by neuromau
neuromau wrote:Testing the installation and running job scripts are covered in the reply to this post.
To test the installation of NEURON, we can create the following test program.

1. Move to the WORK file system:

Code: Select all

 cdw
2. Create a directory called test:

Code: Select all

mkdir test
3. Move to the test directory:

Code: Select all

cd test
4. In the test directory, create a file called test0.hoc, and include the following text in the file:

Code: Select all

objref pc
pc = new ParallelContext()
{printf("I am %d of %d\n", pc.id, pc.nhost)}
{pc.runworker()}
{pc.done()}
quit()
2. In the test directory, create a mod file called test.mod, and include the following text in the file:

Code: Select all

NEURON {SUFFIX nothing}
3. Compile the test mechanism:

Code: Select all

nrnivmodl
Now we want to run the test code. To do this, we must create a job script and submit the job script to the batch queue.

4. Create a file called mytest.sh and include the following lines:

Code: Select all

#!/bin/bash
#$-V
#$-cwd
#$-N myMPItest
#$-j y
#$ -o $JOB_NAME.o$JOB_ID
#$ -pe 16way 32
#$ -q development
#$ -l h_rt=00:05:00
#$ -M youremail@domain.edu
#$ -m e
set -x
ibrun x86_64/special -mpi test0.hoc
5. Now submit this script to the batch queue by entering the following at the command prompt:

Code: Select all

 qsub mytest.sh
The batch queue software will perform a series of checks before assigning our job a run number, adding our job to the queue, and printing a confirmation of submission:
Your job 1926013 ("myMPItest") has been submitted
6. We can check on the status of our job using the following command:

Code: Select all

qstat
The Ranger User Guide (1) includes much more detail about submitting jobs, including the various options in the job script and other commands for checking the status of our job or the various queues.

7. Once the job is complete, an output file called myMPItest.o1926013 is generated. It concatenated the name we specified in our job script options with a run number generated by the batch queue. When we open this file, we should see some notes from the batch queue, a copy of the job script we submitted, and any output generated by our job (most of the output lines have been omitted in the example below):
TACC: Setting memory limits for job 1924223 to unlimited KB
TACC: Dumping job script:
--------------------------------------------------------------------------------
#!/bin/bash
#$-V
#$-cwd
#$-N myMPItest
#$-j y
#$ -o $JOB_NAME.o$JOB_ID
#$ -pe 16way 32
#$ -q development
#$ -l h_rt=00:05:00
#$ -M youremail@domain.edu
#$ -m e
set -x
ibrun x86_64/special -mpi test0.hoc

--------------------------------------------------------------------------------
TACC: Done.
+ ibrun x86_64/special -mpi test0.hoc
TACC: Starting up job 1924223
TACC: Setting up parallel environment for MVAPICH ssh-based mpirun.
TACC: Setup complete. Running job script.
TACC: starting parallel tasks...
numprocs=32
NEURON -- VERSION 7.2 (436:cbc0331180f8) 2010-04-05

Duke, Yale, and the BlueBrain Project -- Copyright 1984-2008
See http://www.neuron.yale.edu/credits.html

Additional mechanisms from files
test.mod
I am 13 of 32
I am 0 of 32
...
I am 20 of 32
I am 31 of 32
TACC: Shutting down parallel environment.
TACC: Shutdown complete. Exiting.
TACC: Cleaning up after job: 1924223
TACC: Done.
Note: If we wish to add any command line arguments (using Michael Hines' default_var function), we can add them to the ibrun line as follows:

Code: Select all

ibrun x86_64/special -mpi -c mytstop=20 test0.hoc
However, if we want to include a command line argument that uses quotes, we must put it in a different file type and have the ibrun reference that other file. In that case, the 'ibrun' line of our job script looks like:

Code: Select all

ibrun ./jobscripts/mytestrun.sh
And another file called mytestrun.sh must be created, which will actually contain the call to NEURON and the command line arguments:

Code: Select all

#!/bin/csh
x86_64/special -mpi -c 'strdef RunName' -c 'RunName="bigrun"' -c mytstop=20 test0.hoc
When we submit the job, we will still submit the main sh file, which will in turn reference the mytestrun.sh csh file

Links
1. Ranger User Guide: http://services.tacc.utexas.edu/index.p ... user-guide



~ Marianne Case

Re: NEURON on Ranger

Posted: Fri Apr 29, 2011 9:42 am
by ted
Thanks for sharing this excellent information, Marianne!