NEURON on Ranger

NEURON on Ranger

Postby neuromau » Thu Apr 28, 2011 9:00 pm

Hello, Michael Hines installed NEURON on Ranger for me and so I'm posting the instructions here for other users of Ranger.

Ranger is a Linux cluster at the University of Texas which, according to the website (1), has "62,976 compute cores, 123 TB of total memory and 1.7 PB of raw global disk space." Everything's bigger in Texas, right? Ranger resources can be used by those not associated with UT by acquiring one of NSF's TeraGrid allocations (2).

Users with an account on Ranger have access to three filesystems: HOME, WORK, and SCRATCH. The setup below guides you through installing NEURON on the HOME filesystem, but jobs should be run out of WORK or SCRATCH.

To install NEURON on your Ranger account:
1. On Ranger, cd to your HOME directory:
Code: Select all
cdh


2. Create a directory named 'neuron':
Code: Select all
mkdir neuron


3. Enter the neuron directory:
Code: Select all
cd neuron


4. Put the desired nrn-nn.tar.gz file (either the current standard distribution (3) or a development version (4)) in the neuron directory. Note that you will not be installing Interviews, so iv-nn.tar.gz is not needed.

5. Unpack the file:
Code: Select all
tar xzf nrn-nn.tar.gz


6. Rename the unpacked directory:
Code: Select all
mv nrn-nn nrn


7. Create the nrninstall.sh script that will install NEURON on Ranger. This file should contain the following text:
Code: Select all
#!/bin/sh
../nrn/configure --prefix=`pwd` --with-nmodl-only --without-x
make
make install

../nrn/configure --prefix=`pwd` '--without-nmodl' '--without-x' \
'--without-memacs' '--with-paranrn' 'CC=mpicc' 'CXX=mpicxx' \
'--disable-shared' linux_nrnmech=no
make
make install


There are a few observations to make about this script. First, to paraphrase a previous comment by Ted & Michael (5), it is cross compiling NEURON because the login machine where you submit the job to run your model code is configured differently than the processors on which your model code will actually run. Second, this particular compilation does not include Python, though that is an option. Third, there are several other arguments that you probably didn't include when installing NEURON on your own machine; these arguments are nicely explained elsewhere on the forum (5).

8. In the neuron directory, create a second folder called 'mpi':
Code: Select all
mkdir mpi


This is the directory in which NEURON will be built. The nrn directory will be kept clean, in case you want to build other versions of NEURON on your Ranger account.

9. Enter the mpi directory:
Code: Select all
cd mpi


10. From the mpi directory, run the install script and write the output and errors to a log:
Code: Select all
sh ../nrninstall.sh > & installnotes.log


Testing the installation and running job scripts are covered in the reply to this post.

Links
1. Ranger User Guide: http://services.tacc.utexas.edu/index.php/ranger-user-guide
2. TeraGrid: http://www.teragrid.org
3. Standard Distribution: http://www.neuron.yale.edu/neuron/download/getstd
4. Development Versions: http://www.neuron.yale.edu/neuron/download/getdevel
5. Cross Compiling NEURON: http://www.neuron.yale.edu/phpBB/viewtopic.php?f=6&t=1171&start=0

~ Marianne Case
neuromau
 
Posts: 94
Joined: Mon Apr 20, 2009 7:02 pm

Re: NEURON on Ranger

Postby neuromau » Thu Apr 28, 2011 9:05 pm

neuromau wrote:Testing the installation and running job scripts are covered in the reply to this post.


To test the installation of NEURON, we can create the following test program.

1. Move to the WORK file system:
Code: Select all
cdw


2. Create a directory called test:
Code: Select all
mkdir test


3. Move to the test directory:
Code: Select all
cd test


4. In the test directory, create a file called test0.hoc, and include the following text in the file:
Code: Select all
objref pc
pc = new ParallelContext()
{printf("I am %d of %d\n", pc.id, pc.nhost)}
{pc.runworker()}
{pc.done()}
quit()


2. In the test directory, create a mod file called test.mod, and include the following text in the file:
Code: Select all
NEURON {SUFFIX nothing}


3. Compile the test mechanism:
Code: Select all
nrnivmodl


Now we want to run the test code. To do this, we must create a job script and submit the job script to the batch queue.

4. Create a file called mytest.sh and include the following lines:
Code: Select all
#!/bin/bash
#$-V
#$-cwd
#$-N myMPItest
#$-j y
#$ -o $JOB_NAME.o$JOB_ID
#$ -pe 16way 32
#$ -q development
#$ -l h_rt=00:05:00
#$ -M youremail@domain.edu
#$ -m e
set -x
ibrun x86_64/special -mpi test0.hoc


5. Now submit this script to the batch queue by entering the following at the command prompt:
Code: Select all
qsub mytest.sh


The batch queue software will perform a series of checks before assigning our job a run number, adding our job to the queue, and printing a confirmation of submission:
Your job 1926013 ("myMPItest") has been submitted


6. We can check on the status of our job using the following command:
Code: Select all
qstat

The Ranger User Guide (1) includes much more detail about submitting jobs, including the various options in the job script and other commands for checking the status of our job or the various queues.

7. Once the job is complete, an output file called myMPItest.o1926013 is generated. It concatenated the name we specified in our job script options with a run number generated by the batch queue. When we open this file, we should see some notes from the batch queue, a copy of the job script we submitted, and any output generated by our job (most of the output lines have been omitted in the example below):

TACC: Setting memory limits for job 1924223 to unlimited KB
TACC: Dumping job script:
--------------------------------------------------------------------------------
#!/bin/bash
#$-V
#$-cwd
#$-N myMPItest
#$-j y
#$ -o $JOB_NAME.o$JOB_ID
#$ -pe 16way 32
#$ -q development
#$ -l h_rt=00:05:00
#$ -M youremail@domain.edu
#$ -m e
set -x
ibrun x86_64/special -mpi test0.hoc

--------------------------------------------------------------------------------
TACC: Done.
+ ibrun x86_64/special -mpi test0.hoc
TACC: Starting up job 1924223
TACC: Setting up parallel environment for MVAPICH ssh-based mpirun.
TACC: Setup complete. Running job script.
TACC: starting parallel tasks...
numprocs=32
NEURON -- VERSION 7.2 (436:cbc0331180f8) 2010-04-05

Duke, Yale, and the BlueBrain Project -- Copyright 1984-2008
See http://www.neuron.yale.edu/credits.html

Additional mechanisms from files
test.mod
I am 13 of 32
I am 0 of 32
...
I am 20 of 32
I am 31 of 32
TACC: Shutting down parallel environment.
TACC: Shutdown complete. Exiting.
TACC: Cleaning up after job: 1924223
TACC: Done.


Note: If we wish to add any command line arguments (using Michael Hines' default_var function), we can add them to the ibrun line as follows:
Code: Select all
ibrun x86_64/special -mpi -c mytstop=20 test0.hoc


However, if we want to include a command line argument that uses quotes, we must put it in a different file type and have the ibrun reference that other file. In that case, the 'ibrun' line of our job script looks like:
Code: Select all
ibrun ./jobscripts/mytestrun.sh


And another file called mytestrun.sh must be created, which will actually contain the call to NEURON and the command line arguments:
Code: Select all
#!/bin/csh
x86_64/special -mpi -c 'strdef RunName' -c 'RunName="bigrun"' -c mytstop=20 test0.hoc


When we submit the job, we will still submit the main sh file, which will in turn reference the mytestrun.sh csh file

Links
1. Ranger User Guide: http://services.tacc.utexas.edu/index.p ... user-guide



~ Marianne Case
Last edited by neuromau on Wed May 18, 2011 7:35 pm, edited 4 times in total.
neuromau
 
Posts: 94
Joined: Mon Apr 20, 2009 7:02 pm

Re: NEURON on Ranger

Postby ted » Fri Apr 29, 2011 9:42 am

Thanks for sharing this excellent information, Marianne!
ted
Site Admin
 
Posts: 5040
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine


Return to UNIX/Linux

Who is online

Users browsing this forum: Google [Bot] and 1 guest