NEURON on Stampede

Post Reply
neuromau
Posts: 97
Joined: Mon Apr 20, 2009 7:02 pm

NEURON on Stampede

Post by neuromau »

Stampede is a supercomputer provided by the University of Texas. It has over 6,400 16-core nodes. Over 90% of Stampede resources are dedicated to XSEDE, NSF's mechanism for providing supercomputing resources to researchers. For information on obtaining an XSEDE allocation, visit their new allocation page.

*** Note: There is an issue on Stampede, that I have seen when using 80 or more processors in a parallel job, that it doesn't respect the quit() command in the standard release of NEURON 7.3. This means your job has the potential to run for the full length of time specified by your hard time limit in your job submission script. To prevent this from happening, install a development version of the NEURON code, version 1029:91f44d14afd5 or later, and to terminate your model code, do not use "pc.runworker()" and "pc.done" but instead call:

Code: Select all

pc.barrier()
quit()
***[/b][/color]

To install NEURON on your Stampede account:
1. On Stampede, cd to your HOME directory:

Code: Select all

cdh
2. Create a directory named 'neuron':

Code: Select all

mkdir neuron
3. Enter the neuron directory:

Code: Select all

cd neuron
4. Get either (the current standard distribution nrn-nn.tar.gz file) or clone a development version (remember to run the build.sh command!) in the neuron directory. Note that you will not be installing Interviews, so iv-nn.tar.gz is not needed. If you get the tar.gz file, then unpack it and rename the unpacked directory:

Code: Select all

tar xzf nrn-nn.tar.gz
mv nrn-nn nrn
5. In the neuron directory, create the nrninstall.sh script that will install NEURON on Stampede. This file should contain the following text:

Code: Select all

#!/bin/sh
../nrn/configure --prefix=`pwd` --with-nmodl-only --without-x
make
make install

../nrn/configure --prefix=`pwd` '--without-nmodl' '--without-x' \
'--without-memacs' '--with-paranrn' 'CC=mpicc' 'CXX=mpicxx' \
'--disable-shared' 'CFLAGS=-g -O0' 'CXXFLAGS=-g -O0'  linux_nrnmech=no
make
make install
There are a few observations to make about this script. First, to paraphrase a previous comment by Ted & Michael, it is cross compiling NEURON because the login machine where you submit the job to run your model code is configured differently than the processors on which your model code will actually run. Second, this particular compilation does not include Python, though that is an option. Third, there are several other arguments that you probably didn't include when installing NEURON on your own machine; these arguments are nicely explained elsewhere on the forum.

6. Within the neuron directory, create a second folder called 'mpi':

Code: Select all

mkdir mpi
This is the directory in which the executable NEURON will be built. The nrn directory will be kept clean, in case you want to build other versions of NEURON on your Stampede account.

7. Enter the mpi directory:

Code: Select all

cd mpi
8. From the mpi directory, run the install script you wrote in step 5 and write the output and errors to a log:

Code: Select all

sh ../nrninstall.sh > & installnotes.log
9. Ensure that your PATH variable is updated to include the path to the executable NEURON. You can add the following line to your profile setup files (depending on your set up: ".profile" or ".profile_user", ".bashrc", ".login" or ".login_user", ".cshrc") in the $HOME directory of your Stampede account:

Code: Select all

export PATH=path_to_your_neuron_directory/mpi/x86_64/bin:$PATH
10. Log out and back into Stampede and check your $PATH variable to ensure it includes the path you just added to your setup files, and then that the nrniv program can be found:

Code: Select all

echo $PATH
which nrniv
Testing the installation and running job scripts are covered in the reply to this post.
Last edited by neuromau on Thu Mar 06, 2014 1:31 pm, edited 1 time in total.
neuromau
Posts: 97
Joined: Mon Apr 20, 2009 7:02 pm

Re: NEURON on Stampede

Post by neuromau »

neuromau wrote:Testing the installation and running job scripts are covered in the reply to this post.
To test the installation of NEURON, we can create the following test program.

1. Move to the WORK file system:

Code: Select all

 cdw
2. Create a directory called test:

Code: Select all

mkdir test
3. Move to the test directory:

Code: Select all

cd test
4. In the test directory, create a file called test0.hoc (for example,

Code: Select all

vi test0.hoc
), and include the following text in the file:

Code: Select all

objref pc
pc = new ParallelContext()
{printf("I am %d of %d\n", pc.id, pc.nhost)}
{pc.barrier()}
quit()
Note that, unlike with parallel code on other supercomputers, we are ending our program with different commands. Stampede may hang on the "pc.runworker" call, so we go straight to a "quit()" call instead (which will work if you have installed the latest development version of NEURON as advised in the post above). However, if one processor calls the "quit()" command before the other processors have reached that point in the code, unexpected behavior may result. Therefore, prior to the "quit()" call, we have added a "pc.barrier()" call to ensure all processors have reached the point where they are ready to quit.

5. In the test directory, create a mod file called test.mod, and include the following text in the file:

Code: Select all

NEURON {SUFFIX nothing}
6. Compile the test mechanism:

Code: Select all

nrnivmodl
7. Create a job script called mytest.sh that contains the following code:

Code: Select all

#!/bin/bash
#SBATCH -J myMPItest # name of job
#SBATCH -o myMPItest.%j.o # output file (errors can be placed in this file, too). %j expands to the job id number.
#SBATCH -n 16 # number of cores 
#SBATCH -p development # queue to use
#SBATCH -t 00:10:00 # time limit
#SBATCH --mail-user=yourname@school.edu # where to send notifications
#SBATCH --mail-type=ALL # types of mail notifications to receive

ibrun tacc_affinity  x86_64/special -mpi   test0.hoc
Details about the job script format are included in the Stampede User Guide. Note the inclusion of 'tacc_affinity' in the ibrun command. Without this option, the '-mpi' flag may not work.

8. After creating this job script, we must submit it. To submit it to the queue, we enter:
sbatch mytest.sh
The batch queue software will perform a series of checks before assigning our job a run number, adding our job to the queue, and printing a confirmation of submission:
Submitted batch job 461814
9. We can check on the status of our job using the following command:

Code: Select all

squeue -u your_user_name
Note that the Stampede User Guide also recommends the showq function, but this function is currently buggy and may return other users' jobs as well. Therefore, squeue is preferred.

10. Once the job is complete, an output file called myMPItest.461814.o is generated. Note that we specified the name of the output file in our submission script, including %j where the job id number generated by the queuing system was inserted. When we open this file, we see the following:
+ ibrun tacc_affinity x86_64/special -mpi test0.hoc
TACC: Starting up job 461814
TACC: Starting parallel tasks...
numprocs=16
NEURON -- VERSION 7.3 (817:7da2b70c0aed) 2013-03-20
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2012
See http://www.neuron.yale.edu/credits.html

Additional mechanisms from files
test.mod
I am 10 of 16
I am 3 of 16
I am 5 of 16
I am 0 of 16
I am 1 of 16
I am 4 of 16
I am 6 of 16
I am 7 of 16
I am 14 of 16
I am 13 of 16
I am 8 of 16
I am 11 of 16
I am 9 of 16
I am 2 of 16
I am 15 of 16
I am 12 of 16

TACC: Shutdown complete. Exiting.
neuromau
Posts: 97
Joined: Mon Apr 20, 2009 7:02 pm

Re: NEURON on Stampede

Post by neuromau »

Occasionally you may want to make changes to the NEURON source code. If this becomes necessary for your static build installation on Stampede, you can make changes by following these steps from Michael:

1. Make the desired changes to the file(s) within your neuron/nrn/src directory.

2. Change directories to neuron/mpi/src:

Code: Select all

cd ../../mpi/src
3. Rerun the installation in the mpi/src directory (in some cases, you can run it just in the subdirectory equivalent to where the you altered the file in the nrn/src directory):

Code: Select all

make install
4. Change directories to your model code location, in our test example from above that would be:

Code: Select all

cdw
cd test
5. Completely remove the compiled mechanism folder and then rerun nrnivmodl to link the modified *.a file generated when you ran "make install". If you do not completely remove the mechanism folder, running nrnivmodl won't actually cause the modifications from "make install" to be linked:

Code: Select all

rm -r x86_64/
nrnivmodl
Post Reply