NEURON on Stampede
Posted: Mon Mar 25, 2013 6:38 pm
Stampede is a supercomputer provided by the University of Texas. It has over 6,400 16-core nodes. Over 90% of Stampede resources are dedicated to XSEDE, NSF's mechanism for providing supercomputing resources to researchers. For information on obtaining an XSEDE allocation, visit their new allocation page.
*** Note: There is an issue on Stampede, that I have seen when using 80 or more processors in a parallel job, that it doesn't respect the quit() command in the standard release of NEURON 7.3. This means your job has the potential to run for the full length of time specified by your hard time limit in your job submission script. To prevent this from happening, install a development version of the NEURON code, version 1029:91f44d14afd5 or later, and to terminate your model code, do not use "pc.runworker()" and "pc.done" but instead call:
***[/b][/color]
To install NEURON on your Stampede account:
1. On Stampede, cd to your HOME directory:
2. Create a directory named 'neuron':
3. Enter the neuron directory:
4. Get either (the current standard distribution nrn-nn.tar.gz file) or clone a development version (remember to run the build.sh command!) in the neuron directory. Note that you will not be installing Interviews, so iv-nn.tar.gz is not needed. If you get the tar.gz file, then unpack it and rename the unpacked directory:
5. In the neuron directory, create the nrninstall.sh script that will install NEURON on Stampede. This file should contain the following text:
There are a few observations to make about this script. First, to paraphrase a previous comment by Ted & Michael, it is cross compiling NEURON because the login machine where you submit the job to run your model code is configured differently than the processors on which your model code will actually run. Second, this particular compilation does not include Python, though that is an option. Third, there are several other arguments that you probably didn't include when installing NEURON on your own machine; these arguments are nicely explained elsewhere on the forum.
6. Within the neuron directory, create a second folder called 'mpi':
This is the directory in which the executable NEURON will be built. The nrn directory will be kept clean, in case you want to build other versions of NEURON on your Stampede account.
7. Enter the mpi directory:
8. From the mpi directory, run the install script you wrote in step 5 and write the output and errors to a log:
9. Ensure that your PATH variable is updated to include the path to the executable NEURON. You can add the following line to your profile setup files (depending on your set up: ".profile" or ".profile_user", ".bashrc", ".login" or ".login_user", ".cshrc") in the $HOME directory of your Stampede account:
10. Log out and back into Stampede and check your $PATH variable to ensure it includes the path you just added to your setup files, and then that the nrniv program can be found:
Testing the installation and running job scripts are covered in the reply to this post.
*** Note: There is an issue on Stampede, that I have seen when using 80 or more processors in a parallel job, that it doesn't respect the quit() command in the standard release of NEURON 7.3. This means your job has the potential to run for the full length of time specified by your hard time limit in your job submission script. To prevent this from happening, install a development version of the NEURON code, version 1029:91f44d14afd5 or later, and to terminate your model code, do not use "pc.runworker()" and "pc.done" but instead call:
Code: Select all
pc.barrier()
quit()
To install NEURON on your Stampede account:
1. On Stampede, cd to your HOME directory:
Code: Select all
cdh
Code: Select all
mkdir neuron
Code: Select all
cd neuron
Code: Select all
tar xzf nrn-nn.tar.gz
mv nrn-nn nrn
Code: Select all
#!/bin/sh
../nrn/configure --prefix=`pwd` --with-nmodl-only --without-x
make
make install
../nrn/configure --prefix=`pwd` '--without-nmodl' '--without-x' \
'--without-memacs' '--with-paranrn' 'CC=mpicc' 'CXX=mpicxx' \
'--disable-shared' 'CFLAGS=-g -O0' 'CXXFLAGS=-g -O0' linux_nrnmech=no
make
make install
6. Within the neuron directory, create a second folder called 'mpi':
Code: Select all
mkdir mpi
7. Enter the mpi directory:
Code: Select all
cd mpi
Code: Select all
sh ../nrninstall.sh > & installnotes.log
Code: Select all
export PATH=path_to_your_neuron_directory/mpi/x86_64/bin:$PATH
Code: Select all
echo $PATH
which nrniv