www.neuron.yale.edu • NEURON on Trestles

Page 1 of 1

NEURON on Trestles

Posted: Tue Apr 02, 2013 5:19 pm

by neuromau

Trestles is a supercomputer provided by the San Diego Supercomputing Center. It has over 10,000 cores. There are multiple ways to gain access to Trestles. One is via the Neuroscience Gateway as detailed in the post Run parallel simulations via the Neuroscience Gateway. Another is via an allocation from NSF's XSEDE program. For information on obtaining an XSEDE allocation, visit https://portal.xsede.org/web/guest/how- ... allocation. Note that, if you use the Neuroscience Gateway, it has been designed to relieve the user of duties such as installing NEURON or dealing with jobscripts. Installing NEURON is only necessary if you access Trestles outside of the Neuroscience Gateway, such as with an XSEDE allocation, AND if you do not want to use the NEURON version already installed for the Neuroscience Gateway.

In general, it should be sufficient to just use the NSG's NEURON installation (even if you are running simulations on your own XSEDE allocation, independently of the NSG). The NSG installation is located at: /projects/ps-nsg/home/nsguser/applicati ... /bin/nrniv. To use it, simply add the following to one of your profile scripts that runs upon login, such as .bashrc:

Code: Select all

# load modules
module purge
module load gnubase
module load gnu
module load openmpi_ib
module load python
module load globus

# add NSG NEURON installation to your path
. /projects/ps-nsg/home/nsguser/applications/neuron7.3/nrnenv

To test, log out and log back in again. Then enter:

Code: Select all

which nrniv

and make sure the NSG installation of nrniv is returned. Then you should be good to go!

However, if you want to customize your install, read on for the installation procedure for Trestles.

To install NEURON on Trestles, follow these steps:

1. On Trestles, cd to your HOME directory:

Code: Select all

cd $HOME

2. Create a directory named 'neuron':

Code: Select all

mkdir neuron

3. Enter the neuron directory:

Code: Select all

cd neuron

4. Put the desired nrn-nn.tar.gz file (either the current standard distribution or a development version) in the neuron directory. Note that you will not be installing Interviews, so iv-nn.tar.gz is not needed.

5. Unpack the file:

Code: Select all

tar xzf nrn-nn.tar.gz

6. Rename the unpacked directory:

Code: Select all

mv nrn-nn nrn

7. Create the nrninstall.sh script that will install NEURON on Trestles. This file should contain the following text:

Code: Select all

module purge
module load gnu openmpi
../nrn/configure --prefix=`pwd` --with-paranrn --with-nrnpython --without-x
make
make install

There are several other arguments that you probably didn't include when installing NEURON on your own machine; these arguments are nicely explained elsewhere on the forum.

8. In the neuron directory, create a second folder called 'mpi':

Code: Select all

mkdir mpi

This is the directory in which NEURON will be built. The nrn directory will be kept clean, in case you want to build other versions of NEURON on your Trestles account.

9. Enter the mpi directory:

Code: Select all

cd mpi

10. From the mpi directory, run the install script and write the output and errors to a log:

Code: Select all

sh ../nrninstall.sh > & installnotes.log

Testing the installation and running job scripts are covered in the reply to this post.

Re: NEURON on trestles

Posted: Tue Apr 02, 2013 5:31 pm

by neuromau

Testing the installation and running job scripts are covered in the reply to this post.

To test the installation of NEURON, we can create the following test program.

1. Move to the HOME file system:

Code: Select all

cd $HOME

2. Create a directory called test:

Code: Select all

mkdir test

3. Move to the test directory:

Code: Select all

cd test

4. In the test directory, create a file called test0.hoc, and include the following text in the file:

Code: Select all

objref pc
pc = new ParallelContext()
{printf("I am %d of %d\n", pc.id, pc.nhost)}
{pc.runworker()}
{pc.done()}
quit()

5. In the test directory, create a mod file called test.mod, and include the following text in the file:

Code: Select all

NEURON {SUFFIX nothing}

6. Compile the test mechanism:

Code: Select all

nrnivmodl

7. To run NEURON jobs, we also need to customize our environment by adding the following lines to our $HOME/.bashrc file:

Code: Select all

module purge
module load gnu openmpi
export PATH=$HOME/neuron/mpi/x86_64/bin:$PATH

Now we want to run the test code. To do this, we must create a job script and submit the job script to the batch queue.

8. Create a file called mytest.sh and include the following lines:

Code: Select all

#!/bin/bash
#PBS -q normal
#PBS -A youraccount
#PBS -l nodes=1:ppn=32
#PBS -l walltime=00:01:00
#PBS -o ./myMPItest.o
#PBS -N myMPItest
#PBS -V
cd $PBS_O_WORKDIR
mpirun -np 32 -hostfile $PBS_NODEFILE  nrniv -mpi  test0.hoc

9. Now submit this script to the batch queue by entering the following at the command prompt:

Code: Select all

qsub mytest.sh

The batch queue software will return the job id as a confirmation of submission:

1356037.trestles-fe1.sdsc.edu

10. We can check on the status of our job using the following command:

Code: Select all

qstat -u yourusername

11. Once the job is complete, an output file called myMPItest.o is generated. It contains the text:

numprocs=32
I am 25 of 32
I am 30 of 32
I am 16 of 32
I am 17 of 32
I am 29 of 32
I am 27 of 32
I am 0 of 32
I am 15 of 32
I am 21 of 32
I am 5 of 32
I am 12 of 32
I am 1 of 32
I am 22 of 32
I am 8 of 32
I am 24 of 32
I am 19 of 32
I am 20 of 32
I am 18 of 32
I am 31 of 32
I am 13 of 32
I am 11 of 32
I am 28 of 32
I am 7 of 32
I am 2 of 32
I am 14 of 32
I am 6 of 32
I am 9 of 32
I am 3 of 32
I am 10 of 32
I am 23 of 32
I am 4 of 32
I am 26 of 32

Another file, called myMPItest.e1356037 (which concatenates our job name with the job id) is produced. This file contains the text:

Unloading compiler-dependent module mvapich2/1.5.1p1
Unloading compiler-dependent module openmpi/1.4.3
Unloading compiler-dependent module mvapich2/1.5.1p1
NEURON -- Release 7.3 (842:0ac0ad194d99) 2013-04-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2013
See http://www.neuron.yale.edu/neuron/credits

Additional mechanisms from files
test.mod
Unloading compiler-dependent module mvapich2/1.5.1p1