NEURON on Trestles

NEURON on Trestles

Postby neuromau » Tue Apr 02, 2013 5:19 pm

Trestles is a supercomputer provided by the San Diego Supercomputing Center. It has over 10,000 cores. There are multiple ways to gain access to Trestles. One is via the Neuroscience Gateway as detailed in the post Run parallel simulations via the Neuroscience Gateway. Another is via an allocation from NSF's XSEDE program. For information on obtaining an XSEDE allocation, visit https://portal.xsede.org/web/guest/how- ... allocation. Note that, if you use the Neuroscience Gateway, it has been designed to relieve the user of duties such as installing NEURON or dealing with jobscripts. Installing NEURON is only necessary if you access Trestles outside of the Neuroscience Gateway, such as with an XSEDE allocation, AND if you do not want to use the NEURON version already installed for the Neuroscience Gateway.

In general, it should be sufficient to just use the NSG's NEURON installation (even if you are running simulations on your own XSEDE allocation, independently of the NSG). The NSG installation is located at: [url]/projects/ps-nsg/home/nsguser/applications/neuron7.3/nrn-7.3/x86_64/bin/nrniv[/url]. To use it, simply add the following to one of your profile scripts that runs upon login, such as .bashrc:
Code: Select all
# load modules
module purge
module load gnubase
module load gnu
module load openmpi_ib
module load python
module load globus

# add NSG NEURON installation to your path
. /projects/ps-nsg/home/nsguser/applications/neuron7.3/nrnenv


To test, log out and log back in again. Then enter:
Code: Select all
which nrniv


and make sure the NSG installation of nrniv is returned. Then you should be good to go!

However, if you want to customize your install, read on for the installation procedure for Trestles.

To install NEURON on Trestles, follow these steps:

1. On Trestles, cd to your HOME directory:
Code: Select all
cd $HOME


2. Create a directory named 'neuron':
Code: Select all
mkdir neuron


3. Enter the neuron directory:
Code: Select all
cd neuron


4. Put the desired nrn-nn.tar.gz file (either the current standard distribution or a development version) in the neuron directory. Note that you will not be installing Interviews, so iv-nn.tar.gz is not needed.

5. Unpack the file:
Code: Select all
tar xzf nrn-nn.tar.gz


6. Rename the unpacked directory:
Code: Select all
mv nrn-nn nrn


7. Create the nrninstall.sh script that will install NEURON on Trestles. This file should contain the following text:
Code: Select all
module purge
module load gnu openmpi
../nrn/configure --prefix=`pwd` --with-paranrn --with-nrnpython --without-x
make
make install


There are several other arguments that you probably didn't include when installing NEURON on your own machine; these arguments are nicely explained elsewhere on the forum.

8. In the neuron directory, create a second folder called 'mpi':
Code: Select all
mkdir mpi


This is the directory in which NEURON will be built. The nrn directory will be kept clean, in case you want to build other versions of NEURON on your Trestles account.

9. Enter the mpi directory:
Code: Select all
cd mpi


10. From the mpi directory, run the install script and write the output and errors to a log:
Code: Select all
sh ../nrninstall.sh > & installnotes.log


Testing the installation and running job scripts are covered in the reply to this post.
Last edited by neuromau on Tue Apr 01, 2014 1:43 pm, edited 2 times in total.
neuromau
 
Posts: 90
Joined: Mon Apr 20, 2009 7:02 pm

Re: NEURON on trestles

Postby neuromau » Tue Apr 02, 2013 5:31 pm

Testing the installation and running job scripts are covered in the reply to this post.


To test the installation of NEURON, we can create the following test program.

1. Move to the HOME file system:
Code: Select all
cd $HOME


2. Create a directory called test:
Code: Select all
mkdir test


3. Move to the test directory:
Code: Select all
cd test


4. In the test directory, create a file called test0.hoc, and include the following text in the file:
Code: Select all
objref pc
pc = new ParallelContext()
{printf("I am %d of %d\n", pc.id, pc.nhost)}
{pc.runworker()}
{pc.done()}
quit()


5. In the test directory, create a mod file called test.mod, and include the following text in the file:
Code: Select all
NEURON {SUFFIX nothing}


6. Compile the test mechanism:
Code: Select all
nrnivmodl


7. To run NEURON jobs, we also need to customize our environment by adding the following lines to our $HOME/.bashrc file:
Code: Select all
module purge
module load gnu openmpi
export PATH=$HOME/neuron/mpi/x86_64/bin:$PATH


Now we want to run the test code. To do this, we must create a job script and submit the job script to the batch queue.

8. Create a file called mytest.sh and include the following lines:
Code: Select all
#!/bin/bash
#PBS -q normal
#PBS -A youraccount
#PBS -l nodes=1:ppn=32
#PBS -l walltime=00:01:00
#PBS -o ./myMPItest.o
#PBS -N myMPItest
#PBS -V
cd $PBS_O_WORKDIR
mpirun -np 32 -hostfile $PBS_NODEFILE  nrniv -mpi  test0.hoc


9. Now submit this script to the batch queue by entering the following at the command prompt:
Code: Select all
qsub mytest.sh


The batch queue software will return the job id as a confirmation of submission:
1356037.trestles-fe1.sdsc.edu


10. We can check on the status of our job using the following command:
Code: Select all
qstat -u yourusername


11. Once the job is complete, an output file called myMPItest.o is generated. It contains the text:
numprocs=32
I am 25 of 32
I am 30 of 32
I am 16 of 32
I am 17 of 32
I am 29 of 32
I am 27 of 32
I am 0 of 32
I am 15 of 32
I am 21 of 32
I am 5 of 32
I am 12 of 32
I am 1 of 32
I am 22 of 32
I am 8 of 32
I am 24 of 32
I am 19 of 32
I am 20 of 32
I am 18 of 32
I am 31 of 32
I am 13 of 32
I am 11 of 32
I am 28 of 32
I am 7 of 32
I am 2 of 32
I am 14 of 32
I am 6 of 32
I am 9 of 32
I am 3 of 32
I am 10 of 32
I am 23 of 32
I am 4 of 32
I am 26 of 32


Another file, called myMPItest.e1356037 (which concatenates our job name with the job id) is produced. This file contains the text:
Unloading compiler-dependent module mvapich2/1.5.1p1
Unloading compiler-dependent module openmpi/1.4.3
Unloading compiler-dependent module mvapich2/1.5.1p1
NEURON -- Release 7.3 (842:0ac0ad194d99) 2013-04-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2013
See http://www.neuron.yale.edu/neuron/credits

Additional mechanisms from files
test.mod
Unloading compiler-dependent module mvapich2/1.5.1p1
neuromau
 
Posts: 90
Joined: Mon Apr 20, 2009 7:02 pm


Return to UNIX/Linux

Who is online

Users browsing this forum: No registered users and 1 guest