Compiling for Blue Gene?

mhanby · Post by **mhanby** » Mon Jul 16, 2007 3:12 pm

Howdy, I notice on the web site that NEURON will run on an IBM Blue Gene.

Does anyone have any specific instructions for compiling NEURON for the Blue Gene?

Thanks, Mike

Post by **hines** » Mon Jul 16, 2007 3:27 pm

Put the sources from the nrn*.tar.gz file in $HOME/neuron/nrn
If you want to start from the svn repository then you will have to install
at least automake 1.9.6, autoconf 2.59,
and libtool 1.5.22 and in the nrn directory type 'sh build.sh'
Then create a $HOME/neuron/nrnmpi directory, move to there and type

Code: Select all

../nrn/configure --prefix=`pwd` --srcdir=../nrn --enable-bluegene --with-nmodl-only
make
make install
../nrn/configure --prefix=`pwd` --srcdir=../nrn --enable-bluegene --with-paranrn
make
make install

Look in nrn/configure.in to see what --enable-bluegene is doing.

mhanby · Post by **mhanby** » Mon Jul 16, 2007 4:31 pm

Thanks for the quick reply.

I attempted this using the nrn-6.0.tar.gz file and: sh build.sh errors at some point with:
extend_depcomp.sh: extend_depcomp.sh: No such file or directory

The file doesn't exist anywhere under the nrn-6.0 directory.

I'm going to try the SVN version, however, my system doesn't quite meet the requirements you specified (and YaST doesn't list any later versions available):

automake 1.8.3-23.1 <-- NO --> 1.9.6
autoconf 2.59-75.1 <-- YES --> 2.59
libtool 1.5.2-56.1 <-- NO --> 1.5.22

Thanks for any suggestions, Mike

mhanby · Post by **mhanby** » Mon Jul 16, 2007 5:28 pm

Update: I ran the sh build.sh and ../nrn/configure commands on a system (RHEL5) with the correct versions of automake/autoconf/libtool, and it still produces the same Makefile (CC = gcc, etc...) even though I specify --enable-bluegene on the configure line

I downloaded the latest SVN and now the sh build.sh completes, however when I run the ../nrn/configure step, the Makefile it generates doesn't contain any of the blrts_xlc and related settings (BGLSYS, CFLAGS, ...).

I'm guessing this has to do with the fact that libtools and automake aren't the correct version?

Unfortunately, this system is SLES 9 and I'm not sure that I can upgrade the automake / libtools beyond what is already installed. YaST says that's the latest and greatest available. SLES 10 might be the answer, however, I don't know that a) I have the media available and b) that SLES 10 is an option for the Blue Gene frontend and service nodes.

If I manually modify ./nrnmpi/Makefile using the settings under bluegene in ../nrn/configure.in before I run the make / make install command, will that work?

Thanks, Mike

mhanby · Post by **mhanby** » Tue Jul 17, 2007 1:37 pm

I believe I have compiled the first portion as it created ./nrnmpi/powerpc64/bin
hoc_e
modlunit
nocmodl

and the file command seems to indicate these are BGL binaries:
file ./powerpc64/bin/nocmodl
./powerpc64/bin/nocmodl: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (embedded), statically linked, not stripped

Here are the steps I followed

1. Get the latest from SVN

# mkdir /gpfs_data/apps/neuron
# mkdir /gpfs_data/apps/neuron/nrnmpi
# cd /gpfs_data/apps/neuron
# svn checkout http://www.neuron.yale.edu/svn/neuron/nrn/trunk nrn
2. Run the build script

# cd ./nrn
# sh build.sh
3. Configure the compiler variables

export NSRC=/gpfs_data/apps/neuron/nrn
export BGLSYS=/bgl/BlueLight/ppcfloor/bglsys
export CC=`which blrts_xlc`
export CXX=`which blrts_xlc++`
export MPICC=$CC
export MPICXX=$CXX
export OPTFLAGS="-qarch=440d -qtune=440 -O3 -qstrict -qhot"
export CFLAGS="$OPTFLAGS -g -I$BGLSYS/include"
export CXXFLAGS=$CFLAGS
export BGL_LIBS="-lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts -ldevices.rts -L/opt/ibmcmp/xlmass/bg/4.3/blrts_lib -lmass"
export LIBS="-L$BGLSYS/lib $BGL_LIBS"

4. Run the make commands

# cd ../nrnmpi
# ../nrn/configure --prefix=`pwd` --srcdir=../nrn --enable-bluegene --with-nmodl-only
# make
# make install
# ../nrn/configure --prefix=`pwd` --srcdir=../nrn --enable-bluegene --with-paranrn
# make
# make install

mhanby · Post by **mhanby** » Tue Jul 17, 2007 1:40 pm

During configure, I noticed the following warning:
do SIGPOLL and I_GETSIG work: no
But see config.log to see why it failed.

However, config.log doesn't clarify the message.

When I attempt to run make after the second configure, it runs for several minutes, and then errors with:

/opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlc -DHAVE_CONFIG_H -I. -I../../../nrn/src/oc -I../.. -I../.. -I../.. -I. -I. -I../../src/parallel -I../../src/nrnjava -I../../src/nrncvode -I../../src/ivos -I../../src/sundials -I../../src/nrnpython -I../../../nrn/src/memacs -I../../../nrn/src/sparse -DOOP=1 -DINTERVIEWS=1 -DHOC=1 -qarch=440d -qtune=440 -O3 -qstrict -qhot -g -I/bgl/BlueLight/ppcfloor/bglsys/include -c ../../../nrn/src/oc/ockludge.c
/bin/sh ../../libtool --mode=link /opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlc -qarch=440d -qtune=440 -O3 -qstrict -qhot -g -I/bgl/BlueLight/ppcfloor/bglsys/include -o oc ocmain.o ocnoiv.o nocable.o modlreg.o ockludge.o libocxt.la liboc.la ../nrnmpi/libnrnmpi.la -L/bgl/BlueLight/ppcfloor/bglsys/lib -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts -ldevices.rts -L/opt/ibmcmp/xlmass/bg/4.3/blrts_lib -lmass -lm
/opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlc -qarch=440d -qtune=440 -O3 -qstrict -qhot -g -I/bgl/BlueLight/ppcfloor/bglsys/include -o oc ocmain.o ocnoiv.o nocable.o modlreg.o ockludge.o ./.libs/libocxt.a -L/bgl/BlueLight/ppcfloor/bglsys/lib -L/opt/ibmcmp/xlmass/bg/4.3/blrts_lib ./.libs/liboc.a ../nrnmpi/.libs/libnrnmpi.a -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts -lmass -lm
make[4]: Leaving directory `/gpfs_data/apps/neuron/nrnmpi/src/oc'
make[3]: Leaving directory `/gpfs_data/apps/neuron/nrnmpi/src/oc'
Making all in nrnoc
make[3]: Entering directory `/gpfs_data/apps/neuron/nrnmpi/src/nrnoc'
(MODLUNIT=../../../nrn/share/lib/nrnunits.lib ; \
export MODLUNIT ; ../nmodl/nocmodl ../../../nrn/src/nrnoc/hh )
out of memory
make[3]: *** [hh.c] Error 1
make[3]: Leaving directory `/gpfs_data/apps/neuron/nrnmpi/src/nrnoc'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/gpfs_data/apps/neuron/nrnmpi/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/gpfs_data/apps/neuron/nrnmpi'
make: *** [all] Error 2

The system has 6 GB's of free memory.

mhanby · Post by **mhanby** » Wed Jul 18, 2007 10:49 am

Ok, I think I've figured out where the out of memory is coming from.

./nrnmpi/src/nmodl/nocmodl

This binary is a BlueGene compiled binary, and the make is attempting to run it locally, which would require nocmodl to be a PowerPC64 compiled binary.

(MODLUNIT=../../../nrn/share/lib/nrnunits.lib ; \
export MODLUNIT ; ../nmodl/nocmodl ../../../nrn/src/nrnoc/hh )
out of memory

mhanby · Post by **mhanby** » Thu Jul 19, 2007 12:18 pm

I believe I have it compiled now. The confusion had to do with the first configure/make step.

This step isn't supposed to use the cross compilers (blrts_xlc). I just assumed that it was supposed to and I was forcing it by setting all of the CC etc variables manually.

The second configure/make does use the cross compilers.

Here are my revised steps (now I just need the user who knows how to run neuron to try the binaries out).

1. Get the latest from SVN

# mkdir /gpfs_data/apps/neuron
# mkdir /gpfs_data/apps/neuron/nrnmpi
# cd /gpfs_data/apps/neuron
# svn checkout http://www.neuron.yale.edu/svn/neuron/nrn/trunk nrn
2. Run the build script

# cd ./nrn
# sh build.sh

3. Run the make commands

# cd ../nrnmpi

You'll notice that this step uses the normal gcc compilers and tools
# ../nrn/configure --prefix=`pwd` --srcdir=../nrn --enable-bluegene --with-nmodl-only
# make
# make install

This step will use the blrts_xlc and related tools
# ../nrn/configure --prefix=`pwd` --srcdir=../nrn --enable-bluegene --with-paranrn
# make
# make install

I didn't update autoconf/automake/libtool from the versions installed / patched in SLES9 update 3.