[SOLVED] Problems compiling MODL on Cray XC40

[silvan]

Hi

I've been trying to compile and run NEURON on Cray XC40. After following the instructions from viewtopic.php?p=8803&sid=72065e087ffc59 ... 5f7a#p8803 I'm stuck.

I should probably add that the login node (where I compile stuff) have Sandy Bridge processors while compute nodes run on Haswell architecture.
I've compiled nmodl with flags for Sandy Bridge and added -fPIC after it complained that it needs it.
Then I've compiled NEURON itself (without nmodl) with flags for Haswell and -fPIC again.

I've tried testing it on the networks from "Translating network models to parallel hardware in NEURON". The ring network works fine. However, I've encountered some problems with the random connectivity network.

1. When I run nrnivmodl (having loaded the craype-haswell module, to make it compile with flags for compute node), it compiles, creates the necessary x86_64/.libs folder but for some reason creates only static libs, not the shared ones. I can convert them using the compiler, so it's not that much of a problem.

2. When I run the simulation (using SLURM), I get only a "Segmentation fault" without any further information.

The model runs on my home computer so it's not likely a typo. I'm really new to running NEURON in parallel, so I would really appreciate some help with debugging this.

Thank you.

[pkumbhar]

Hi Silvan,

I have tested this on Titan XK7 system at ORNL and see the issue with static/shared libraries when we invoke nrnivmodl. You have to add "linux_nrnmech=no" in configure line. My installation script for Titan system looks like below:

Code: Select all

module swap craype-mc8 craype-interlagos
module unload PrgEnv-pgi
module load PrgEnv-cray
export CC=cc
export CXX=CC
export MPICC=cc
export CXX=CC

MY_HOME=/ccs/home/kumbhar/workarena/systems/titan/repos/ext/MY_INSTALL
INSTALL_DIR=/lustre/atlas/scratch/kumbhar/csc192/install

cd $MY_HOME/neuron
cd nrn && ./build.sh        #here I have cloned neuron mercurial repository

cd $MY_HOME/neuron       
mkdir -p nrnmpi
cd nrnmpi

#target-cpu flag added for login node
../nrn/configure --prefix=$INSTALL_DIR --without-x --with-nmodl-only CFLAGS="-target-cpu=mc8" CXXFLAGS="-target-cpu=mc8"
make
make install

#no specific flags, cc and CC are for compute node
../nrn/configure --prefix=$INSTALL_DIR --without-x --with-paranrn --without-nmodl --disable-shared  linux_nrnmech=no
make
make -j12 install

With "linux_nrnmech=no", now when you invoke nrnivmodl, the special will be built as a statically linked executable (similar to BG-Q).

Now about the segfaul at runtime:

This is bit tricky and difficult to guess. I did run Traub's model on Titan with NEURON and cray compiler without any issues.
As you know, the login nodes and compute nodes have different processor architecture and mismatch in compiler flags usually throw "Illegal Instruction" error (which is not the case in your example).

Could you re-install latest mercurial version of NEURON (using instructions above) and try again?

[silvan]

EDIT: I edited the post so that it contains a solution to the problem

Pramod,

Thank you for your reply. The server was down for a few days now. Today I were able to test your solution. My compile script looked like that

Code: Select all

module swap craype-haswell craype-sandybridge
set MY_HOME=/lustre/tetyda/home/silvan/MY_INSTALL
set INSTALL_DIR=/lustre/tetyda/home/silvan/neuron
cd $MY_HOME/neuron
cd nrn && ./build.sh
cd $MY_HOME/neuron
mkdir -p nrnmpi
cd nrnmpi
make clean
../nrn/configure --prefix=$INSTALL_DIR --without-x --with-nmodl-only CFLAGS="-target-cpu=corei7-avx" CXXFLAGS="-target-cpu=corei7-avx"
make
make install

../nrn/configure --prefix=$INSTALL_DIR --without-x --with-paranrn --without-nmodl --disable-shared --with-mpi --without-memacs --without-readline linux_nrnmech=no CC=cc CXX=CC
module swap craype-sandybridge craype-haswell
make
make -j12 install

and it did compile (the mercurial version) using cray compiler, you were right. Right now, when I invoke nrnivmodl (with PrgEnv-cray and craype-haswell module loaded), it creates a binary special file in the x86_64 subdirectory.

My sbatch file is

Code: Select all

#!/bin/bash
#SBATCH -J rand_network_neuron
#SBATCH -N 1
#SBATCH --ntasks-per-node 12
#SBATCH --mem 100
#SBATCH --time=00:02:00

mpiexec -n 12 x86_64/special -mpi random_parallel.hoc

And it works. :)