How to execute CoreNEURON with GPU?

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
john.gwak
Posts: 5
Joined: Thu Jul 26, 2018 4:04 am

How to execute CoreNEURON with GPU?

Post by john.gwak » Fri Jul 27, 2018 12:20 am

I tried a ringtest example from GitHub below.
https://github.com/nrnhines/ringtest

I could do all the steps before 'Building CoreNEURON with GPU support'.
When I tried GPU part, I have some problems...

There was an error while doing cmake as shown below.

Code: Select all

module purge
module load pgi/18.4 openmpi/2.1.2/2018
module load cuda/9.0
export CC=mpicc
export CXX=mpicxx

cd $SOURCE_DIR/ringtest
mkdir -p coreneuron_x86_gpu && cd coreneuron_x86_gpu
cmake $BASE_DIR/sources/CoreNeuron -DADDITIONAL_MECHPATH=$SOURCE_DIR/ringtest/mod -DCMAKE_C_FLAGS:STRING="-O2" -DCMAKE_CXX_FLAGS:STRING="-O2" -DCOMPILE_LIBRARY_TYPE=STATIC -DCUDA_HOST_COMPILER=`which gcc` -DCUDA_PROPAGATE_HOST_FLAGS=OFF -DENABLE_SELECTIVE_GPU_PROFILING=ON -DENABLE_OPENACC=ON

Code: Select all

-- Found MPI_C: /usr/pgi/linux86-64/2018/mpi/openmpi-2.1.2/bin/mpicc (found version "3.1")
-- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS)
CMake Error at /usr/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find MPI (missing: MPI_CXX_FOUND) (found version "3.1")
Call Stack (most recent call first):
  /usr/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.11/Modules/FindMPI.cmake:1663 (find_package_handle_standard_args)
  CMakeLists.txt:97 (find_package)
So, I tried to not use openmpi provided with pgi package and tried to use openmpi installed from apt-get (3.1.1.real-4)
I could solve the above problem, but, make generated below errors.

Code: Select all

module purge
module load pgi/18.4
module load cuda/9.0
export CC=mpicc
export CXX=mpicxx

cd $SOURCE_DIR/ringtest
mkdir -p coreneuron_x86_gpu && cd coreneuron_x86_gpu
cmake $BASE_DIR/sources/CoreNeuron -DADDITIONAL_MECHPATH=$SOURCE_DIR/ringtest/mod -DCMAKE_C_FLAGS:STRING="-O2" -DCMAKE_CXX_FLAGS:STRING="-O2" -DCOMPILE_LIBRARY_TYPE=STATIC -DCUDA_HOST_COMPILER=`which gcc` -DCUDA_PROPAGATE_HOST_FLAGS=OFF -DENABLE_SELECTIVE_GPU_PROFILING=ON -DENABLE_OPENACC=ON
make VERBOSE=1 -j

Code: Select all

Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: "Release"
-- Installing: /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/external/mod2c/share/nrnunits.lib
-- Installing: /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/external/mod2c/bin/mod2c_core
make[3]: Leaving directory '/home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/mod2c-prefix/src/mod2c-build'
cd /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/mod2c-prefix/src/mod2c-build && /usr/bin/cmake -E touch /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/mod2c-prefix/src/mod2c-stamp/mod2c-install
[ 10%] Completed 'mod2c'
/usr/bin/cmake -E make_directory /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/CMakeFiles
/usr/bin/cmake -E touch /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/CMakeFiles/mod2c-complete
/usr/bin/cmake -E touch /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/mod2c-prefix/src/mod2c-stamp/mod2c-done
make[2]: Leaving directory '/home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu'
[ 10%] Built target mod2c
make[1]: Leaving directory '/home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu'
make: *** [Makefile:141: all] Error 2
tried again...

Code: Select all

nvcc fatal   : Value 'sm_20' is not defined for option 'gpu-architecture'
CMake Error at cudacoreneuron_generated_cuda_profile.cu.o.RelWithDebInfo.cmake:215 (message):
  Error generating
  /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/coreneuron/CMakeFiles/cudacoreneuron.dir/nrniv/./cudacoreneuron_generated_cuda_profile.cu.o


nvcc fatal   : Value 'sm_20' is not defined for option 'gpu-architecture'
CMake Error at cudacoreneuron_generated_nrnran123.cu.o.RelWithDebInfo.cmake:215 (message):
  Error generating
  /home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o


make[2]: *** [coreneuron/CMakeFiles/cudacoreneuron.dir/build.make:65: coreneuron/CMakeFiles/cudacoreneuron.dir/nrniv/cudacoreneuron_generated_cuda_profile.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [coreneuron/CMakeFiles/cudacoreneuron.dir/build.make:72: coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o] Error 1
make[2]: Leaving directory '/home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu'
make[1]: *** [CMakeFiles/Makefile2:1020: coreneuron/CMakeFiles/cudacoreneuron.dir/all] Error 2
make[1]: Leaving directory '/home/john.gwak/coreneuron_tutorial/sources/ringtest/coreneuron_x86_gpu'
make: *** [Makefile:141: all] Error 2
Our linux version

Code: Select all

Linux hodgkin 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
Any advice will help me a lot!!


And... I have some questions about parallel simulations.

According to this tutorial, there are many ways to run parallel simulations.
NEURON with MPI, CoreNEURON on CPU (maybe with MPI?) and CoreNEURON on GPU (without MPI?).

Then, what is the difference between NEURON with MPI and CoreNEURON on CPU (with MPI)? what makes CoreNEURON faster??
In case of CoreNEURON with CPU and GPU, what will be running on GPU?
Thanks in advance :)

hines
Site Admin
Posts: 1526
Joined: Wed May 18, 2005 3:32 pm

Re: How to execute CoreNEURON with GPU?

Post by hines » Mon Jul 30, 2018 10:09 am

With respect to cmake, I've asked my more knowledgable colleague to weigh in. That may save some flailing around on my part.

CoreNEURON should be able to use MPI with either CPU or GPU. That is, in either case the cpu manages spike exchange and the event will
be processed by the GPU in the latter case as well as all computation that takes the model state from t to t+dt. CoreNEURON has high performance on GPU due to Structure of array style memory layout. I believe that helps considerably also with the Intel phi pipeline.

pkumbhar
Posts: 6
Joined: Fri Mar 11, 2016 5:57 am

Re: How to execute CoreNEURON with GPU?

Post by pkumbhar » Mon Jul 30, 2018 3:15 pm

Dear John,

Code: Select all

-- Found MPI_C: /usr/pgi/linux86-64/2018/mpi/openmpi-2.1.2/bin/mpicc (found version "3.1")
-- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS)
CMake Error at /usr/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find MPI (missing: MPI_CXX_FOUND) (found version "3.1")
Call Stack (most recent call first):
Your MPI C compiler seems to work but not MPI CXX compiler. Could you try compiling simple hello_world C++ program with mpicxx ? Problems seems to be installation related. I just installed exact same version of compiler and it seems to work fine (see below install script).

Code: Select all

nvcc fatal   : Value 'sm_20' is not defined for option 'gpu-architecture'
CMake Error at cudacoreneuron_generated_cuda_profile.cu.o.RelWithDebInfo.cmake:215 (message):
  Error generating
The issue is that latest cuda version doesn't recognise old SM arch version. I have fixed this issue. You can clone repository again. Here is how I just built everything:

Code: Select all

#!/bin/bash
set -e

module purge all
export MODULEPATH=/gpfs/path-to-install/pgi-18.4/modulefiles/:$MODULEPATH
module load cuda/9.0.176
module load openmpi/2.1.2/2018 pgi/18.4
module load cmake

export CC=mpicc
export CXX=mpicxx
FLAGS="-O2 -ta=tesla:cuda9.0 -Minline=size:1000,levels:100,totalsize:40000,maxsize:4000"

#directory where we will install
export BASE_DIR=`pwd`/coreneuron_home
export INSTALL_DIR=$BASE_DIR/install
export SOURCE_DIR=$BASE_DIR/sources

mkdir -p $SOURCE_DIR
mkdir -p $INSTALL_DIR
cd $SOURCE_DIR

# clone branch pgi_fixes for time being
if [ ! -d "coreneuron" ]; then
    git clone --recursive https://github.com/BlueBrain/CoreNeuron.git coreneuron
fi

mkdir -p coreneuron/build
cd coreneuron/build

cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR -DCMAKE_C_FLAGS="$FLAGS" -DCMAKE_CXX_FLAGS="$FLAGS" -DCMAKE_BUILD_TYPE=MY_CUSTOM -DCORENEURON_OPENMP=OFF -DCOMPILE_LIBRARY_TYPE=STATIC -DENABLE_MPI=ON -DCUDA_HOST_COMPILER=`which gcc` -DCUDA_PROPAGATE_HOST_FLAGS=OFF -DENABLE_SELECTIVE_GPU_PROFILING=ON -DENABLE_OPENACC=ON
make VERBOSE=1 -j8
make install

Note that if you don't have MPI working then you can build without MPI using option -DENABLE_MPI=OFF

Let me know if you see any issue and I will be happy to help.

Edit : updating post as changes are already merged to master

john.gwak
Posts: 5
Joined: Thu Jul 26, 2018 4:04 am

Re: How to execute CoreNEURON with GPU?

Post by john.gwak » Tue Jul 31, 2018 6:59 am

Dear hines and pkumbhar,

Deeply appreciate your help :)

First of all, I tried hello_world with mpicxx compiler (pgc++), but, I got a below error message.

Code: Select all

"/usr/include/x86_64-linux-gnu/bits/floatn.h", line 79: error: 128-bit
          floating-point types are not supported in this configuration
  typedef _Complex float __cfloat128 __attribute__ ((__mode__ (__TC__)));
                                                     ^
I tried to find related error or information from the PGI user forum and I found only one article and put two more lines as shown below.
(Is this error caused by version conflict among packages such as PGI(18.4), glibc(2.27), gcc(6.4 default), or even Debian and something else..?)

Code: Select all

#define __CUDACC__ 
#include <math.h> 
https://www.pgroup.com/userforum/viewtopic.php?t=5797.

This solved above problem and I could do cmake without any errors.
However, I got another error while doing 'make VERBOS=1 -j8'.

Code: Select all

"/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/Ra
          ndom123/features/nvccfeatures.h", line 36: catastrophic error: #error
          directive: "why are we in nvccfeatures.h if CUDART_VERSION is not
          defined"
  #error "why are we in nvccfeatures.h if CUDART_VERSION is not defined"
   ^

1 catastrophic error detected in the compilation of "/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/nrniv/multisend_setup.cpp".
Compilation terminated.
make[2]: *** [coreneuron/CMakeFiles/coreneuron.dir/build.make:262: coreneuron/CMakeFiles/coreneuron.dir/nrniv/multisend_setup.cpp.o] Error 2
make[2]: *** Waiting for unfinished jobs....
"/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/Ra
          ndom123/features/nvccfeatures.h", line 36: catastrophic error: #error
          directive: "why are we in nvccfeatures.h if CUDART_VERSION is not
          defined"
  #error "why are we in nvccfeatures.h if CUDART_VERSION is not defined"
   ^

1 catastrophic error detected in the compilation of "/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/nrnoc/capac.cpp".
Compilation terminated.

Code: Select all

"/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/Ra
          ndom123/features/sse.h", line 68: warning: variable "eax" was set but
          never used
      unsigned int eax, ebx, ecx, edx;
                   ^

"/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/Ra
          ndom123/features/sse.h", line 68: warning: variable "ebx" was set but
          never used
      unsigned int eax, ebx, ecx, edx;
                        ^

"/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/Ra
          ndom123/features/sse.h", line 68: warning: variable "edx" was set but
          never used
      unsigned int eax, ebx, ecx, edx;
                                  ^
I am a beginner of Linux system and parallel NEURON.
Your advice will help me a lot, thank you :)

Jeremy
Posts: 5
Joined: Tue Jul 31, 2018 10:20 am

Re: How to execute CoreNEURON with GPU?

Post by Jeremy » Tue Jul 31, 2018 11:35 am

Dear John,

I try to understand why this isnt working. Can I suggest you the following:

1. Checking the compilation command used to compile /home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/Ra
ndom123/features/nvccfeatures.h. It might be that the wrong compiler is used for this file.

2. Just in case: checking that PGI compiler is correctly installed with Nvidia option as it is not the default value when installing PGI.

3. Checking that the appropriate toolkit is found by PGI compiler: for a cuda toolkit not installed with PGI release, this can be fixed by setting CUDA_HOME environment variable if not set.

Code: Select all

export CUDA_HOME=<cuda toolkit directory>
I will continue to investigate if anything comes in mind.

My best regards,
Jeremy.

john.gwak
Posts: 5
Joined: Thu Jul 26, 2018 4:04 am

Re: How to execute CoreNEURON with GPU?

Post by john.gwak » Thu Aug 02, 2018 6:43 am

Dear Jeremy,

Thank you for your help.

I tried to follow your suggestions, however, I could not solve problems because I am a beginner for these systems... :(

Firstly, I tried to find a compilation command for 'nvccfeatures.h', however, It is not easy to follow all 'make' steps for me.
For the second and third suggestions, I tried to check environment variables with a 'pgcc -show' command as below.
However, I do not know exactly how should I check the correct installation of PGI compiler.
(mpicxx worked with a simple hello_world example, but, pgc++ did not work...)

Code: Select all

CUCOMMANDS          Commands to run for CUDA files
CUDA_HOME           =/usr/local/cuda-9.0
CUDA_HOME_ENV       =/usr/local/cuda-9.0
CUDA_NOATTACH       =0
CUDAINC
CUDALIBDIR          =/usr/local/cuda-9.0/lib64
CUDALIBDIRNAME      =lib64
CUDALIBINSTALL
CUDALIBSTATIC
CUDANEEDED          =0
CUDAROOT            =/usr/local/cuda-9.0
CUDARPATH
CUDARPATHNEEDED     =0
CUDARTLIBS          =cudadevrt cudart
CUDARTNEEDED        =0
CUDARTSTATIC
CUDASITELIB
CUDAVERSION         =8.0
CUDAVERSIONDEFINE
CUDAXDY             =8.0
CUDAXXYY            =8000
CUDAXXYY1
CUDAXXYY2           =8000
CUDAXY              =80
USECUDAROOT         =/usr/local/cuda-9.0

One more question is when I do the 'cmake' step, there was a warning message related to 'libcudart_static'.
I can find this library file under $CUDA_HOME/lib64, what makes this warning and this will be okay with next steps?

Code: Select all

CMake Warning at /usr/share/cmake-3.11/Modules/FindCUDA.cmake:887 (message):
  Expecting to find librt for libcudart_static, but didn't find it.
Please give me any advice or how to check the correct installation?
Thanks a lot in advance !!

Jeremy
Posts: 5
Joined: Tue Jul 31, 2018 10:20 am

Re: How to execute CoreNEURON with GPU?

Post by Jeremy » Thu Aug 02, 2018 12:22 pm

Hi John,
I have check on our system the result of pgic++ -show, and got similar results than yours.
Is it possible that you post the output result of compilation using the following command ?

Code: Select all

make -j VERBOSE=1 > output_compilation 2>&1
If I am not misleaded,
There is two compilation units that might point to nvccfeatures.h : nrnran123.cpp and nrnran123.cu. Can you check if you have correct flags for both ?

Concerning the cpp file, it must be:

Code: Select all

cd */Devel/coreneuron_dev/CoreNeuron/build/coreneuron && */pgi-18.4/linux86-64/18.4/bin/pgc++  -DCORENEURON_BUILD -DCUDA_PROFILING -DDISABLE_HOC_EXP -DHAVE_MALLOC_H -DLAYOUT=0 -DNRNMPI=0 -DNRN_MULTISEND=0 -DPG_ACC_BUGS -DSWAP_ENDIAN_DISABLE_ASM -I*/CoreNeuron/coreneuron/utils/randoms -I*/CoreNeuron/build/coreneuron -I*/CoreNeuron -I*/CoreNeuron/coreneuron -*/cuda-9.0.176-irkhk2/include  -acc -Minline=size:200,levels:10 -O2 -ta=tesla:cuda9.0 -Minline=size:1000,levels:100,totalsize:40000,maxsize:4000   -DDISABLE_OPENACC -o CMakeFiles/coreneuron.dir/utils/randoms/nrnran123.cpp.o -c */CoreNeuron/coreneuron/utils/randoms/nrnran123.cpp

( I have replaced paths with * which are not really interesting here :-)
Best,
Jeremy

john.gwak
Posts: 5
Joined: Thu Jul 26, 2018 4:04 am

Re: How to execute CoreNEURON with GPU?

Post by john.gwak » Thu Aug 02, 2018 10:30 pm

Dear Jeremy,

I really appreciate your help :)

This is the output results and I use Grep to find lines included "nrnran123.c".

Code: Select all

[  1%] Building NVCC (Device) object coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o
cd /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms && /usr/bin/cmake -D verbose:BOOL=1 -D build_configuration:STRING=MY_CUSTOM -D generated_file:STRING=/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o -D generated_cubin_file:STRING=/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o.cubin.txt -P /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.MY_CUSTOM.cmake
-- Removing /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o
/usr/bin/cmake -E remove /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o
-- Generating dependency file: /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.NVCC-depend
/usr/local/cuda-9.0/bin/nvcc -M -D__CUDACC__ /home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/nrnran123.cu -o /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.NVCC-depend -ccbin /usr/bin/gcc -m64 -DNRNMPI=1 -DLAYOUT=0 -DDISABLE_HOC_EXP -DHAVE_MALLOC_H -DCORENEURON_BUILD -DSWAP_ENDIAN_DISABLE_ASM -DPG_ACC_BUGS -DCUDA_PROFILING -gencode=arch=compute_30,code=sm_30 -gencode=arch=compute_60,code=sm_60 -DNVCC -I/usr/local/cuda-9.0/include -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms -I/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron
-- Generating temporary cmake readable file: /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend.tmp
/usr/bin/cmake -D input_file:FILEPATH=/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.NVCC-depend -D output_file:FILEPATH=/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend.tmp -D verbose=1 -P /usr/share/cmake-3.11/Modules/FindCUDA/make2cmake.cmake
-- Copy if different /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend.tmp to /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend
/usr/bin/cmake -E copy_if_different /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend.tmp /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend
-- Removing /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend.tmp and /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.NVCC-depend
/usr/bin/cmake -E remove /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.depend.tmp /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o.NVCC-depend
-- Generating /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o
/usr/local/cuda-9.0/bin/nvcc /home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/nrnran123.cu -dc -o /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o -ccbin /usr/bin/gcc -m64 -DNRNMPI=1 -DLAYOUT=0 -DDISABLE_HOC_EXP -DHAVE_MALLOC_H -DCORENEURON_BUILD -DSWAP_ENDIAN_DISABLE_ASM -DPG_ACC_BUGS -DCUDA_PROFILING -gencode=arch=compute_30,code=sm_30 -gencode=arch=compute_60,code=sm_60 -DNVCC -I/usr/local/cuda-9.0/include -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms -I/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron
Generated /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o successfully.
cd /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron && /usr/local/cuda-9.0/bin/nvcc -gencode=arch=compute_30,code=sm_30 -gencode=arch=compute_60,code=sm_60 -m64 -ccbin /usr/bin/gcc -dlink /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/nrniv/./cudacoreneuron_generated_cuda_profile.cu.o /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/utils/randoms/./cudacoreneuron_generated_nrnran123.cu.o -o /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron/CMakeFiles/cudacoreneuron.dir/./cudacoreneuron_intermediate_link.o
/usr/bin/ar qc libcudacoreneuron.a  CMakeFiles/cudacoreneuron.dir/nrniv/cudacoreneuron_generated_cuda_profile.cu.o CMakeFiles/cudacoreneuron.dir/utils/randoms/cudacoreneuron_generated_nrnran123.cu.o CMakeFiles/cudacoreneuron.dir/cudacoreneuron_intermediate_link.o
[ 64%] Building CXX object coreneuron/CMakeFiles/coreneuron.dir/utils/randoms/nrnran123.cpp.o
cd /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron && /usr/pgi/linux86-64/2018/mpi/openmpi-2.1.2/bin/mpicxx  -DCORENEURON_BUILD -DCUDA_PROFILING -DDISABLE_HOC_EXP -DHAVE_MALLOC_H -DLAYOUT=0 -DNRNMPI=1 -DPG_ACC_BUGS -DSWAP_ENDIAN_DISABLE_ASM -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms -I/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron -I/usr/local/cuda-9.0/include  -acc -Minline=size:200,levels:10 -O2 -ta=tesla:cuda9.0 -Minline=size:1000,levels:100,totalsize:40000,maxsize:4000   -DDISABLE_OPENACC -o CMakeFiles/coreneuron.dir/utils/randoms/nrnran123.cpp.o -c /home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/nrnran123.cpp
1 catastrophic error detected in the compilation of "/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/nrnran123.cpp".
make[2]: *** [coreneuron/CMakeFiles/coreneuron.dir/build.make:1231: coreneuron/CMakeFiles/coreneuron.dir/utils/randoms/nrnran123.cpp.o] Error 2

Code: Select all

cd /home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron && /usr/pgi/linux86-64/2018/mpi/openmpi-2.1.2/bin/mpicxx  -DCORENEURON_BUILD -DCUDA_PROFILING -DDISABLE_HOC_EXP -DHAVE_MALLOC_H -DLAYOUT=0 -DNRNMPI=1 -DPG_ACC_BUGS -DSWAP_ENDIAN_DISABLE_ASM -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms -I/home/john.gwak/coreneuron_home/sources/coreneuron/build/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron -I/home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron -I/usr/local/cuda-9.0/include  -acc -Minline=size:200,levels:10 -O2 -ta=tesla:cuda9.0 -Minline=size:1000,levels:100,totalsize:40000,maxsize:4000   -DDISABLE_OPENACC -o CMakeFiles/coreneuron.dir/utils/randoms/nrnran123.cpp.o -c /home/john.gwak/coreneuron_home/sources/coreneuron/coreneuron/utils/randoms/nrnran123.cpp
I found that compilation command (for me using 'mpicxx') and flags are slightly different from your case.
What is the reason for this difference and could you let me know how to fix this problem...?

Thank you for your concern !!

Best regards,
John.

Jeremy
Posts: 5
Joined: Tue Jul 31, 2018 10:20 am

Re: How to execute CoreNEURON with GPU?

Post by Jeremy » Fri Aug 03, 2018 7:57 am

Dear John,
I am still investigating so sorry that is not straightforward as I cannot reproduce the same from my machine, but we will get it.

It seems to me that we enter an include file that must be entered only when compiling with NVCC with -D____CUDACC__ option.

Can you check if __CUDACC__ is defined in your environment (it should not be defined) ?

Code: Select all

 echo $__CUDACC__
Can you also provide the output for:

Code: Select all

module show cuda/9.0.176
and

Code: Select all

module show pgi/18.4

My best regards,
Jeremy

john.gwak
Posts: 5
Joined: Thu Jul 26, 2018 4:04 am

Re: How to execute CoreNEURON with GPU?

Post by john.gwak » Sun Aug 05, 2018 9:37 pm

Dear Jeremy,

Thank you for your effort!

According to the '__CUDACC__', I got no message with the echo command.
But, I defined '__CUDACC__' to overcome a "Could NOT find MPI_CXX (missing: MPI_CXX_WORKS)" error.
(Please check my 2nd article here, I think this may cause including unwanted files...)

Code: Select all

"/usr/include/x86_64-linux-gnu/bits/floatn.h", line 79: error: 128-bit
          floating-point types are not supported in this configuration
  typedef _Complex float __cfloat128 __attribute__ ((__mode__ (__TC__)));

#define __CUDACC__ 
#include <math.h>

(I put above two lines in the floatn.h file)

Code: Select all

-------------------------------------------------------------------
/etc/modulefiles/cuda/9.0:

module-whatis   sets up environment for CUDA 9.0
setenv          CUDA_HOME       /usr/local/cuda-9.0
append-path     PATH /usr/local/cuda-9.0/bin
append-path     MANPATH /usr/local/cuda-9.0/doc/man
append-path     LD_LIBRARY_PATH /usr/local/cuda-9.0/lib64
append-path     INCLUDE /usr/local/cuda-9.0/include
append-path     CPATH /usr/local/cuda-9.0/include
append-path     LIBRARY_PATH /usr/local/cuda-9.0/lib64
conflict        cuda
-------------------------------------------------------------------

-------------------------------------------------------------------
/usr/pgi/modulefiles/pgi/18.4:

module-whatis   Changes the PGI home directory to linux86-64 18.4
setenv          PGI     /usr/pgi
setenv          CC      /usr/pgi/linux86-64/18.4/bin/pgcc
setenv          FC      /usr/pgi/linux86-64/18.4/bin/pgfortran
setenv          F90     /usr/pgi/linux86-64/18.4/bin/pgf90
setenv          F77     /usr/pgi/linux86-64/18.4/bin/pgf77
setenv          CPP     /usr/pgi/linux86-64/18.4/bin/pgcc -Mcpp
setenv          CXX     /usr/pgi/linux86-64/18.4/bin/pgc++
prepend-path    PATH /usr/pgi/linux86-64/18.4/bin
prepend-path    MANPATH /usr/pgi/linux86-64/18.4/man
prepend-path    LD_LIBRARY_PATH /usr/pgi/linux86-64/18.4/lib
conflict        pgi
-------------------------------------------------------------------
Above is the output of the module command.
I really appreciate your help, Jeremy :)

Best regards,
John.

Jeremy
Posts: 5
Joined: Tue Jul 31, 2018 10:20 am

Re: How to execute CoreNEURON with GPU?

Post by Jeremy » Mon Aug 06, 2018 9:47 am

Dear John,
my mistake, indeed as you posted before, defining __CUDACC__ permit to define the float128 but with this trick, the C compiler pretends to be a CUDA compiler for all compiled files.

I can suggest you the following:
1. It is better to use a glibc fully supported by PGI if it is possible: you can find your glibc version by:

Code: Select all

ldd `which pgcc` | grep libc 
then just execute the libc path to obtain the library version:
for instance:

Code: Select all

$ /usr/lib64/libc.so.6

GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.8.5 20150623 (Red Hat 4.8.5-16).
Compiled on a Linux 3.10.0 system on 2017-11-22.
Available extensions:
	The C stubs add-on version 2.1.2.
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
	RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.


2. If really you cannot change libc, One trick that might work but I havent tested it yet: you can quick patch Coreneuron for your own compilation:
replace all instance of __CUDACC__ in Coreneuron source code by CUDART_VERSION

Hoping those will fix,
My best regards,
Jeremy.

Jeremy
Posts: 5
Joined: Tue Jul 31, 2018 10:20 am

Re: How to execute CoreNEURON with GPU?

Post by Jeremy » Thu Aug 09, 2018 3:59 am

Dear John,
can I ask you on which machine you are trying to execute ?
Is there a chance that we can access it ?

Best regards,
Jeremy.

Post Reply