Difference between revisions of "User:Ppomorsk"

Revision as of 17:12, 4 January 2018 (view source) Ppomorsk (Talk | contribs) (→‎Octopus instructions) ← Older edit		Revision as of 17:12, 4 January 2018 (view source) Ppomorsk (Talk | contribs) (→‎Install clFFT) Newer edit →
Line 219:		Line 219:
	git clone https://github.com/clMathLibraries/clFFT.git		git clone https://github.com/clMathLibraries/clFFT.git
		+	...
	module load cuda/8.0.44		module load cuda/8.0.44
	module load fftw/3.3.6		module load fftw/3.3.6

---

Revision as of 17:12, 4 January 2018

Pawel Pomorski

High Performance Computing Programming Specialist

University of Waterloo

User:Ppomorsk/Workspace

User:Ppomorsk/Toward Exascale Simulations of 3D Radiative Transfer for Cloudy Atmospheres

Important Pages

• CUDA

Software which I help maintain

• CPMD

• CUDA

• GCC

• NAMD

• OpenCL

• Python

• SIESTA

• XCrySDen

• MKL

Help Wiki stats

Wiki stats page is here.

This wiki has 6,532 users, among them 0 active users (users who performed an action within last 91 days). The total number of page edits is 19,165. The total number of page views is Template:NUMBEROFVIEWS. There are 498 pages and 415 articles.

NAMD instructions

These instructions for monk, current as of Feb.2014, with default modules:

intel/12.1.3            
cuda/5.5.22           

Preliminaries:

module unload openmpi
module load openmpi/intel/1.7.4
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/sharcnet/intel/12.1.3/icc/composer_xe_2011_sp1.9.293/compiler/lib/intel64

OpenMPI 1.7.4 is compiled with CUDA support. Environment variable is needed to locate all the libraries at runtime.

Get all the required source files:

tar xvfz NAMD_2.9_Source.tar.gz 
cd NAMD_2.9_Source

Download and install precompiled TCL and FFTW libraries: (working in NAMD source directory)

 wget http://www.ks.uiuc.edu/Research/namd/libraries/fftw-linux-x86_64.tar.gz
 tar xzf fftw-linux-x86_64.tar.gz
 mv linux-x86_64 fftw
 wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64.tar.gz
 wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64-threaded.tar.gz
 tar xzf tcl8.5.9-linux-x86_64.tar.gz
 tar xzf tcl8.5.9-linux-x86_64-threaded.tar.gz
 mv tcl8.5.9-linux-x86_64 tcl
 mv tcl8.5.9-linux-x86_64-threaded tcl-threaded

Compile charm, choosing which parallelization is to be used (threaded, MPI, MPI-SMP):

tar xvf charm-6.4.0.tar 
cd charm-6.4.0

Compile charm - MPI build:

env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64  --no-build-shared --with-production

Compile charm - MPI-SMP build:

env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64 smp --no-build-shared --with-production

Compile charm - threaded build:

./build charm++ multicore-linux64 --no-build-shared  icc8 --with-production

Configure NAMD and compile (with CUDA support enabled) - replace CHARM_ARCHITECTURE with one of the three options above:

./config  Linux-x86_64-icc   --charm-arch CHARM_ARCHITECTURE --with-cuda --cuda-prefix /opt/sharcnet/cuda/5.5.22/toolkit/
cd Linux-x86_64-icc
make

Test charm if needed:

From the directory charm is in,

cd architecture_directory/tests/charm++/megatest
make pgm
mpirun -n 4 ./pgm

Python installation instructions

Even though Python modules are provided by SHARCNET, sometimes there may be a need to compile your own Python and Numpy. Here are instructions on how to do it (tested in September, 2014):

To get the source tarballs, do:

wget --no-check-certificate https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
wget http://sourceforge.net/projects/numpy/files/NumPy/1.8.2/numpy-1.8.2.tar.gz

and unpack these somewhere in your directories.

These were built under modules:

module unload intel 
module unload mkl 
module unload openmpi 
module load gcc/4.8.2 
module load openmpi/gcc/1.8.1

(openmpi is not necessary for this built, but I would suggest loading the gcc/1.8.1 anyway)

Python was built with:

./configure --enable-shared --prefix=~/software_installs/python/2.7.8/gcc/installdir
make
make install

Then set ( in .bashrc for permanent change):

export PATH=~/software_installs/python/2.7.8/gcc/installdir/bin:$PATH
export LD_LIBRARY_PATH=~/software_installs/python/2.7.8/gcc/installdir/lib:$LD_LIBRARY_PATH

Numpy was built with:

unset LDFLAGS
python setup.py build --fcompiler=gnu95
python setup.py install --prefix=~/software_installs/numpy/1.8.2/gcc/installdir

Also did:

ln -sf ~/software_installs/numpy/1.8.2/gcc/installdir/lib/python2.7/site-packages/numpy/core/include/numpy ~/software_installs/python/2.7.8/gcc/installdir/include/python2.7

and finally

export PYTHONPATH=~/software_installs/numpy/1.8.2/gcc/installdir/lib/python2.7/site-packages/

CNVnator instructions

Install root prerequisite

Note: compiling this package takes a long time, so it's best to use /tmp directory for faster disk access.

wget https://root.cern.ch/download/root_v6.06.06.source.tar.gz
tar xvfz root_v6.06.06.source.tar.gz
module unload intel openmpi mkl
module load gcc/4.9.2
module load python/gcc/2.7.8
mkdir builddir
cd builddir
cmake ../ -DCMAKE_INSTALL_PREFIX=/work/lianglab/bin/install_root -Dgnuinstall=ON 
cmake --build .                 
cmake --build . --target install

Install CNVnator

(load same modules as for root above)

module unload intel openmpi mkl
module load gcc/4.9.2
module load python/gcc/2.7.8

wget http://sv.gersteinlab.org/cnvnator/CNVnator_v0.3.zip
unzip CNVnator_v0.3.zip
cd CNVnator_v0.3/src/samtools/
make 
cd ..

now edit the Makefile so it has

ROOTLIBS  = -L$(ROOTSYS)/lib/root -lCore -lRIO -lNet -lHist -lGraf -lGraf3d \
               -lGpad -lTree -lRint -lMatrix -lPhysics \
               -lMathCore -lThread -lGui
CXX    = g++ -std=c++11 $(ROOTFLAGS) -DCNVNATOR_VERSION=\"$(VERSION)\"
SAMDIR = samtools
INC    = -I$(ROOTSYS)/include/root -I$(SAMDIR)
SAMLIB = $(SAMDIR)/libbam.a

Note that -lCint library was removed from ROOTLIBS.

Finally, run:

export ROOTSYS=/work/lianglab/bin/install_root
export LD_LIBRARY_PATH=/work/lianglab/bin/install_root/lib/root:$LD_LIBRARY_PATH

make

This will produce the executable. In the future, run the above export LD_LIBRARY_PATH command to be able to run it.

Checking global work location

which-global-work
USER=ppomorsk which-global-work

Octopus instructions

Install cblas

module load openblas/0.2.20 
module load cuda/8.0.44
module load boost/1.60.0
FC=ifort CC=icc CXX=icpc cmake ..  -DNetlib_BLAS_LIBRARY=$EBROOTOPENBLAS   -DNetlib_INCLUDE_DIRS=$EBROOTOPENBLAS/include -
DCMAKE_INSTALL_PREFIX=~/clblasinstall -DBUILD_TEST:BOOL=OFF
make
make install
ln -s ~/clblasinstall/lib64 ~/clblasinstall/lib

Install clFFT

git clone https://github.com/clMathLibraries/clFFT.git
...
module load cuda/8.0.44
module load fftw/3.3.6
module load boost/1.60.0
FC=ifort CC=icc CXX=icpc cmake .. -DCMAKE_INSTALL_PREFIX=~/clfftinstall  -DOpenCL_LIBRARY=$CUDA_HOME/lib64/libOpenCL.so

Install Octopus

module load openblas/0.2.20
module load cuda/8.0.44
module load boost/1.60.0
module load libxc/3.0.0
module load fftw/3.3.6
module load gsl/2.3
./configure FC='ifort -mkl' CC='icc -mkl' CXX='icpc -mkl'  --prefix=/home/ppomorsk/test_compile/octopus-exec --enable-opencl --with-clblas-prefix=/home/ppomorsk/clblasinstall --with-clfft-prefix=/home/ppomorsk/clfftinstall