Pawel Pomorski
High Performance Computing Programming Specialist
University of Waterloo
User:Ppomorsk/Toward Exascale Simulations of 3D Radiative Transfer for Cloudy Atmospheres
Contents
Important Pages
Software which I help maintain
Help Wiki stats
Wiki stats page is here.
This wiki has 6,533 users, among them 0 active users (users who performed an action within last 91 days). The total number of page edits is 19,165. The total number of page views is Template:NUMBEROFVIEWS. There are 498 pages and 415 articles.
NAMD instructions
These instructions for monk, current as of Feb.2014, with default modules:
intel/12.1.3 cuda/5.5.22
Preliminaries:
module unload openmpi module load openmpi/intel/1.7.4 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/sharcnet/intel/12.1.3/icc/composer_xe_2011_sp1.9.293/compiler/lib/intel64
OpenMPI 1.7.4 is compiled with CUDA support. Environment variable is needed to locate all the libraries at runtime.
Get all the required source files:
tar xvfz NAMD_2.9_Source.tar.gz cd NAMD_2.9_Source
Download and install precompiled TCL and FFTW libraries: (working in NAMD source directory)
wget http://www.ks.uiuc.edu/Research/namd/libraries/fftw-linux-x86_64.tar.gz tar xzf fftw-linux-x86_64.tar.gz mv linux-x86_64 fftw wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64.tar.gz wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl8.5.9-linux-x86_64-threaded.tar.gz tar xzf tcl8.5.9-linux-x86_64.tar.gz tar xzf tcl8.5.9-linux-x86_64-threaded.tar.gz mv tcl8.5.9-linux-x86_64 tcl mv tcl8.5.9-linux-x86_64-threaded tcl-threaded
Compile charm, choosing which parallelization is to be used (threaded, MPI, MPI-SMP):
tar xvf charm-6.4.0.tar cd charm-6.4.0
Compile charm - MPI build:
env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64 --no-build-shared --with-production
Compile charm - MPI-SMP build:
env MPICXX=mpicxx ./build charm++ mpi-linux-x86_64 smp --no-build-shared --with-production
Compile charm - threaded build:
./build charm++ multicore-linux64 --no-build-shared icc8 --with-production
Configure NAMD and compile (with CUDA support enabled) - replace CHARM_ARCHITECTURE with one of the three options above:
./config Linux-x86_64-icc --charm-arch CHARM_ARCHITECTURE --with-cuda --cuda-prefix /opt/sharcnet/cuda/5.5.22/toolkit/ cd Linux-x86_64-icc make
Test charm if needed:
From the directory charm is in,
cd architecture_directory/tests/charm++/megatest make pgm mpirun -n 4 ./pgm
Python installation instructions
Even though Python modules are provided by SHARCNET, sometimes there may be a need to compile your own Python and Numpy. Here are instructions on how to do it (tested in September, 2014):
To get the source tarballs, do:
wget --no-check-certificate https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz wget http://sourceforge.net/projects/numpy/files/NumPy/1.8.2/numpy-1.8.2.tar.gz
and unpack these somewhere in your directories.
These were built under modules:
module unload intel module unload mkl module unload openmpi module load gcc/4.8.2 module load openmpi/gcc/1.8.1
(openmpi is not necessary for this built, but I would suggest loading the gcc/1.8.1 anyway)
Python was built with:
./configure --enable-shared --prefix=~/software_installs/python/2.7.8/gcc/installdir make make install
Then set ( in .bashrc for permanent change):
export PATH=~/software_installs/python/2.7.8/gcc/installdir/bin:$PATH export LD_LIBRARY_PATH=~/software_installs/python/2.7.8/gcc/installdir/lib:$LD_LIBRARY_PATH
Numpy was built with:
unset LDFLAGS python setup.py build --fcompiler=gnu95 python setup.py install --prefix=~/software_installs/numpy/1.8.2/gcc/installdir
Also did:
ln -sf ~/software_installs/numpy/1.8.2/gcc/installdir/lib/python2.7/site-packages/numpy/core/include/numpy ~/software_installs/python/2.7.8/gcc/installdir/include/python2.7
and finally
export PYTHONPATH=~/software_installs/numpy/1.8.2/gcc/installdir/lib/python2.7/site-packages/
CNVnator instructions
Install root prerequisite
Note: compiling this package takes a long time, so it's best to use /tmp directory for faster disk access.
wget https://root.cern.ch/download/root_v6.06.06.source.tar.gz tar xvfz root_v6.06.06.source.tar.gz module unload intel openmpi mkl module load gcc/4.9.2 module load python/gcc/2.7.8 mkdir builddir cd builddir cmake ../ -DCMAKE_INSTALL_PREFIX=/work/lianglab/bin/install_root -Dgnuinstall=ON cmake --build . cmake --build . --target install
Install CNVnator
(load same modules as for root above)
module unload intel openmpi mkl module load gcc/4.9.2 module load python/gcc/2.7.8
wget http://sv.gersteinlab.org/cnvnator/CNVnator_v0.3.zip unzip CNVnator_v0.3.zip cd CNVnator_v0.3/src/samtools/ make cd ..
now edit the Makefile so it has
ROOTLIBS = -L$(ROOTSYS)/lib/root -lCore -lRIO -lNet -lHist -lGraf -lGraf3d \
-lGpad -lTree -lRint -lMatrix -lPhysics \
-lMathCore -lThread -lGui
CXX = g++ -std=c++11 $(ROOTFLAGS) -DCNVNATOR_VERSION=\"$(VERSION)\"
SAMDIR = samtools
INC = -I$(ROOTSYS)/include/root -I$(SAMDIR)
SAMLIB = $(SAMDIR)/libbam.a
Note that -lCint library was removed from ROOTLIBS.
Finally, run:
export ROOTSYS=/work/lianglab/bin/install_root export LD_LIBRARY_PATH=/work/lianglab/bin/install_root/lib/root:$LD_LIBRARY_PATH
make
This will produce the executable. In the future, run the above export LD_LIBRARY_PATH command to be able to run it.
Checking global work location
which-global-work USER=ppomorsk which-global-work
Octopus instructions
Install cblas
module load openblas/0.2.20 module load cuda/8.0.44 module load boost/1.60.0
FC=ifort CC=icc CXX=icpc cmake .. -DNetlib_BLAS_LIBRARY=$EBROOTOPENBLAS -DNetlib_INCLUDE_DIRS=$EBROOTOPENBLAS/include - DCMAKE_INSTALL_PREFIX=~/clblasinstall -DBUILD_TEST:BOOL=OFF make make install
ln -s ~/clblasinstall/lib64 ~/clblasinstall/lib
Install clFFT
git clone https://github.com/clMathLibraries/clFFT.git module load cuda/8.0.44 module load fftw/3.3.6 module load boost/1.60.0
FC=ifort CC=icc CXX=icpc cmake .. -DCMAKE_INSTALL_PREFIX=~/clfftinstall -DOpenCL_LIBRARY=$CUDA_HOME/lib64/libOpenCL.so
Install Octopus
module load openblas/0.2.20 module load cuda/8.0.44 module load boost/1.60.0 module load libxc/3.0.0 module load fftw/3.3.6 module load gsl/2.3
./configure FC='ifort -mkl' CC='icc -mkl' CXX='icpc -mkl' --prefix=/home/ppomorsk/test_compile/octopus-exec --enable-opencl --with-clblas-prefix=/home/ppomorsk/clblasinstall --with-clfft-prefix=/home/ppomorsk/clfftinstall