Purpose
The grid solver makes use of the »Portable, Extensible Toolkit for Scientific Computation« (PETSc). PETSc is developed as open-source software and can be downloaded from the Argonne National Laboratory.
Compatibility
- PETSc version 3.11.x is required by DAMASK v3-alpha
- PETSc version 3.10.x is required by DAMASK since revision v2.0.3 (and, hence, for the current release version.)
- PETSc version 3.9.x is required by DAMASK since revision v2.0.2
- PETSc version 3.6.x is required by DAMASK since revision 4007
- PETSc version 3.5.x is required by DAMASK since revision 3460
- PETSc version 3.4.x is required by DAMASK since revision 2794
- PETSc version 3.3.x is required by DAMASK prior to revision 2794
Installation
DAMASK requires $PETSC_DIR to be set correctly. In case of multiple parallel PETSc installations on a system, make sure that $PETSC_DIR and/or $PETSC_ARCH refer to the intended version.
Install from source
For the manual installation, download the source and read the installation instruction. For DAMASK, PETSc needs to be compiled with support for all Fortran 2003 features and needs to be aware of certain external packages.
The following instructions outline how PETSc can be installed.
Automatic resolution of all externalities
The easiest way to configure PETSc for DAMASK is to let PETSc download and install all auxiliary libraries, automatically compile and install them alongside PETSc itself. This process can be done by executing the following configure command inside the directory to which PETSc was downloaded:> ./configure \
--with-fc=$F90 --with-cc=$CC --with-cxx=$CXX \
--download-mpich \
--download-fftw \
--download-hdf5 \
--download-fblaslapack \
--download-chaco \
--download-hypre \
--download-metis \
--download-ml \
--download-mumps \
--download-parmetis \
--download-scalapack \
--download-suitesparse \
--download-superlu \
--download-superlu_dist \
--download-triangle \
--download-zlib \
--with-cxx-dialect=C++11 \
--with-c2html=0 \
--with-debugging=0 \
--with-ssl=0 \
--with-x=0 \
COPTFLAGS="-O3 -xHost -no-prec-div" \
CXXOPTFLAGS="-O3 -xHost -no-prec-div" \
FOPTFLAGS="-O3 -xHost -no-prec-div" \
PETSC_DIR=$(pwd)
where the variables $F90, $CC, and $CXX need to point to your Fortran, C, and C++ compiler. After a successful configuration (watch out for error and warning messages), build PETSc with> make
> make test
This will compile and install a PETSc library with support for Fortran 2003 features, along with all necessary third-party libraries. FFTW, HDF5, BLAS/LAPACK, and MPI are the minimum requirements in configuring PETSc for the DAMASK spectral solver.
Remarks
- Switching off certain features (
--with-c2html=0
,--with-x=0
,--with-ssl=0
, and--with-debugging=0
) is optional. - Optimization flags (
COPTFLAGS
,CXXOPTFLAGS
, andFOPTFLAGS
) might be adjusted.
Customizations
MPI library
If MPI is already installed on your system, the following commands should identify themselves as the compiler of your choice. These names might depend on your MPI library and/or compiler suite.> mpif90 -v # equivalent to ifort -v / gortran -v
> mpicc -v # equivalent to icc -v / gcc -v
> mpicxx -v # equivalent to icpc -v / g++ -v
In that case, alter the configuration to --with-fc=$F90 \
--with-cc=$CC \
--with-cxx=$CXX \
--download-mpich \
--with-fc=mpif90 \
--with-cc=mpicc \
--with-cxx=mpicxx \
LAPACK
Instead of downloading a linear algebra package with --download-fblaslapack
, LAPACK, installed as a system package, can be used and will be autodetected: --download-fblaslapack
alternatively, IMKL (located at $MKLROOT/lib/intel64) can be used with
--download-fblaslapack
--with-blaslapack-dir=$MKLROOT/lib/intel64
FFTW
If FFTW is available (exemplarily located at /opt/fftw), don’t download it automatically but specify its location
--download-fftw
--with-fftw-dir=/opt/fftw
HDF5
If HDF5 is available (exemplarily located at /opt/hdf5), don’t download it automatically but specify its location
--download-hdf5
--with-hdf5-dir=/opt/hdf5
Deployment directory
To deploy PETSc after building, specify the intended file path with the --prefix
option../configure \
--prefix=${deploymentDir} \
Known issues
- segmentation fault — Using PETSc compiled with Intel Fortran but compiling DAMASK using GNU Fortran or using PETSc compiled with GNU Fortran but compiling DAMASK using Intel Fortran might cause a segmentation fault. In an environment where both compilers should be used, compile PETSc with both compilers and select your PETSc version via setting of $PETSC_ARCH.
- C compiler you provided with –with-cc=icc does not work — This happens if your C compiler (icc in this example) is not located in one of the standard locations but rather available via $PATH and you’re installing via
sudo
. Usesudo su
instead ofsudo ./configure ...