PETSc

Purpose

The grid solver makes use of the »Portable, Extensible Toolkit for Scientific Computation« (PETSc). PETSc is developed as open-source software and can be downloaded from the Argonne National Laboratory.

Compatibility

• PETSc version 3.11.x is required by DAMASK v3-alpha
• PETSc version 3.10.x is required by DAMASK since revision v2.0.3 (and, hence, for the current release version.)
• PETSc version 3.9.x is required by DAMASK since revision v2.0.2
• PETSc version 3.6.x is required by DAMASK since revision 4007
• PETSc version 3.5.x is required by DAMASK since revision 3460
• PETSc version 3.4.x is required by DAMASK since revision 2794
• PETSc version 3.3.x is required by DAMASK prior to revision 2794

Installation

DAMASK requires $PETSC_DIR to be set correctly. In case of multiple parallel PETSc installations on a system, make sure that $PETSC_DIR and/or $PETSC_ARCH refer to the intended version.

Install from source

For the manual installation, download the source and read the installation instruction. For DAMASK, PETSc needs to be compiled with support for all Fortran 2003 features and needs to be aware of certain external packages.

The following instructions outline how PETSc can be installed.

Automatic resolution of all externalities

The easiest way to configure PETSc for DAMASK is to let PETSc download and install all auxiliary libraries, automatically compile and install them alongside PETSc itself. This process can be done by executing the following configure command inside the directory to which PETSc was downloaded:

> ./configure \
--with-fc=$F90 --with-cc=$CC --with-cxx=$CXX \
--download-mpich \
--download-fftw \
--download-hdf5 \
--download-fblaslapack \
--download-chaco \
--download-hypre \
--download-metis \
--download-ml \
--download-mumps \
--download-parmetis \
--download-scalapack \
--download-suitesparse \
--download-superlu \
--download-superlu_dist \
--download-triangle \
--download-zlib \
--with-cxx-dialect=C++11 \
--with-c2html=0 \
--with-debugging=0 \
--with-ssl=0 \
--with-x=0 \
COPTFLAGS="-O3 -xHost -no-prec-div" \
CXXOPTFLAGS="-O3 -xHost -no-prec-div" \
FOPTFLAGS="-O3 -xHost -no-prec-div" \
PETSC_DIR=$(pwd)

where the variables $F90, $CC, and $CXX need to point to your Fortran, C, and C++ compiler. After a successful configuration (watch out for error and warning messages), build PETSc with

> make
> make test

This will compile and install a PETSc library with support for Fortran 2003 features, along with all necessary third-party libraries. FFTW, HDF5, BLAS/LAPACK, and MPI are the minimum requirements in configuring PETSc for the DAMASK spectral solver.

Remarks

• Switching off certain features (--with-c2html=0, --with-x=0, --with-ssl=0, and --with-debugging=0) is optional.
• Optimization flags (COPTFLAGS, CXXOPTFLAGS, and FOPTFLAGS) might be adjusted.

Customizations

MPI library

If MPI is already installed on your system, the following commands should identify themselves as the compiler of your choice. These names might depend on your MPI library and/or compiler suite.

> mpif90 -v # equivalent to ifort -v / gortran -v
> mpicc -v # equivalent to icc -v / gcc -v
> mpicxx -v # equivalent to icpc -v / g++ -v

In that case, alter the configuration to

--with-fc=$F90 \
--with-cc=$CC \
--with-cxx=$CXX \
--download-mpich \
--with-fc=mpif90 \
--with-cc=mpicc \
--with-cxx=mpicxx \

LAPACK

Instead of downloading a linear algebra package with --download-fblaslapack, LAPACK, installed as a system package, can be used and will be autodetected:

--download-fblaslapack

alternatively, IMKL (located at $MKLROOT/lib/intel64) can be used with

--download-fblaslapack
--with-blaslapack-dir=$MKLROOT/lib/intel64

FFTW

If FFTW is available (exemplarily located at /opt/fftw), don’t download it automatically but specify its location

--download-fftw
--with-fftw-dir=/opt/fftw

HDF5

If HDF5 is available (exemplarily located at /opt/hdf5), don’t download it automatically but specify its location

--download-hdf5
--with-hdf5-dir=/opt/hdf5

Deployment directory

To deploy PETSc after building, specify the intended file path with the --prefix option.

./configure \
--prefix=${deploymentDir} \

Known issues

• segmentation fault — Using PETSc compiled with Intel Fortran but compiling DAMASK using GNU Fortran or using PETSc compiled with GNU Fortran but compiling DAMASK using Intel Fortran might cause a segmentation fault. In an environment where both compilers should be used, compile PETSc with both compilers and select your PETSc version via setting of $PETSC_ARCH.
• C compiler you provided with –with-cc=icc does not work — This happens if your C compiler (icc in this example) is not located in one of the standard locations but rather available via $PATH and you’re installing via sudo. Use sudo su instead of sudo ./configure ...