6. NWChem

Overview

Teaching: min
Exercises: min

Questions

Objectives

Table of Content

Generalities June 20, morning

1.1. Installation using a Package Manager

1.2. Building from Source
Introduction to NWChem June 20, morning
DFT in NWChem June 20, morning
TDDFT in NWChem June 20, afternoon
The GW approximation June 21, morning
Correlated methods in NWChem June 21, afternoon
Gaussian-basis MD (QMD) in NWChem June 21, afternoon
Video Recordings

8.1. Morning Session, June 20

8.2. Afternoon Session, June 20

8.3. Morning Session, June 21

8.4. Afternoon Session, June 21

1. Generalities

1.1. Installation Using a Package Manager

An NWChem installation is available to all participants with access to UB CCR. The appropriate environment can be initialized by executing

eval "$(/projects/academic/cyberwksp21/Software/nwchem_conda0/bin/conda shell.bash hook)"

in a bash shell. In order to verify that the environment has been properly loaded, check the output of the which nwchem command which should be :

/projects/academic/cyberwksp21/Software/nwchem_conda0/bin/nwchem

Please note that the conda-based installation is suitable for single-node calculations but there is a performance impact as compared to native builds or containers with other optimizations enabled.

For those who want to install NWChem in their local machines, the following code snippet will download the miniconda package and install NWChem in a clean environment. We strongly suggest to avoid installing NWChem in a pre-existing conda environment, as it could lead to some incompatibilities with mpi libraries.

mkdir -p ~/conda 
cd ~/conda 
wget https://repo.anaconda.com/miniconda/Miniconda3-py38_23.1.0-1-Linux-x86_64.sh 
bash Miniconda3-py38_23.1.0-1-Linux-x86_64.sh -b -p ~/conda/miniconda3 
eval "$(~/conda/miniconda3/bin/conda shell.bash hook)" 
conda install -c conda-forge -y micromamba  
micromamba activate ~/conda/miniconda3   
eval "$(micromamba shell hook --shell=bash)" 
micromamba install -c conda-forge -y nwchem=7.2.0

Alternatively, NWChem can be installed using homebrew using the following commands

bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
brew install nwchem

1.2 Building from Source

The following set of instructions will result in an optimized NWChem binary suitable for multi-node runs in UB CCR. Keep in mind that compiling NWChem from source can be a lengthy process.

Note: the modules listed below are available using the new software infrastructure, i.e. ~/.ccr_new_modules should exist.

1.2.1 Downloading NWChem source

The first step is to download the NWChem source. Here, we will download the master branch into the user’s home directory

cd ~
git clone --depth=1 https://github.com/nwchemgit/nwchem.git

1.2.2 Setting Environment

We will start by setting the mandatory variables

export NWCHEM_TOP=$HOME/nwchem
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all xtb"
export ARMCI_NETWORK=MPI-PR
export USE_MPI=1

The use of ARMCI_NETWORK=MPI-PR is recommended for best performance as it reserves one MPI task per node for communication purposes. NWCHEM_MODULES="all xtb" will compile all modules available in NWChem plus an interface to the TBlite library that enables xTB GFN1 and GFN2 calculations.

Next, we will define the optional variables

export USE_TBLITE=1
export USE_HWOPT=1
export USE_OPENMP=1
export USE_LIBXC=1
export USE_SIMINT=1
export SIMINT_MAXAM=5
export BUILD_PLUMED=1
export BUILD_SCALAPACK=1
export BUILD_ELPA=1

The variable USE_TBLITE must be set if xtb is present in NWCHEM_MODULES. Setting USE_HWOPT=1 enables the use of hardware-specific optimizations. Setting USE_OPENMP=1 enables the use of OpenMP multithreading in some modules. Setting USE_LIBXC=1 instructs NWChem to download and compile the latest release version of LibXC. Setting USE_SIMINT=1 instructs NWChem to download and compile the SIMINT library supporting up to h-type functions (SIMINT_MAXAM=5). Setting BUILD_PLUMED=1 instructs NWChem to download and compile the PLUMED library to use enhanced-sampling algorithms in the Gaussian-basis AIMD module of NWChem. Setting BUILD_SCALAPACK=1 instructs NWChem to download and compile the reference ScaLAPACK library. Setting BUILD_ELPA=1 instructs NWChem to download en compile the ELPA eigensolver.

Finally, we need to load the desired compiler, MPI, and BLAS/LAPACK libraries. A common choice is the use the GNU compilers in combination with OpenMPI and Intel MKL

module load gcc/11.2.0
module load openmpi/4.1.1
module load imkl/2022.0.1
module load cmake/3.22.1
export LD_LIBRARY_PATH=${LIBRARY_PATH}:${CMAKE_LIBRARY_PATH}

export FC=gfortran
export CC=gcc
export CXX=g++
export BLAS_SIZE=8
export SCALAPACK_SIZE=${BLAS_SIZE}
export BLASOPT="-L${MKLROOT}/lib/intel64 -lmkl_gf_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl"
export LAPACK_LIB=${BLASOPT}

If Intel compilers are chosen, then one can load the OneAPI components instead

module load intel/2022.00
module load cmake/3.22.1
export LD_LIBRARY_PATH=${LIBRARY_PATH}:${CMAKE_LIBRARY_PATH}

export FC=ifort
export CC=icc
export CXX=icpc
export BLAS_SIZE=8
export SCALAPACK_SIZE=${BLAS_SIZE}
export BLASOPT="-L${MKLROOT}/lib/intel64 -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"
export LAPACK_LIB=${BLASOPT}

1.2.3 Compiling

There were some issues while compiling the ELPA library due to a faulty configure script. If USE_ELPA=1 is set, then apply execute the following command

cd ${NWCHEM_TOP}/src/libext/elpa
patch <<'EOF'
--- original/build_elpa.sh      2023-06-19 01:14:12.659994922 -0400
+++ build_elpa.sh       2023-06-19 01:03:32.458119095 -0400
@@ -14,10 +14,11 @@
 else
     rm -rf elpa*
 echo    curl -L https://gitlab.mpcdf.mpg.de/elpa/elpa/-/archive/${VERSION}/elpa-${VERSION}.tar.gz -o elpa-${VERSION}.tar.gz
-    curl -L https://gitlab.mpcdf.mpg.de/elpa/elpa/-/archive/${VERSION}/elpa-${VERSION}.tar.gz -o elpa-${VERSION}.tar.gz
+#    curl -L https://gitlab.mpcdf.mpg.de/elpa/elpa/-/archive/${VERSION}/elpa-${VERSION}.tar.gz -o elpa-${VERSION}.tar.gz
+    curl -L https://elpa.mpcdf.mpg.de/software/tarball-archive/Releases/${SHORTVERSION}/elpa-${SHORTVERSION}.tar.gz -o elpa-${SHORTVERSION}.tar.gz
 fi
-tar xzf elpa-${VERSION}.tar.gz
-ln -sf elpa-${VERSION} elpa
+tar xzf elpa-${SHORTVERSION}.tar.gz
+ln -sf elpa-${SHORTVERSION} elpa
 cd elpa
 UNAME_S=$(uname -s)
 if [[ ${UNAME_S} == Linux ]]; then
@@ -112,7 +113,7 @@
 if [[  -z "${FORCETARGET}" ]]; then
 FORCETARGET="-disable-sse -disable-sse-assembly --disable-avx --disable-avx2  --disable-avx512  "
 fi #FORCETARGET
-if [[ "${USE_HWOPT}" == "1" ]] && [[ "${USE_HWOPT}" == "y" ]] &&[[ "${USE_HWOPT}" != "Y" ]] && [[ ${UNAME_S} == Linux ]]; then
+if [[ "${USE_HWOPT}" == "1" ]] || [[ "${USE_HWOPT}" == "y" ]] ||[[ "${USE_HWOPT}" == "Y" ]] && [[ ${UNAME_S} == Linux ]]; then
 if [[ ${UNAME_S} == Linux ]]; then
     CPU_FLAGS=$(cat /proc/cpuinfo | grep flags |tail -n 1)
     CPU_FLAGS_2=$(cat /proc/cpuinfo | grep flags |tail -n 1)
EOF

All the environment should now be ready to begin the NWChem compilation

cd ${NWCHEM_TOP}/src
make nwchem_config
make -j12 &> make.log &

More information about how to obtain and compile NWChem can be found on the NWChem website.

1.3 Running NWChem @ UB CCR

1.3.1 Running conda binary

#!/bin/bash
#SBATCH --account=cyberwksp21
#SBATCH --partition=valhalla  --qos=valhalla
#SBATCH --clusters=faculty
#SBATCH -N 1
#SBATCH --ntasks-per-node=24
#SBATCH -C CPU-E5-2650v4
eval "$(/projects/academic/cyberwksp21/Software/nwchem_conda0/bin/conda shell.bash hook)"
mpirun -n ${SLURM_NTASKS} nwchem [input]

1.3.2 Running native build

#!/bin/bash
#SBATCH --account=cyberwksp21
#SBATCH --partition=valhalla  --qos=valhalla
#SBATCH --clusters=faculty
#SBATCH -N 1
#SBATCH --ntasks-per-node=24
#SBATCH --cpus-per-task=1
#SBATCH -C CPU-E5-2650v4
module load gcc/11.2.0
module load openmpi/4.1.1
module load imkl/2022.0.1
export LD_LIBRARY_PATH=${LIBRARY_PATH}:${CMAKE_LIBRARY_PATH}

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

srun --mpi=pmix_v4 --cpu-bind core \
       ${NWCHEM_TOP}/bin/LINUX64/nwchem [input]