Compiling OpenFOAM with Spack

There are some issues when installing OpenFOAM of OpenCFD with Spack.

An environment was set up to manage the packages with Spack following the instruction in the UL HPC tutorial. In short, the following dependencies where defined for the SPACK packages,

(access node)$ cat << EOF >> $SPACK_ROOT/etc/spack/packages.yaml
        - spec: slurm@22.05.5
          prefix: /usr
        buildable: False
        - spec: libevent@2.1.8
          prefix: /usr
        buildable: False
        - spec: pmix@4.2.3 
          prefix: /usr
        buildable: False
        - spec: hwloc@2.2.0
          prefix: /usr
        buildable: False

and the following options were defined for the build cache directories,

(access node)$ cat << EOF >> $SPACK_ROOT/etc/spack/config.yaml
        - /dev/shm/$user/spack-stage

to speed-up compilation using the ramdisk file system. Before proceeding to the installation of OpenFOAM the desired version of Open MPI is installed with the system compiler:

(compute node)$ spack install -j openmpi@4.0.5 +pmi schedulers=slurm ^pmix@4.2.3 ^hwloc@2.2.0

The distribution of OpenFOAM is installed using the system compiler as well. The available components of OpenFOAM are:

(access node)$ spack list openfoam@2306
    Name [Default]            When    Allowed values    Description
    ======================    ====    ==============    ==================================================

    build_system [generic]    --      generic           Build systems supported by the package
    int64 [off]               --      on, off           With 64-bit labels
    kahip [off]               --      on, off           With kahip decomposition
    knl [off]                 --      on, off           Use KNL compiler settings
    metis [off]               --      on, off           With metis decomposition
    mgridgen [off]            --      on, off           With mgridgen support
    paraview [off]            --      on, off           Build paraview plugins and runtime post-processing
    precision [dp]            --      sp, dp, spdp      Precision option
    scotch [on]               --      on, off           With scotch/ptscotch decomposition
    source [on]               --      on, off           Install library/application sources and tutorials
    vtk [off]                 --      on, off           With VTK runTimePostProcessing
    zoltan [off]              --      on, off           With zoltan renumbering

OpenFOAM is installed using all the available partitioners (metis, scotch, zoltan, kahip) and the multi-grid generator components (mgridgen, section of the documentation) with the command:

(compute node)$ spack install -j openfoam@2306 +source precision=dp +metis +scotch +zoltan +kahip +mgridgen ~knl ~int64 ~paraview ^openmpi@4.0.5 

However, the resulting installation is missing multiple components. For instance, executing the manual post installation verification in doc/,

# Create the user "run" directory:
mkdir -p "$FOAM_RUN"
# Change to the user "run" directory:
# Copy tutorial
cp -r "$FOAM_TUTORIALS"/incompressible/simpleFoam/pitzDaily ./
# Run the tutorial
( cd pitzDaily && blockMesh && simpleFoam )

both the mesh generator blockMesh and the solver simpleFoam are missing.

It turns out that the METIS partitioner causes the installation of multiple components to fail. Recompiling with only the SCOTCH partitioner,

(compute node)$  spack install openfoam@2306  +source +scotch +mgridgen precision=dp ~int64 ~kahip ~knl ~metis ~paraview ~vtk ~zoltan ^openmpi@4.0.5

solves the issue. All solver components are installed, but a few utilities are still missing.

We use the following script to detect all missing components:


set -euo pipefail

declare variant="${1}" # e.g. @2306%gcc@8.5.0 @2306+mgridgen
declare type="${2}" # solvers, utilities

find_makefile_directories() {
    local directory="${1}"

    find "${directory}" -type d | grep -E '\/Make$' || true

extract_executable_name() {
    local makefile_directory="${1}"

    cat "${makefile_directory}/files" | ( grep -E '^EXE[[:space:]]*=' || true ) | sed 's/^EXE[[:space:]]*=[[:space:]]*\$(FOAM_APPBIN)\///g'

main() {
    local variant="${1}"
    local type="${2}"

    local foam_location="$(spack location -i openfoam"${variant}")"

    local pagkage_directory=""
    local executable_name=""
    local installed=""
    local package_makefile_directory=""
    while read -r package_makefile_directory; do
        executable_name=$( extract_executable_name "${package_makefile_directory}" )

        if [ -n "${executable_name}" ]; then
            if [ -x "${foam_location}/platforms/linux64GccDPInt32-spack/bin/${executable_name}" ]; then
            package_directory="$( echo "${package_makefile_directory}" | xargs -I % dirname % )"

            echo "${installed} : ${executable_name} => ${package_directory#${foam_location}}"
    done < <( find_makefile_directories "${foam_location}/applications/${type}" )

main "${variant}" "${type}"

A simple run of the script indicates that the following utilities are missing:

(access node)$ ./list_spack_openfoam utilities @2306 | grep -E '^N'
N : addr2line => /applications/utilities/miscellaneous/OSspecific/addr2line
N : foamyHexMeshSurfaceSimplify => /applications/utilities/mesh/generation/foamyMesh/foamyHexMeshSurfaceSimplify
N : foamyHexMeshBackgroundMesh => /applications/utilities/mesh/generation/foamyMesh/foamyHexMeshBackgroundMesh
N : cellSizeAndAlignmentGrid => /applications/utilities/mesh/generation/foamyMesh/cellSizeAndAlignmentGrid
N : foamToCcm => /applications/utilities/mesh/conversion/ccm/foamToCcm
N : ccmToFoam => /applications/utilities/mesh/conversion/ccm/ccmToFoam

The root cause is that the OpenFOAM build system proceeds to the whole build and ignore compilation errors. A failed OpenFOAM build just end successfully!!!

EasyBuild mitigates this issue with sanity checks that ensure that a specific list of executables and libraries are present in the installation directory.
cf line 412

Is there a similar mechanism in Spack?

Yes indeed, the OpenFOAM build system does ignore failed components. We are building the COM variant, but it behaves the same as the ORG variant.

We have to use Spack since the COM variant is not readily available in EasyBuild. Spack does not implement any checks for the final outcome at the moment, so we have to check the resulting build manually.