There are some issues when installing OpenFOAM of OpenCFD with Spack.
An environment was set up to manage the packages with Spack following the instruction in the UL HPC tutorial. In short, the following dependencies where defined for the SPACK packages,
(access node)$ cat << EOF >> $SPACK_ROOT/etc/spack/packages.yaml
packages:
slurm:
externals:
- spec: slurm@22.05.5
prefix: /usr
buildable: False
libevent:
externals:
- spec: libevent@2.1.8
prefix: /usr
buildable: False
pmix:
externals:
- spec: pmix@4.2.3
prefix: /usr
buildable: False
hwloc:
externals:
- spec: hwloc@2.2.0
prefix: /usr
buildable: False
EOF
and the following options were defined for the build cache directories,
(access node)$ cat << EOF >> $SPACK_ROOT/etc/spack/config.yaml
config:
build_stage:
- /dev/shm/$user/spack-stage
EOF
to speed-up compilation using the ramdisk file system. Before proceeding to the installation of OpenFOAM the desired version of Open MPI is installed with the system compiler:
(compute node)$ spack install -j openmpi@4.0.5 +pmi schedulers=slurm ^pmix@4.2.3 ^hwloc@2.2.0
The distribution of OpenFOAM is installed using the system compiler as well. The available components of OpenFOAM are:
(access node)$ spack list openfoam@2306
...
Variants:
Name [Default] When Allowed values Description
====================== ==== ============== ==================================================
build_system [generic] -- generic Build systems supported by the package
int64 [off] -- on, off With 64-bit labels
kahip [off] -- on, off With kahip decomposition
knl [off] -- on, off Use KNL compiler settings
metis [off] -- on, off With metis decomposition
mgridgen [off] -- on, off With mgridgen support
paraview [off] -- on, off Build paraview plugins and runtime post-processing
precision [dp] -- sp, dp, spdp Precision option
scotch [on] -- on, off With scotch/ptscotch decomposition
source [on] -- on, off Install library/application sources and tutorials
vtk [off] -- on, off With VTK runTimePostProcessing
zoltan [off] -- on, off With zoltan renumbering
...
OpenFOAM is installed using all the available partitioners (metis, scotch, zoltan, kahip) and the multi-grid generator components (mgridgen, section 6.3.1.4 of the documentation) with the command:
(compute node)$ spack install -j openfoam@2306 +source precision=dp +metis +scotch +zoltan +kahip +mgridgen ~knl ~int64 ~paraview ^openmpi@4.0.5
However, the resulting installation is missing multiple components. For instance, executing the manual post installation verification in doc/Build.md,
# Create the user "run" directory:
mkdir -p "$FOAM_RUN"
# Change to the user "run" directory:
run
# Copy tutorial
cp -r "$FOAM_TUTORIALS"/incompressible/simpleFoam/pitzDaily ./
# Run the tutorial
( cd pitzDaily && blockMesh && simpleFoam )
both the mesh generator blockMesh and the solver simpleFoam are missing.
It turns out that the METIS partitioner causes the installation of multiple components to fail. Recompiling with only the SCOTCH partitioner,
(compute node)$ spack install openfoam@2306 +source +scotch +mgridgen precision=dp ~int64 ~kahip ~knl ~metis ~paraview ~vtk ~zoltan ^openmpi@4.0.5
solves the issue. All solver components are installed, but a few utilities are still missing.
We use the following script to detect all missing components:
#!/usr/bin/bash
set -euo pipefail
declare variant="${1}" # e.g. @2306%gcc@8.5.0 @2306+mgridgen
declare type="${2}" # solvers, utilities
find_makefile_directories() {
local directory="${1}"
find "${directory}" -type d | grep -E '\/Make$' || true
}
extract_executable_name() {
local makefile_directory="${1}"
cat "${makefile_directory}/files" | ( grep -E '^EXE[[:space:]]*=' || true ) | sed 's/^EXE[[:space:]]*=[[:space:]]*\$(FOAM_APPBIN)\///g'
}
main() {
local variant="${1}"
local type="${2}"
local foam_location="$(spack location -i openfoam"${variant}")"
local pagkage_directory=""
local executable_name=""
local installed=""
local package_makefile_directory=""
while read -r package_makefile_directory; do
executable_name=$( extract_executable_name "${package_makefile_directory}" )
if [ -n "${executable_name}" ]; then
if [ -x "${foam_location}/platforms/linux64GccDPInt32-spack/bin/${executable_name}" ]; then
installed="Y"
else
installed="N"
fi
package_directory="$( echo "${package_makefile_directory}" | xargs -I % dirname % )"
echo "${installed} : ${executable_name} => ${package_directory#${foam_location}}"
fi
done < <( find_makefile_directories "${foam_location}/applications/${type}" )
}
main "${variant}" "${type}"
A simple run of the script indicates that the following utilities are missing:
(access node)$ ./list_spack_openfoam utilities @2306 | grep -E '^N'
N : addr2line => /applications/utilities/miscellaneous/OSspecific/addr2line
N : foamyHexMeshSurfaceSimplify => /applications/utilities/mesh/generation/foamyMesh/foamyHexMeshSurfaceSimplify
N : foamyHexMeshBackgroundMesh => /applications/utilities/mesh/generation/foamyMesh/foamyHexMeshBackgroundMesh
N : cellSizeAndAlignmentGrid => /applications/utilities/mesh/generation/foamyMesh/cellSizeAndAlignmentGrid
N : foamToCcm => /applications/utilities/mesh/conversion/ccm/foamToCcm
N : ccmToFoam => /applications/utilities/mesh/conversion/ccm/ccmToFoam