diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e7f00907..471c7a66 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -78,9 +78,24 @@ jobs: if: always() run: rm -rf build/ + examples: + name: Build examples + needs: unit_tests_and_static_analysis + runs-on: self-hosted + container: kubagalecki/l3ster-ci:latest + env: + BUILD_TYPE: Release + steps: + - uses: actions/checkout@v4.2.0 + - name: Build examples + run: scripts/ci/build-examples.sh + - name: Cleanup + if: always() + run: rm -rf examples/build/ + coverage: name: Code coverage - needs: [ sanitizers, deployment ] + needs: [ sanitizers, deployment, examples ] runs-on: self-hosted container: kubagalecki/l3ster-ci:latest env: diff --git a/README.md b/README.md index 465d1c6f..010b5ff0 100644 --- a/README.md +++ b/README.md @@ -2,19 +2,174 @@ [![Tests](https://github.com/kubagalecki/L3STER/workflows/tests/badge.svg)](https://github.com/kubagalecki/L3STER/actions) [![codecov](https://codecov.io/gh/kubagalecki/L3STER/branch/main/graph/badge.svg?token=6VT1TVS7FG)](https://codecov.io/gh/kubagalecki/L3STER) -# L3STER - -**Currently under construction.** - -The name L3STER is derived from "**L**east-**S**quares **S**calable **S**pec**T**ral/*hp* **E**lement f**R**amework". -The goal of the project is to develop, based on the least-squares spectral/*hp* element method, a scalable C++ framework for the numerical solution of partial differential equations. -Although the aim is to develop a generic framework, the author's intended classes of applications are as follows: -- incompressible flow with stabilized boundary conditions -- optimal control problems involving fluid-structure interaction - -Some attractive features of the code include: -- utilization of state of the art algebraic solvers available in the Trilinos library -- modern implementation leveraging features from C++20 -- linear scaling with problem size thanks to the synergistic combination of the least-squares method with a multigrid solver (theoretically) -- L3STER is header-only, allowing you to take full advantage of your compiler's optimizer -- a low entry barrier - given a set of PDEs and a mesh, you should be able to set up your first simulation within an afternoon! +# L3STER :cheese: + +L3STER (pronounced like the delicious English cheese) stands for "**L**east-**S**quares **S**calable **S**pec**T**ral/*hp* **E**lement f**R**amework". +L3STER provides a scalable, flexible framework for the solution of systems of partial differential equations. +Thanks to the employment of the least-squares finite element method, no weak formulation is needed - you can directly implement any first-order PDE: + +$$ \mathcal{A}(u) = f \hspace{.1cm} \mathrm{in} \hspace{.1cm} \Omega $$ + +$$ \mathcal{B}(u) = g \hspace{.1cm} \mathrm{on} \hspace{.1cm} \partial\Omega $$ + +The guiding philosophy of the project is: *"From a set of PDEs and a mesh to a working simulation within an afternoon!"* + +Features of the library include: +- A modern implementation leveraging C++23 +- Scalability using hybrid parallelism (MPI + multithreading) +- Computational efficiency thanks to a high-order discretization +- Mesh import from Gmsh +- Results export to VTK, simple postprocessing (flux integrals etc.) available natively +- Easy setup (all dependencies available in Spack) + +If you'd like to use L3STER, but you need a different I/O format, please drop us an issue! + +## Formulating the system of PDEs + +If your equation is of a higher order, you'll first need to recast it by introducing auxiliary unknowns (e.g. gradients). +At the end of the day, each equation takes the form of: + +$$ A_0 u + \left( \sum_{i=1}^{D} A_i \frac{\partial}{\partial x_i} \right) u = f $$ + +where $D \in \{ 2,3 \}$ is the spatial dimension of the problem, +$u : \Omega \rightarrow \mathbb{R}^U$ is the unknown vector field, +$A_i : \Omega \rightarrow \mathbb{R}^{E \times U}$ describe the first-order differential operator, +$f : \Omega \rightarrow \mathbb{R}^E$ is the source term, +$E$ is the number of equations, and $U$ the number of unknowns ($E$ and $U$ may not be equal). + +### Boundary conditions + +You can also use the approach outlined above to describe an arbitrary boundary condition. +In L3STER, the only difference between domain equations and boundary conditions is that when defining BCs, you gain access to the boundary normal vector. + +Only Dirichlet BCs are treated in a special fashion. +This is because they are strongly imposed on the resulting algebraic system. +It is possible to define them in the equation sense, but this is not recommended. + +### Time-dependent problems + +Note that this formulation does not contain a time derivative. +If you are solving an unsteady problem, you'll first need to discretize your problem in time. +For example, you can use the backward Euler scheme: + +$$ \frac{\partial u}{\partial t} \approx \frac{u_{n+1} - u_n}{\Delta t} $$ + +You can then add $I \Delta t$ to $A_0$ and add $u_n / \Delta t$ to the source term to obtain a PDE for $u$ at the next time step. + +### Non-linear problems + +If your problem is non-linear, you'll first need to linearize it, e.g., using Newton's method. +You can then iterate to obtain your solution. +L3STER provides a convenient way of accessing previously computed fields (and their derivatives) when defining your equations. +This mechanism can also be used for time-stepping (where the previous value(s) of $u$ are needed) or dependencies between different systems, e.g., solving an advection-diffusion equation on a previously computed flow field. + +## Installation + +L3STER is a header-only library, which means you don't need to install it. +Simply point your CMake project at the directory where L3STER resides and use the provided target: + +```cmake +# In your project's CMakeLists.txt +add_subdirectory( path/to/L3STER L3STER-bin ) +target_link_libraries( my-executable-target L3STER ) +``` + +That being said, L3STER has several dependencies, which will need to be installed first: + +- CMake 3.24 or newer +- A C++ 23 compiler, gcc 13 or newer will work +- MPI +- Hwloc +- Metis +- Trilinos 14.0 or newer. The following packages are currently used: + - Kokkos (which can be built separately from Trilinos) + - Tpetra + - Belos - optional, needed for iterative solvers + - Amesos2 - optional, needed for direct solvers + - Ifpack2 - optional, used for preconditioners (L3STER provides a few simple ones natively) +- Intel OneTBB +- Eigen version 3.4 + +All of these dependencies are available via [Spack](https://spack.readthedocs.io/en/latest/index.html). +You can easily install them as follows: + +```bash +# Get spack and set up the shell +git clone -c feature.manyFiles=true https://github.com/spack/spack.git +cd spack +git checkout tags/releases/latest +. share/spack/setup-env.sh # consider adding this to your .bashrc + +# Find some common packages so that spack doesn't have to build them from scratch (saves time) +spack external find binutils cmake coreutils curl diffutils findutils git gmake openssh perl python sed tar + +# If you have a sufficiently recent compiler, skip this section +spack install gcc +spack load gcc +spack compiler find + +# Create a spack environment for L3STER dependencies and install them +# Some of the libraries listed above are not mentioned explicitly, they will be built as dependencies of other packages +spack env create l3ster +spacktivate l3ster +spack add eigen intel-oneapi-tbb parmetis kokkos+openmp trilinos cxxstd=17 +openmp +amesos2 +belos +tpetra +ifpack2 +spack concretize +spack install + +# Cleanup to save disc space +spack gc -y +spack clean -a +``` + +When using L3STER, all you need to do is call `spacktivate l3ster` before invoking CMake. + +> Your cluster administrators may provide a global spack instance. +> You can take advantage of it using spack chaining. +> If not, you should use the MPI installation provided for you by the admins, not build your own. +> Please consult the spack documentation on how to use external packages. + +## Running L3STER applications + +L3STER follows the MPI+X paradigm (hybrid parallelism). +It uses TBB for multithreading, and MPI for multiprocessing. +It is recommended that you launch one MPI rank per CPU, not CPU *core*. + +> Trilinos uses OpenMP for multithreading. L3STER and Trilinos parallel regions never overlap, so oversubscription is not an issue. + +> L3STER uses Hwloc to detect your machine's topology and limits the SMT parallelism where appropriate. You don't need to worry about hyperthreading, L3STER will just do the right thing. + +### Desktops + +On desktops, where presumably you have only one CPU, you can launch your application directly. +Parallelism is achieved via multithreading on a single MPI rank. + +```bash +./my-l3ster-app +``` + +Note that you still need to build with MPI (sorry). + +### Clusters + +Example slurm script demonstrating L3STER usage: + +```bash +#SBATCH -N [number of nodes you'd like to run on] +#SBATCH -n [number of nodes you'd like to run on multiplied by number of sockets/node (often 2)] +#SBATCH -c [number of cores per socket] +#SBATCH --ntasks-per-socket 1 + +# Set up environment via spack and/or system modules + +cd /my/project/dir +mkdir build +cd build +cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpic++ .. || exit 1 +cmake --build . || exit 1 +srun my-l3ster-app +``` + +## Usage + +We are working on fully documenting the L3STER library. +For the time being, please refer to the examples. \ No newline at end of file diff --git a/cmake/ImportLibrary.cmake b/cmake/ImportLibrary.cmake index 9146a945..ed1ea306 100644 --- a/cmake/ImportLibrary.cmake +++ b/cmake/ImportLibrary.cmake @@ -17,14 +17,14 @@ function( importLibrary lib_name ) unset( ${lib_name}_LIB CACHE ) find_library( ${lib_name}_LIB ${lib_name} NO_CACHE ) - if ( ${lib_name}_LIB-NOTFOUND ) + if ( ${${lib_name}_LIB} STREQUAL "${lib_name}_LIB-NOTFOUND" ) message( STATUS "Detecting ${lib_name} - not found" ) if ( required ) - message( FATAL_ERROR "The library \"${lib_name}\" was not found. Try setting ${lib_name}_ROOT=/path/to/library" ) + message( SEND_ERROR "The library \"${lib_name}\" was not found. Try setting ${lib_name}_ROOT=/path/to/library" ) else () set( ${lib_name}_FOUND "FALSE" PARENT_SCOPE ) - return() endif () + return() else () message( STATUS "Detecting ${lib_name} - found: ${${lib_name}_LIB}" ) endif () diff --git a/examples/hello-world/CMakeLists.txt b/examples/01-hello-world/CMakeLists.txt similarity index 79% rename from examples/hello-world/CMakeLists.txt rename to examples/01-hello-world/CMakeLists.txt index 2a650356..f58544bf 100644 --- a/examples/hello-world/CMakeLists.txt +++ b/examples/01-hello-world/CMakeLists.txt @@ -1,4 +1,4 @@ -cmake_minimum_required( VERSION 3.19 ) +cmake_minimum_required( VERSION 3.24 ) project( hello-world-l3ster ) add_subdirectory( ../.. L3STER-bin ) add_executable( hello-world source.cpp ) diff --git a/examples/hello-world/source.cpp b/examples/01-hello-world/source.cpp similarity index 100% rename from examples/hello-world/source.cpp rename to examples/01-hello-world/source.cpp diff --git a/examples/advection-2D/CMakeLists.txt b/examples/02-advection-2D/CMakeLists.txt similarity index 79% rename from examples/advection-2D/CMakeLists.txt rename to examples/02-advection-2D/CMakeLists.txt index 671e7f9f..e2acba50 100644 --- a/examples/advection-2D/CMakeLists.txt +++ b/examples/02-advection-2D/CMakeLists.txt @@ -1,4 +1,4 @@ -cmake_minimum_required( VERSION 3.19 ) +cmake_minimum_required( VERSION 3.24 ) project( advection-2D ) add_subdirectory( ../.. L3STER-bin ) add_executable( advection-2D source.cpp ) diff --git a/examples/advection-2D/README.md b/examples/02-advection-2D/README.md similarity index 100% rename from examples/advection-2D/README.md rename to examples/02-advection-2D/README.md diff --git a/examples/advection-2D/source.cpp b/examples/02-advection-2D/source.cpp similarity index 100% rename from examples/advection-2D/source.cpp rename to examples/02-advection-2D/source.cpp diff --git a/examples/karman-2D/CMakeLists.txt b/examples/03-karman-2D/CMakeLists.txt similarity index 78% rename from examples/karman-2D/CMakeLists.txt rename to examples/03-karman-2D/CMakeLists.txt index 06d55b26..fd8f7f49 100644 --- a/examples/karman-2D/CMakeLists.txt +++ b/examples/03-karman-2D/CMakeLists.txt @@ -1,4 +1,4 @@ -cmake_minimum_required( VERSION 3.19 ) +cmake_minimum_required( VERSION 3.24 ) project( karman-2D ) add_subdirectory( ../.. L3STER-bin ) add_executable( karman-2D source.cpp ) diff --git a/examples/karman-2D/README.md b/examples/03-karman-2D/README.md similarity index 100% rename from examples/karman-2D/README.md rename to examples/03-karman-2D/README.md diff --git a/examples/karman-2D/karman.msh b/examples/03-karman-2D/karman.msh similarity index 100% rename from examples/karman-2D/karman.msh rename to examples/03-karman-2D/karman.msh diff --git a/examples/karman-2D/source.cpp b/examples/03-karman-2D/source.cpp similarity index 100% rename from examples/karman-2D/source.cpp rename to examples/03-karman-2D/source.cpp diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt new file mode 100644 index 00000000..834dc72e --- /dev/null +++ b/examples/CMakeLists.txt @@ -0,0 +1,19 @@ +# Note: this file is only intended to be used by the CI infrastructure, to make sure all examples build correctly +# If you are a human reader, please consider the individual examples, including their respective CMakeLists.txt files + +cmake_minimum_required( VERSION 3.24 ) +project( L3STER-examples-all ) +add_subdirectory( .. L3STER-bin ) +file( GLOB subdirs ${CMAKE_CURRENT_SOURCE_DIR}/* ) +foreach ( dir IN LISTS subdirs ) + if ( IS_DIRECTORY "${dir}" ) + cmake_path( GET dir STEM name ) + if ( "${name}" STREQUAL build ) + continue() + endif () + message( STATUS "Found example: ${name}" ) + file( GLOB src "${dir}/*.cpp" ) + add_executable( ${name} ${src} ) + target_link_libraries( ${name} L3STER ) + endif () +endforeach () \ No newline at end of file diff --git a/scripts/ci/build-examples.sh b/scripts/ci/build-examples.sh new file mode 100755 index 00000000..9698789a --- /dev/null +++ b/scripts/ci/build-examples.sh @@ -0,0 +1,13 @@ +#!/bin/bash + +. /spack/share/spack/setup-env.sh +spack env activate build-env + +cd examples || exit 1 +mkdir -p build +cd build || exit 1 +cmake \ + -DCMAKE_BUILD_TYPE="$BUILD_TYPE" \ + -DCMAKE_CXX_COMPILER=mpic++ \ + .. || exit 1 +cmake --build . -- -j "$(grep -c ^processor /proc/cpuinfo)" || exit 1 diff --git a/tests/MatrixFreeTests.cpp b/tests/MatrixFreeTests.cpp index 1c358a92..d70c42d7 100644 --- a/tests/MatrixFreeTests.cpp +++ b/tests/MatrixFreeTests.cpp @@ -13,7 +13,7 @@ using namespace lstr; using namespace lstr::algsys; using namespace lstr::mesh; -constexpr auto node_dist = std::array{0., 1., 2., 3., 4., 5., 6.}; +constexpr auto node_dist = std::array{0., 1., 2., 3., 4.}; auto makeMesh(const MpiComm& comm, auto probdef_ctwrpr) { diff --git a/tests/MpiImportExportTest.cpp b/tests/MpiImportExportTest.cpp index 18118ba2..9399c8d0 100644 --- a/tests/MpiImportExportTest.cpp +++ b/tests/MpiImportExportTest.cpp @@ -12,7 +12,7 @@ using GID = long; using LID = int; using Scalar = GID; constexpr size_t mv_cols = 5; -constexpr size_t elems_per_rank = 100'000; +constexpr size_t elems_per_rank = 1000; void runTest(const MpiComm& comm, Import< Scalar, LID >& importer,