README update (#60)

- README now contains some useful documentation
- added examples to CI

.github/workflows/ci.yml

@@ -78,9 +78,24 @@ jobs:
         if: always()
         run: rm -rf build/
 
+  examples:
+    name: Build examples
+    needs: unit_tests_and_static_analysis
+    runs-on: self-hosted
+    container: kubagalecki/l3ster-ci:latest
+    env:
+      BUILD_TYPE: Release
+    steps:
+      - uses: actions/[email protected]
+      - name: Build examples
+        run: scripts/ci/build-examples.sh
+      - name: Cleanup
+        if: always()
+        run: rm -rf examples/build/
+
   coverage:
     name: Code coverage
-    needs: [ sanitizers, deployment ]
+    needs: [ sanitizers, deployment, examples ]
     runs-on: self-hosted
     container: kubagalecki/l3ster-ci:latest
     env:

README.md

@@ -2,19 +2,174 @@
 [![Tests](https://github.com/kubagalecki/L3STER/workflows/tests/badge.svg)](https://github.com/kubagalecki/L3STER/actions)
 [![codecov](https://codecov.io/gh/kubagalecki/L3STER/branch/main/graph/badge.svg?token=6VT1TVS7FG)](https://codecov.io/gh/kubagalecki/L3STER)
 
-# L3STER
-
-**Currently under construction.**
-
-The name L3STER is derived from "**L**east-**S**quares **S**calable **S**pec**T**ral/*hp* **E**lement f**R**amework".
-The goal of the project is to develop, based on the least-squares spectral/*hp* element method, a scalable C++ framework for the numerical solution of partial differential equations.
-Although the aim is to develop a generic framework, the author's intended classes of applications are as follows:
-- incompressible flow with stabilized boundary conditions
-- optimal control problems involving fluid-structure interaction
-
-Some attractive features of the code include:
-- utilization of state of the art algebraic solvers available in the Trilinos library
-- modern implementation leveraging features from C++20
-- linear scaling with problem size thanks to the synergistic combination of the least-squares method with a multigrid solver (theoretically)
-- L3STER is header-only, allowing you to take full advantage of your compiler's optimizer
-- a low entry barrier - given a set of PDEs and a mesh, you should be able to set up your first simulation within an afternoon!
+# L3STER :cheese:
+
+L3STER (pronounced like the delicious English cheese) stands for "**L**east-**S**quares **S**calable **S**pec**T**ral/*hp* **E**lement f**R**amework".
+L3STER provides a scalable, flexible framework for the solution of systems of partial differential equations.
+Thanks to the employment of the least-squares finite element method, no weak formulation is needed - you can directly implement any first-order PDE:
+
+$$ \mathcal{A}(u) = f \hspace{.1cm} \mathrm{in} \hspace{.1cm} \Omega $$
+
+$$ \mathcal{B}(u) = g \hspace{.1cm} \mathrm{on} \hspace{.1cm} \partial\Omega $$
+
+The guiding philosophy of the project is: *"From a set of PDEs and a mesh to a working simulation within an afternoon!"*
+
+Features of the library include:
+- A modern implementation leveraging C++23
+- Scalability using hybrid parallelism (MPI + multithreading)
+- Computational efficiency thanks to a high-order discretization
+- Mesh import from Gmsh
+- Results export to VTK, simple postprocessing (flux integrals etc.) available natively
+- Easy setup (all dependencies available in Spack)
+
+If you'd like to use L3STER, but you need a different I/O format, please drop us an issue!
+
+## Formulating the system of PDEs
+
+If your equation is of a higher order, you'll first need to recast it by introducing auxiliary unknowns (e.g. gradients).
+At the end of the day, each equation takes the form of:
+
+$$ A_0 u + \left( \sum_{i=1}^{D} A_i \frac{\partial}{\partial x_i} \right) u = f $$
+
+where $D \in \{ 2,3 \}$ is the spatial dimension of the problem,
+$u : \Omega \rightarrow \mathbb{R}^U$ is the unknown vector field,
+$A_i : \Omega \rightarrow \mathbb{R}^{E \times U}$ describe the first-order differential operator,
+$f : \Omega \rightarrow \mathbb{R}^E$ is the source term,
+$E$ is the number of equations, and $U$ the number of unknowns ($E$ and $U$ may not be equal).
+
+### Boundary conditions
+
+You can also use the approach outlined above to describe an arbitrary boundary condition.
+In L3STER, the only difference between domain equations and boundary conditions is that when defining BCs, you gain access to the boundary normal vector.
+
+Only Dirichlet BCs are treated in a special fashion.
+This is because they are strongly imposed on the resulting algebraic system.
+It is possible to define them in the equation sense, but this is not recommended.
+
+### Time-dependent problems
+
+Note that this formulation does not contain a time derivative.
+If you are solving an unsteady problem, you'll first need to discretize your problem in time.
+For example, you can use the backward Euler scheme:
+
+$$ \frac{\partial u}{\partial t} \approx \frac{u_{n+1} - u_n}{\Delta t} $$
+
+You can then add $I \Delta t$ to $A_0$ and add $u_n / \Delta t$ to the source term to obtain a PDE for $u$ at the next time step.
+
+### Non-linear problems
+
+If your problem is non-linear, you'll first need to linearize it, e.g., using Newton's method.
+You can then iterate to obtain your solution.
+L3STER provides a convenient way of accessing previously computed fields (and their derivatives) when defining your equations.
+This mechanism can also be used for time-stepping (where the previous value(s) of $u$ are needed) or dependencies between different systems, e.g., solving an advection-diffusion equation on a previously computed flow field.
+
+## Installation
+
+L3STER is a header-only library, which means you don't need to install it.
+Simply point your CMake project at the directory where L3STER resides and use the provided target:
+
+```cmake
+# In your project's CMakeLists.txt
+add_subdirectory( path/to/L3STER L3STER-bin )
+target_link_libraries( my-executable-target L3STER )
+```
+
+That being said, L3STER has several dependencies, which will need to be installed first:
+
+- CMake 3.24 or newer
+- A C++ 23 compiler, gcc 13 or newer will work
+- MPI
+- Hwloc
+- Metis
+- Trilinos 14.0 or newer. The following packages are currently used:
+  - Kokkos (which can be built separately from Trilinos)
+  - Tpetra
+  - Belos - optional, needed for iterative solvers
+  - Amesos2 - optional, needed for direct solvers
+  - Ifpack2 - optional, used for preconditioners (L3STER provides a few simple ones natively)
+- Intel OneTBB
+- Eigen version 3.4
+
+All of these dependencies are available via [Spack](https://spack.readthedocs.io/en/latest/index.html).
+You can easily install them as follows:
+
+```bash
+# Get spack and set up the shell
+git clone -c feature.manyFiles=true https://github.com/spack/spack.git
+cd spack
+git checkout tags/releases/latest
+. share/spack/setup-env.sh # consider adding this to your .bashrc
+
+# Find some common packages so that spack doesn't have to build them from scratch (saves time)
+spack external find binutils cmake coreutils curl diffutils findutils git gmake openssh perl python sed tar
+
+# If you have a sufficiently recent compiler, skip this section
+spack install gcc
+spack load gcc
+spack compiler find
+
+# Create a spack environment for L3STER dependencies and install them
+# Some of the libraries listed above are not mentioned explicitly, they will be built as dependencies of other packages
+spack env create l3ster
+spacktivate l3ster
+spack add eigen intel-oneapi-tbb parmetis kokkos+openmp trilinos cxxstd=17 +openmp +amesos2 +belos +tpetra +ifpack2
+spack concretize
+spack install
+
+# Cleanup to save disc space
+spack gc -y
+spack clean -a
+```
+
+When using L3STER, all you need to do is call `spacktivate l3ster` before invoking CMake.
+
+> Your cluster administrators may provide a global spack instance.
+> You can take advantage of it using spack chaining.
+> If not, you should use the MPI installation provided for you by the admins, not build your own.
+> Please consult the spack documentation on how to use external packages.
+
+## Running L3STER applications
+
+L3STER follows the MPI+X paradigm (hybrid parallelism).
+It uses TBB for multithreading, and MPI for multiprocessing.
+It is recommended that you launch one MPI rank per CPU, not CPU *core*.
+
+> Trilinos uses OpenMP for multithreading. L3STER and Trilinos parallel regions never overlap, so oversubscription is not an issue.
+
+> L3STER uses Hwloc to detect your machine's topology and limits the SMT parallelism where appropriate. You don't need to worry about hyperthreading, L3STER will just do the right thing.
+
+### Desktops
+
+On desktops, where presumably you have only one CPU, you can launch your application directly.
+Parallelism is achieved via multithreading on a single MPI rank.
+
+```bash
+./my-l3ster-app
+```
+
+Note that you still need to build with MPI (sorry).
+
+### Clusters
+
+Example slurm script demonstrating L3STER usage:
+
+```bash
+#SBATCH -N [number of nodes you'd like to run on]
+#SBATCH -n [number of nodes you'd like to run on multiplied by number of sockets/node (often 2)]
+#SBATCH -c [number of cores per socket]
+#SBATCH --ntasks-per-socket 1 
+
+# Set up environment via spack and/or system modules
+
+cd /my/project/dir
+mkdir build
+cd build
+cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpic++ .. || exit 1
+cmake --build .                                                 || exit 1
+srun my-l3ster-app
+```
+
+## Usage
+
+We are working on fully documenting the L3STER library.
+For the time being, please refer to the examples.

cmake/ImportLibrary.cmake

@@ -17,14 +17,14 @@ function( importLibrary lib_name )
 
     unset( ${lib_name}_LIB CACHE )
     find_library( ${lib_name}_LIB ${lib_name} NO_CACHE )
-    if ( ${lib_name}_LIB-NOTFOUND )
+    if ( ${${lib_name}_LIB} STREQUAL "${lib_name}_LIB-NOTFOUND" )
         message( STATUS "Detecting ${lib_name} - not found" )
         if ( required )
-            message( FATAL_ERROR "The library \"${lib_name}\" was not found. Try setting ${lib_name}_ROOT=/path/to/library" )
+            message( SEND_ERROR "The library \"${lib_name}\" was not found. Try setting ${lib_name}_ROOT=/path/to/library" )
         else ()
             set( ${lib_name}_FOUND "FALSE" PARENT_SCOPE )
-            return()
         endif ()
+        return()
     else ()
         message( STATUS "Detecting ${lib_name} - found: ${${lib_name}_LIB}" )
     endif ()

examples/hello-world/CMakeLists.txt → examples/01-hello-world/CMakeLists.txt

@@ -1,4 +1,4 @@
-cmake_minimum_required( VERSION 3.19 )
+cmake_minimum_required( VERSION 3.24 )
 project( hello-world-l3ster )
 add_subdirectory( ../.. L3STER-bin )
 add_executable( hello-world source.cpp )

examples/advection-2D/CMakeLists.txt → examples/02-advection-2D/CMakeLists.txt

@@ -1,4 +1,4 @@
-cmake_minimum_required( VERSION 3.19 )
+cmake_minimum_required( VERSION 3.24 )
 project( advection-2D )
 add_subdirectory( ../.. L3STER-bin )
 add_executable( advection-2D source.cpp )

examples/karman-2D/CMakeLists.txt → examples/03-karman-2D/CMakeLists.txt

@@ -1,4 +1,4 @@
-cmake_minimum_required( VERSION 3.19 )
+cmake_minimum_required( VERSION 3.24 )
 project( karman-2D )
 add_subdirectory( ../.. L3STER-bin )
 add_executable( karman-2D source.cpp )

examples/CMakeLists.txt

@@ -0,0 +1,19 @@
+# Note: this file is only intended to be used by the CI infrastructure, to make sure all examples build correctly
+# If you are a human reader, please consider the individual examples, including their respective CMakeLists.txt files
+
+cmake_minimum_required( VERSION 3.24 )
+project( L3STER-examples-all )
+add_subdirectory( .. L3STER-bin )
+file( GLOB subdirs ${CMAKE_CURRENT_SOURCE_DIR}/* )
+foreach ( dir IN LISTS subdirs )
+    if ( IS_DIRECTORY "${dir}" )
+        cmake_path( GET dir STEM name )
+        if ( "${name}" STREQUAL build )
+            continue()
+        endif ()
+        message( STATUS "Found example: ${name}" )
+        file( GLOB src "${dir}/*.cpp" )
+        add_executable( ${name} ${src} )
+        target_link_libraries( ${name} L3STER )
+    endif ()
+endforeach ()

scripts/ci/build-examples.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+. /spack/share/spack/setup-env.sh
+spack env activate build-env
+
+cd examples || exit 1
+mkdir -p build
+cd build || exit 1
+cmake \
+  -DCMAKE_BUILD_TYPE="$BUILD_TYPE" \
+  -DCMAKE_CXX_COMPILER=mpic++ \
+  .. || exit 1
+cmake --build . -- -j "$(grep -c ^processor /proc/cpuinfo)" || exit 1

tests/MatrixFreeTests.cpp

@@ -13,7 +13,7 @@ using namespace lstr;
 using namespace lstr::algsys;
 using namespace lstr::mesh;
 
-constexpr auto node_dist = std::array{0., 1., 2., 3., 4., 5., 6.};
+constexpr auto node_dist = std::array{0., 1., 2., 3., 4.};
 
 auto makeMesh(const MpiComm& comm, auto probdef_ctwrpr)
 {

tests/MpiImportExportTest.cpp

@@ -12,7 +12,7 @@ using GID                       = long;
 using LID                       = int;
 using Scalar                    = GID;
 constexpr size_t mv_cols        = 5;
-constexpr size_t elems_per_rank = 100'000;
+constexpr size_t elems_per_rank = 1000;
 
 void runTest(const MpiComm&                    comm,
              Import< Scalar, LID >&            importer,