From 1ce4f5f31b0976331e14932a5160ab377c8f5bb1 Mon Sep 17 00:00:00 2001 From: Rasmus Kronberg <43936697+rkronberg@users.noreply.github.com> Date: Mon, 13 May 2024 16:31:41 +0300 Subject: [PATCH] Merge overlapping CMake tutorials (#332) * Suggested changes for compilers exercise * projappl note * rephrase * Merge overlapping CMake tutorials * Merge overlapping CMake tutorials * add link * fix index * fixes --- part-2/installing/cmake.md | 231 +++++++++++++++++++++++-------------- part-2/installing/hpc.md | 115 ------------------ part-2/installing/index.md | 3 +- 3 files changed, 145 insertions(+), 204 deletions(-) delete mode 100644 part-2/installing/hpc.md diff --git a/part-2/installing/cmake.md b/part-2/installing/cmake.md index 29321333..cc8dcf3c 100644 --- a/part-2/installing/cmake.md +++ b/part-2/installing/cmake.md @@ -11,93 +11,150 @@ permalink: /hands-on/installing/installing_cmake.html # Installing using CMake -💬 CMake is an extensible, open-source system that manages the build process in an operating system and in a compiler-independent manner. +💬 CMake is an extensible, open-source software build system. CMake uses simple +`CMakeLists.txt` configuration files placed in each source directory to +generate standard `Makefile` build files. CMake generates a native build +environment that will compile source code, create libraries, generate wrappers +and build executable binaries in arbitrary combinations. + +## Preparations + +💬 In this tutorial we'll use CMake to install a simple C++ Hello World +application. + +1. It is recommended to use the fast local disk when compiling software on CSC + supercomputers to move I/O load away from the parallel file system. Go there + and download the source code from Allas: -💬 CMake uses simple configuration files placed in each source directory (called CMakeLists.txt files) to generate standard build files (e.g., Makefiles). + ```bash + cd $TMPDIR + wget https://a3s.fi/CSC_training/hello-cmake.tgz + ``` + +2. Extract the package and move into the `hello-cmake` directory: -💬 CMake generates a native build environment that will compile source code, create libraries, generate wrappers and build executable binaries in arbitrary combinations. + ```bash + tar xvf hello-cmake.tgz + cd hello-cmake + ``` + +3. Most codes ship with a `README` or `INSTALL` file outlining the installation + procedure. When compiling other codes, start by reading these carefully. In + this case, the `README` just points to this tutorial page. +4. Before compiling, one should ensure all the dependencies needed by the + software are available. Remember to always first check whether these can be + found on the CSC supercomputers as pre-installed modules. This example + depends on the Boost library, so check if it is available: + + ```bash + module spider boost + ``` -💬 CMake supports in-place and out-of-place builds, and can therefore support multiple builds from a single source tree. - -## Example: Installing a test C++ application - -💬 In this example we'll install an simple test C++ application `Hello World`. - -1. Create a personal folder (if not done already) under your project's `/projappl` directory and move there. - -2. Download the application using git: - -```bash -git clone https://github.com/jameskbride/cmake-hello-world.git -``` - -{:style="counter-reset:step-counter 2"} -3. Move into the folder `cmake-hello-world`. Run `cmake` command. Set the source directory (`-S`) and the build directory (`-B`). This will generate the `Makefile`: - -```bash -cd cmake-hello-world -cmake -S . -B build # Here we create a build directory named `build`. If the build directory does not exist already, cmake creates it. -``` - -{:style="counter-reset:step-counter 3"} -4. Now invoke the build system - -```bash -cmake --build build -``` - -💡 Some 'real world' applications are large and possibly require more resources for faster compilation. In these cases, use the `-j` flag to set the number of cores for compilation. *Please check the number of cores (CPUs) available for use in your system. E.g., In linux, you can find this using `lscpu` command.* - -```bash -cmake --build build -j 8 # This will build the system using 8 cores. -``` - -{:style="counter-reset:step-counter 4"} - -5. Check if the installation was successful: - -```bash -$PWD/build/CMakeHelloWorld -``` - -You should see the output `Hello, world!` - -💡 For 'real world' applications, the binary file is usually created in a default `bin` folder. - -```bash -$PWD/build/bin/application_binary_file -``` - -### Alternate method : Using make command - -1. Move into the folder `cmake-hello-world` and create a build directory (`build`) - - -```bash -cd cmake-hello-world -mkdir build -``` - -{:style="counter-reset:step-counter 1"} - -2. Move into the `build` folder and run cmake. Set the installation path to the current directory (`build`) - -```bash -cd build -cmake .. -DCMAKE_INSTALL_PREFIX=$PWD -``` - -{:style="counter-reset:step-counter 2"} - -3. Now compile and install using the `make` command - -```bash -make -make install -``` - -💡 For 'real world' applications, set the number of cores to be used for compilation using the `-j` flag. - -```bash -make -j 8 -``` +5. Load the module with: + + ```bash + module load boost + ``` + +## Building with CMake + +💡 When your build generates files, they have to go somewhere. An *in-source* +build puts them in your source tree, while an *out-of-source* build puts them +in a completely separate directory. The latter alternative, which is covered +below, is typically recommended, as you can build multiple variants of the code +in separate directories. + +1. Load the `cmake` module: + + ```bash + module load cmake + ``` + +2. Create and move to a `build` directory in the root of the source code: + + ```bash + mkdir build + cd build + ``` + +3. It is recommended to install own software under the `/projappl` directory + of your project. The installation directory can be specified using the + CMake flag `-DCMAKE_INSTALL_PREFIX=`. CMake also + requires you to specify the source code root directory, so run the `cmake` + command as: + + ```bash + cmake .. -DCMAKE_INSTALL_PREFIX=/projappl//$USER/hello-cmake # replace with your CSC project, e.g. project_2001234 + ``` +4. If you get errors, try to fix the problems. For example, if you did not load + the `boost` module, you'll get an error `Could NOT find Boost`. Often it is + easiest to delete everything in the `build` directory and start from the + beginning after fixing all issues. +5. After running `cmake`, run `make` to compile the application: + + ```bash + make + ``` + + 💡 `make` can also be run in parallel. For example, to run using 8 cores, + use `make -j 8`. However, do not compile in parallel on the login node, + but use instead an interactive session. See CSC's + [usage policy](https://docs.csc.fi/computing/usage-policy/) for details. + +6. Finally, install into the specified directory under `/projappl` with: + + ```bash + make install + ``` + +7. Check what was installed and where: + + ```bash + cat install_manifest.txt + ``` + +## Test the installed application + +1. Own software installations are not automatically added to your `PATH`, which + means that the installed binaries (commands) cannot be used without + specifying the full path. To access the commands directly from anywhere, the + directory containing the binaries should be added to `PATH`: + + ```bash + export PATH="/projappl//$USER/hello-cmake/bin:$PATH" # replace with your CSC project, e.g. project_2001234 + ``` + +2. Try to run the program by typing: + + ```bash + hi + ``` + +3. You probably got an error like: + + ```bash + hi: error while loading shared libraries: libhello.so: cannot open shared object file: No such file or directory + ``` + +4. This happens because the program depends on a library file `libhello.so` and + does not know where to look for it. This is a quite common issue with + self-installed software. To fix the situation, add the `lib` path to the + `LD_LIBRARY_PATH` environment variable: + + ```bash + export LD_LIBRARY_PATH="/projappl//$USER/hello-cmake/bin:$LD_LIBRARY_PATH" # replace with your CSC project, e.g. project_2001234 + ``` + +5. Retry running the program, it should now work! + +## More information + +- Not all software use CMake build system. For an example how to build software + using the traditional configure-make procedure, see + [this tutorial](https://csc-training.github.io/csc-env-eff/hands-on/installing/installing_hands-on_mcl.html). +- If you get stuck when compiling your own software, don't hesitate to ask for + help from [CSC Service Desk](https://docs.csc.fi/support/contact/) +- Documentation on how to compile on + [Puhti](https://docs.csc.fi/computing/compiling-puhti/), + [Mahti](https://docs.csc.fi/computing/compiling-mahti/) and + [LUMI](https://docs.lumi-supercomputer.eu/development/). diff --git a/part-2/installing/hpc.md b/part-2/installing/hpc.md deleted file mode 100644 index 9173c1c5..00000000 --- a/part-2/installing/hpc.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -layout: default -title: Installing own C, C++ or Fortran programs -parent: 9. Installing own software -grand_parent: Part 2 -nav_order: 7 -has_children: false -has_toc: false -permalink: /hands-on/installing/installing_hands-on_hpc.html ---- - -# Installing and developing your own C, C++, or Fortran program in the CSC computing environment - -## Case A: you have the source code, you want to install and run it - -1. Create a directory for your code. The recommended location is under the `/projappl` directory of your project: - -```bash -mkdir -p /projappl//myprog # replace with your CSC project, e.g. project_2001234 -``` - -{:style="counter-reset:step-counter 1"} -2. You need the source files of the code. Depending on the software, you can typically download them from e.g. GitHub. If you have the source code on your own computer, use e.g. [`scp`](https://docs.csc.fi/data/moving/scp/) to upload the data to the supercomputer. - -3. If the source files are distributed as a zip file, use `unzip` to decompress: - -```bash -unzip filename.zip # modify the filename accordingly -``` - -{:style="counter-reset:step-counter 3"} -4. Read and follow any instructions on how to install. Usually, the code comes with a `README` or `INSTALL` file outlining the installation procedure. - -5. When compiling, consider using the fast local disk on the login nodes (`$TMPDIR`) to move I/O load away from the parallel file system. - -### Scenario A1: The code uses `cmake` - -1. Load the `cmake` module: - -```bash -module load cmake -``` - -{:style="counter-reset:step-counter 1"} -2. Download and install external libraries that the code might need. Always check first if these can be found as pre-installed modules (the most common ones are available): - -```bash -module spider # replace , e.g. fftw -``` - -{:style="counter-reset:step-counter 2"} -3. Create and move to a `build` directory in the root of the source code: - -```bash -mkdir build -cd build -``` - -{:style="counter-reset:step-counter 3"} -4. Run `cmake` with `cmake /path/to/source/code`: - -```bash -cmake .. -``` - -{:style="counter-reset:step-counter 4"} -5. If you get errors, try to fix the problems. Sometimes it might be easiest to remove everything and start from the beginning, i.e. by unzipping the `.zip` file. -6. After `cmake`, run `make` to compile the specific applications you want to use: - -```bash -make -``` - -{:style="counter-reset:step-counter 6"} -7. Ask help from [CSC Service Desk](https://docs.csc.fi/support/contact/) if you get stuck. - -### Scenario A2: The code comes with a Makefile - -1. `module load` or install separately all required libraries. Check for availability of modules with: - -```bash -module spider # replace , e.g. fftw -``` - -{:style="counter-reset:step-counter 1"} -2. Load the library modules with: - -```bash -module load / # replace /, e.g. fftw/3.3.10-mpi -``` - -{:style="counter-reset:step-counter 2"} -3. Edit the `Makefile` manually or by running `./configure` to replace compile and link commands with proper ones for [Mahti](https://docs.csc.fi/computing/compiling-mahti/) or [Puhti](https://docs.csc.fi/computing/compiling-puhti/). -4. Run the command `make` to compile and `make install` to install. Custom installation location is typically specified with the `--prefix` option of the `configure` script. - -```bash -./configure --prefix=/projappl//myprog # replace with your CSC project, e.g. project_2001234 -make -make install -``` - -{:style="counter-reset:step-counter 4"} -5. Read any error messages and try to fix the possible issues. -6. Ask help from [CSC Service Desk](https://docs.csc.fi/support/contact/) if you get stuck. - -## Case B: You want to write your own code - -1. You need [an editor](https://docs.csc.fi/support/tutorials/env-guide/text-and-image-processing/). -2. Launch an editor and write the code. If not developing locally, consider using the [Puhti web interface](https://www.puhti.csc.fi) and e.g. [VSCode](https://docs.csc.fi/computing/webinterface/vscode/). -3. Compile your code [on Puhti](https://docs.csc.fi/computing/compiling-puhti/) or [on Mahti](https://docs.csc.fi/computing/compiling-mahti/). -4. Fix bugs until compiler accepts code. - -## Exercise - -- Write your own simple C/C++/Fortran code, compile it and run it. diff --git a/part-2/installing/index.md b/part-2/installing/index.md index b9c1e7aa..15406a43 100644 --- a/part-2/installing/index.md +++ b/part-2/installing/index.md @@ -20,5 +20,4 @@ has_toc: false 5. [Tutorial - Installing Perl applications and libraries]({{ site.baseurl }}{% link part-2/installing/perl.md %}) 6. [Tutorial - Installing Java applications]({{ site.baseurl }}{% link part-2/installing/java.md %}) 7. [Tutorial - Installing using CMake]({{ site.baseurl }}{%link part-2/installing/cmake.md %}) -8. [Exercise - Installing own C, C++ or Fortran programs]({{ site.baseurl }}{% link part-2/installing/hpc.md %}) -9. [Exercise - Optimizing compiler options]({{ site.baseurl }}{% link part-2/installing/compilers.md %}) +8. [Exercise - Optimizing compiler options]({{ site.baseurl }}{% link part-2/installing/compilers.md %})