SARC 114 - Doc sphinx

.github/workflows/python-package.yml

@@ -49,7 +49,7 @@ jobs:
       matrix:
         python-version: ['3.10']
         toxenv: [black, isort, lint]
-      
+
 
     steps:
     - uses: actions/checkout@v3
@@ -67,4 +67,26 @@ jobs:
 
     - name: Tests with tox
       run: poetry run tox -e ${{ matrix.toxenv }}
-
+
+  test-generate-docs:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.10']
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install poetry
+        poetry install
+
+    - name: Tests doc generation
+      run: poetry run sphinx-build -b html docs/ docs/_build

.gitignore

@@ -13,3 +13,4 @@ clockwork_ldap.key
 .tox/
 sarc-cache/
 .idea
+_build

.readthedocs.yaml

@@ -0,0 +1,32 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+    # You can also specify other tool versions:
+    # nodejs: "19"
+    # rust: "1.64"
+    # golang: "1.19"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+   configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+# formats:
+#    - pdf
+#    - epub
+
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+    - requirements: docs/requirements.txt

README.md

@@ -79,3 +79,12 @@ sarc/account_matching/update_account_matches_in_database.py
 sarc/inode_storage_scanner/get_diskusage.py  (stub)
 ```
 
+### How to generate doc
+
+To generate documentation in HTML format in folder `docs\_build`:
+
+```
+sphinx-build -b html docs/ docs/_build
+```
+
+You can then open `docs\_build\index.html` on a web browser.

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/account_matching.md

@@ -1,4 +1,3 @@
-
 # Quick guide on LDAP retrieval and account matching
 
 ## Structure in MongoDB

docs/account_matching_manual_procedure.md

@@ -1,7 +1,6 @@
 # Account Matching procedure
 
----
-# In a nutshell
+## In a nutshell
 
 The manual procedure steps are:
 
@@ -12,8 +11,7 @@ The manual procedure steps are:
 - run the `acquire users` command
 - if any errors, adjust the exceptions in the `secrets/make_matches_config.json` file, and re-run the account matching script.
 
----
-# In details
+## In details
 
 This procedure consists in making matches between the DRAC user account and the Mila acocunts.
 
@@ -22,7 +20,7 @@ This procedure consists in making matches between the DRAC user account and the
 
 After that, the `users` collection of mongoDB contains the aggregated users database.
 
-## Access rights
+### Access rights
 
 The operator executing the account matching procedure must have wite access to the DRAC folder.
 
@@ -31,12 +29,12 @@ Two possible scenarios :
 - have write access to the running SARC server (production server)
 - use SARC from a local machine, with a SSH connection to the production server (see below). **This is the prefered method.**
 
-### Remote access to MongoDB (via SSH tunneling) 
+#### Remote access to MongoDB (via SSH tunneling) 
 
-#### SSH config
+##### SSH config
 Refer to `remote_mongo_access.md` for ssh connection with port redirection, to connect to mongoDB form the local machine.
 
-#### SARC config file
+##### SARC config file
 To use the remote mongoDB connection, tunneled from localhost:27018, the `mongo` section in the config file like this:
 
 ```
@@ -46,7 +44,7 @@ To use the remote mongoDB connection, tunneled from localhost:27018, the `mongo`
     },
 ```
 
-## data source 1: Mila LDAP credentials
+### data source 1: Mila LDAP credentials
 
 The credentials for the Mila LDAP are in the `secrets/ldap` folder.
 
@@ -61,21 +59,21 @@ They are refered to in the ldap section of the sarc config file :
 
 ```
 
-## data source 2: DRAC account files
+### data source 2: DRAC account files
 
 Compute Canada must provide 2 CSV files:
 - One "members" file
 - One "roles" file 
 
-### copy the files in the right directory
+#### copy the files in the right directory
 
 The two file must be copied to the `secrets/account_matching/` folder of SARC, on the server or the local machine, depending on the scenario. 
 
-### Configuration file
+#### Configuration file
 
 
 
-## Exceptions handling
+### Exceptions handling
 
 The exception are manually handled in the `secrets/make_matches_config.json` file.
 
@@ -101,9 +99,9 @@ The procedure is:
 - run the matching script
 - if there are mathcing errors, modify `make_matches_config.json` accordingly and re-run the matching script.
 
-## Run the matching script
+### Run the matching script
 
 From the SARC folder:
 ```
 $ SARC_CONFIG=<path_to_config_file> poetry run sarc acquire users
-```
+```

docs/conf.py

@@ -0,0 +1,26 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "SARC"
+copyright = "2023, Mila"
+author = "Mila"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = ["myst_parser"]
+
+templates_path = ["_templates"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]

docs/index.rst

@@ -0,0 +1,21 @@
+.. SARC documentation master file, created by
+   sphinx-quickstart on Mon Jul 24 14:36:51 2023.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to SARC's documentation!
+================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   account_matching
+   account_matching_manual_procedure
+   deployment
+   remote_mongo_access
+   secrets
+   scripts/README
+   scripts/remote/README
+   scripts/systemd/README
+   scripts/deploy/README

docs/remote_mongo_access.md

@@ -1,5 +1,6 @@
+# Remove Mongo access
 
-# MongoDB port tunneling
+## MongoDB port tunneling
 
 With an SSH access to the production machine, you can easilly tunnel the mongoDB. Example in the `~/.ssh/config` file:
 
@@ -9,7 +10,7 @@ Host sarc
     LocalForward 27018 127.0.0.1:27017
 ```
 
-# SARC config file
+## SARC config file
 
 Simply modify the config JSON file you use:
 
@@ -19,4 +20,4 @@ Simply modify the config JSON file you use:
         "database_name": "sarc"
     },
 
-```
+```

docs/requirements.txt

@@ -0,0 +1,3 @@
+sphinx<7
+myst-parser<3
+sphinx-rtd-theme<2

docs/scripts/README.md

@@ -0,0 +1,9 @@
+# Setup Scripts
+
+## remote
+
+[These scripts](remote/README.md) must be run from a computer from which you have ssh access to the sarc server.
+
+## systemd
+
+[These services](systemd/README.md) must be injected in systemd; see the [deployment doc](../deployment.md) and the [systemd README.md](systemd/README.md)

docs/scripts/deploy/README.md

@@ -0,0 +1,10 @@
+# Deploy scripts
+
+`sudo install_services.sh` will install and start the services for SARC.
+
+## Prerequisite: podman containers
+
+The podman containers must have been created before `sarc_contaners.service` can be started and/or enabled on boot.
+See the [deployment doc](../../deployment.md) for further info.
+
+

docs/scripts/remote/README.md

@@ -0,0 +1,27 @@
+# Remote scripts
+
+## serverinit.sh
+Installs base packages on a bare new OS.
+SSH access and sudo rights on the target server needed.
+
+```
+$ chmod +x serverinit.sh
+$ ./serverinit.sh <server>
+```
+
+## mongo.sh
+Install, start and stop mongo package on the server
+
+```
+$ ./mongo.sh <server> [start|stop|status]
+```
+
+You **MUST** run `./mongo.sh <server> install` once before deploying systemd scripts.
+
+## setup_github_keys.sh
+Generate the ssh "deploy keys" needed to deploy the code to the server
+```
+$ ./setup_github_keys.sh <server> 
+```
+
+See [The deployment doc](../../deployment.md) for more info.

docs/scripts/systemd/README.md

@@ -0,0 +1,9 @@
+# systemd services
+
+These service scripts must be linked from `/etc/systemd/system/`
+
+See [The deploy README](../deploy/README.md) for more info.
+
+- Execute jobs scraper with `systemctl start sarc_scrapers.service` (one-shot)
+- Activate the timer with `systemctl start sarc_scrapers.timer`
+- Enable at boot with `systemctl enable sarc_scrapers.timer`