Skip to content

Commit

Permalink
Merge pull request #2 from NatLibFi/lingsoft_changes
Browse files Browse the repository at this point in the history 
Initial E-kirjasto changes (Authentication, OpenSearch analytics, Docker customisation)
  • Loading branch information
@mikko-work
mikko-work authored Jan 16, 2024
2 parents f459202 + 6a0491d commit f8f1795
Show file tree
Hide file tree
Showing 41 changed files with 3,476 additions and 148 deletions.
8 changes: 4 additions & 4 deletions .github/workflows/build-base-image.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Build Base Image
on:
schedule:
# Every Monday at 6:22am Eastern Time
- cron: '22 10 * * 1'
- cron: "22 10 * * 1"
workflow_dispatch:
# Allow us to manually trigger build

Expand Down Expand Up @@ -46,11 +46,11 @@ jobs:
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/circ-baseimage
images: ghcr.io/${{ github.repository_owner }}/ekirjasto-circ-baseimage
# Generate tags for the image
# We use the following tags:
# - The date in YYYYww format, which is the year and week number. 202052 is the last week of 2020.
# - The latest tag
# - The date in YYYYww format, which is the year and week number. 202052 is the last week of 2020.
# - The latest tag
tags: |
type=schedule,pattern={{date 'YYYYww'}}
type=raw,value=latest
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/test-build.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ jobs:
id: baseimage-meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/circ-baseimage
images: ghcr.io/${{ github.repository_owner }}/ekirjasto-circ-baseimage
tags: |
type=ref,event=branch
type=sha
Expand All @@ -169,7 +169,7 @@ jobs:
id: baseimage-latest
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/circ-baseimage
images: ghcr.io/${{ github.repository_owner }}/ekirjasto-circ-baseimage
tags: |
type=raw,value=latest
Expand All @@ -192,7 +192,7 @@ jobs:
type=registry,ref=${{ fromJSON(steps.baseimage-latest.outputs.json).tags[0] }}
type=registry,ref=${{ fromJSON(steps.baseimage-meta.outputs.json).tags[0] }}
cache-to: |
type=inline
type=inline
platforms: linux/amd64, linux/arm64
tags: ${{ steps.baseimage-meta.outputs.tags }}
labels: ${{ steps.baseimage-meta.outputs.labels }}
Expand All @@ -213,7 +213,7 @@ jobs:
elif [[ $tag_exists -eq 0 ]]; then
tag="${{ fromJSON(steps.baseimage-latest.outputs.json).tags[0] }}"
else
tag="ghcr.io/thepalaceproject/circ-baseimage:latest"
tag="ghcr.io/natlibfi/ekirjasto-circ-baseimage:latest"
fi
echo "Base image tag: $tag"
echo tag="$tag" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -358,7 +358,7 @@ jobs:
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/circ-${{ matrix.image }}
images: ghcr.io/${{ github.repository_owner }}/ekirjasto-circ-${{ matrix.image }}
tags: |
type=semver,pattern={{major}}.{{minor}},priority=10
type=semver,pattern={{version}},priority=20
Expand Down
149 changes: 81 additions & 68 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,39 +1,23 @@
# Palace Manager
# E-kirjasto Circulation Manager

[![Test & Build](https://github.com/ThePalaceProject/circulation/actions/workflows/test-build.yml/badge.svg)](https://github.com/ThePalaceProject/circulation/actions/workflows/test-build.yml)
[![codecov](https://codecov.io/github/thepalaceproject/circulation/branch/main/graph/badge.svg?token=T09QW6DLH6)](https://codecov.io/github/thepalaceproject/circulation)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
![Python: 3.8,3.9,3.10,3.11](https://img.shields.io/badge/Python-3.8%20|%203.9%20|%203.10%20|%203.11-blue)
This is the E-kirjasto fork of the [The Palace Project](https://thepalaceproject.org) Palace Manager (which is a fork of
[Library Simplified](http://www.librarysimplified.org/) Circulation Manager).

This is a [The Palace Project](https://thepalaceproject.org) maintained fork of the NYPL
[Library Simplified](http://www.librarysimplified.org/) Circulation Manager.
<!-- ## Installation
## Installation
Docker images created from this code will be available at:
Docker images created from this code are available at:
- [circ-webapp](https://github.com/NatLibFi/circulation/pkgs/container/ekirjasto-circ-webapp)
- [circ-scripts](https://github.com/NatLibFi/circulation/pkgs/container/ekirjasto-circ-scripts)
- [circ-exec](https://github.com/NatLibFi/circulation/pkgs/container/ekirjasto-circ-exec)
- [circ-webapp](https://github.com/ThePalaceProject/circulation/pkgs/container/circ-webapp)
- [circ-scripts](https://github.com/ThePalaceProject/circulation/pkgs/container/circ-scripts)
- [circ-exec](https://github.com/ThePalaceProject/circulation/pkgs/container/circ-exec)

Docker images are the preferred way to deploy this code in a production environment.
Docker images are the preferred way to deploy this code in a production environment. -->

## Git Branch Workflow

| Branch | Python Version |
| -------- | -------------- |
| main | Python 3 |
| python2 | Python 2 |

The default branch is `main` and that's the working branch that should be used when branching off for bug fixes or new
features.

Python 2 stopped being supported after January 1st, 2020 but there is still a `python2` branch which can be used. As of
August 2021, development will be done in the `main` branch and the `python2` branch will not be updated unless
absolutely necessary.

## Set Up

### Docker Compose
Expand Down Expand Up @@ -77,7 +61,21 @@ Most distributions will offer Python packages. On Arch Linux, the following comm
pacman -S python
```

#### pyenv
You need to install dependencies: https://devguide.python.org/getting-started/setup-building/#build-dependencies

Enable Source Packages:
Uncomment a deb-src in `/etc/apt/sources.list` e.g. `jammy main`

Install build dependencies:

```sh
$ sudo apt-get update
$ sudo apt-get build-dep python3
$ sudo apt-get install pkg-config
$ sudo apt install libxmlsec1 libxmlsec1-dev
```

#### pyenv (Optional)

[pyenv](https://github.com/pyenv/pyenv) pyenv lets you easily switch between multiple versions of Python. It can be
[installed](https://github.com/pyenv/pyenv-installer) using the command `curl https://pyenv.run | bash`. You can then
Expand Down Expand Up @@ -110,14 +108,14 @@ Poetry can be installed using the command `curl -sSL https://install.python-poet
More information about installation options can be found in the
[poetry documentation](https://python-poetry.org/docs/master/#installation).

### Opensearch
### OpenSearch

Palace now supports Opensearch: please use it instead of Elasticsearch.
Palace now supports OpenSearch: please use it instead of Elasticsearch.
Elasticsearch is no longer supported.

#### Docker

We recommend that you run Opensearch with docker using the following docker commands:
We recommend that you run OpenSearch with docker using the following docker commands:

```sh
docker run --name opensearch -d --rm -p 9006:9200 -e "discovery.type=single-node" -e "plugins.security.disabled=true" "opensearchproject/opensearch:1"
Expand Down Expand Up @@ -171,20 +169,20 @@ a storage service, you can set the following environment variables:
public access to the files.
- `PALACE_STORAGE_ANALYTICS_BUCKET`: Required if you want to use the storage service to store analytics data.
- `PALACE_STORAGE_ACCESS_KEY`: The access key (optional).
- If this key is set it will be passed to boto3 when connecting to the storage service.
- If it is not set boto3 will attempt to find credentials as outlined in their
[documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- If this key is set it will be passed to boto3 when connecting to the storage service.
- If it is not set boto3 will attempt to find credentials as outlined in their
[documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- `PALACE_STORAGE_SECRET_KEY`: The secret key (optional).
- `PALACE_STORAGE_REGION`: The AWS region of the storage service (optional).
- `PALACE_STORAGE_ENDPOINT_URL`: The endpoint of the storage service (optional). This is used if you are using a
s3 compatible storage service like [minio](https://min.io/).
- `PALACE_STORAGE_URL_TEMPLATE`: The url template to use when generating urls for files stored in the storage service
(optional).
- The default value is `https://{bucket}.s3.{region}.amazonaws.com/{key}`.
- The following variables can be used in the template:
- `{bucket}`: The name of the bucket.
- `{key}`: The key of the file.
- `{region}`: The region of the storage service.
- The default value is `https://{bucket}.s3.{region}.amazonaws.com/{key}`.
- The following variables can be used in the template:
- `{bucket}`: The name of the bucket.
- `{key}`: The key of the file.
- `{region}`: The region of the storage service.

#### Reporting

Expand All @@ -209,16 +207,17 @@ the logging:
- `PALACE_LOG_CLOUDWATCH_INTERVAL`: The interval in seconds to send logs to CloudWatch. Default is `60`.
- `PALACE_LOG_CLOUDWATCH_CREATE_GROUP`: Whether to create the log group if it does not exist. Default is `true`.
- `PALACE_LOG_CLOUDWATCH_ACCESS_KEY`: The access key to use when sending logs to CloudWatch. This is optional.
- If this key is set it will be passed to boto3 when connecting to CloudWatch.
- If it is not set boto3 will attempt to find credentials as outlined in their
[documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- If this key is set it will be passed to boto3 when connecting to CloudWatch.
- If it is not set boto3 will attempt to find credentials as outlined in their
[documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- `PALACE_LOG_CLOUDWATCH_SECRET_KEY`: The secret key to use when sending logs to CloudWatch. This is optional.

#### Patron `Basic Token` authentication

Enables/disables patron "basic token" authentication through setting the designated environment variable to any
(case-insensitive) value of "true"/"yes"/"on"/"1" or "false"/"no"/"off"/"0", respectively.
If the value is the empty string or the variable is not present in the environment, it is disabled by default.

- `SIMPLIFIED_ENABLE_BASIC_TOKEN_AUTH`

```sh
Expand All @@ -228,7 +227,8 @@ export SIMPLIFIED_ENABLE_BASIC_TOKEN_AUTH=true
#### Firebase Cloud Messaging

For Firebase Cloud Messaging (FCM) support (e.g., for notifications), `one` (and only one) of the following should be set:
- `SIMPLIFIED_FCM_CREDENTIALS_JSON` - the JSON-format Google Cloud Platform (GCP) service account key or

- `SIMPLIFIED_FCM_CREDENTIALS_JSON` - the JSON-format Google Cloud Platform (GCP) service account key or
- `SIMPLIFIED_FCM_CREDENTIALS_FILE` - the name of the file containing that key.

```sh
Expand Down Expand Up @@ -258,6 +258,14 @@ Local analytics are enabled by default. S3 analytics can be enabled via the foll

- PALACE_S3_ANALYTICS_ENABLED: A boolean value to disable or enable s3 analytics. The default is false.

## OpenSearch Analytics (E-Kirjasto, Finland)

OpenSearch analytics can be enabled via the following environment variables:

- PALACE_OPENSEARCH_ANALYTICS_ENABLED: A boolean value to disable or enable OpenSearch analytics. The default is false.
- PALACE_OPENSEARCH_ANALYTICS_URL: The url of your OpenSearch instance, eg. "http://localhost:9200"
- PALACE_OPENSEARCH_ANALYTICS_INDEX_PREFIX: The prefix of the event index name, eg. "circulation-events"

#### Email

### Email sending
Expand Down Expand Up @@ -323,6 +331,7 @@ Run the application with:
poetry run python app.py
```

psear
Check that there is now a web server listening on port `6500`:

```sh
Expand All @@ -333,7 +342,7 @@ curl http://localhost:6500/

#### Access

By default, the application is configured to provide a built-in version of the [admin web interface](https://github.com/ThePalaceProject/circulation-admin).
By default, the application is configured to provide a built-in version of the [admin web interface](https://github.com/NatLibFi/ekirjasto-circulation-admin).
The admin interface can be accessed by visiting the `/admin` endpoint:

```sh
Expand Down Expand Up @@ -404,9 +413,9 @@ service has been configured.
#### Configuring Search

Navigate to `System Configuration → Search` and add a new search configuration. The required URL is
the URL of the [Opensearch instance configured earlier](#opensearch):
the URL of the [OpenSearch instance configured earlier](#opensearch):

![Opensearch](.github/readme/search.png)
![OpenSearch](.github/readme/search.png)

#### Generating Search Indices

Expand Down Expand Up @@ -534,13 +543,14 @@ the different lints that pre-commit runs.
#### Built in

Pre-commit ships with a [number of lints](https://pre-commit.com/hooks.html) out of the box, we are configured to use:

- `trailing-whitespace` - trims trailing whitespace.
- `end-of-file-fixer` - ensures that a file is either empty, or ends with one newline.
- `check-yaml` - checks yaml files for parseable syntax.
- `check-json` - checks json files for parseable syntax.
- `check-ast` - simply checks whether the files parse as valid python.
- `check-shebang-scripts-are-executable` - ensures that (non-binary) files with a shebang are executable.
- `check-executables-have-shebangs` - ensures that (non-binary) executables have a shebang.
- `check-executables-have-shebangs` - ensures that (non-binary) executables have a shebang.
- `check-merge-conflict` - checks for files that contain merge conflict strings.
- `check-added-large-files` - prevents giant files from being committed.
- `mixed-line-ending` - replaces or checks mixed line ending.
Expand Down Expand Up @@ -593,7 +603,7 @@ with service dependencies running in docker containers.
#### Python version

| Factor | Python Version |
|--------|----------------|
| ------ | -------------- |
| py38 | Python 3.8 |
| py39 | Python 3.9 |
| py310 | Python 3.10 |
Expand All @@ -617,10 +627,10 @@ missing Python versions in your system for local testing.

#### Module

| Factor | Module |
| ----------- | ----------------- |
| core | core tests |
| api | api tests |
| Factor | Module |
| ------ | ---------- |
| core | core tests |
| api | api tests |

#### Docker

Expand Down Expand Up @@ -703,7 +713,7 @@ enabled by setting environment variables while starting the application.
- `PALACE_XRAY_NAME`: The name of the service shown in x-ray for these traces.
- `PALACE_XRAY_ANNOTATE_`: Any environment variable starting with this prefix will be added to to the trace as an
annotation.
- For example setting `PALACE_XRAY_ANNOTATE_KEY=value` will set the annotation `key=value` on all xray traces sent
- For example setting `PALACE_XRAY_ANNOTATE_KEY=value` will set the annotation `key=value` on all xray traces sent
from the application.
- `PALACE_XRAY_INCLUDE_BARCODE`: If this environment variable is set to `true` then the tracing code will try to include
the patrons barcode in the user parameter of the trace, if a barcode is available.
Expand All @@ -725,8 +735,9 @@ module under the hood to do the profiling.
path specified in the environment variable.
- The profile data will have the extension `.prof`.
- The data can be accessed using the
[`pstats.Stats` class](https://docs.python.org/3/library/profile.html#the-stats-class).
[`pstats.Stats` class](https://docs.python.org/3/library/profile.html#the-stats-class).
- Example code to print details of the gathered statistics:

```python
import os
from pathlib import Path
Expand All @@ -747,23 +758,25 @@ This profiler uses [PyInstrument](https://pyinstrument.readthedocs.io/en/latest/

- `PALACE_PYINSTRUMENT`: Profiling will the enabled if this variable is set. The saved profile data will be available at
path specified in the environment variable.
- The profile data will have the extension `.pyisession`.
- The data can be accessed with the

- The profile data will have the extension `.pyisession`.
- The data can be accessed with the
[`pyinstrument.session.Session` class](https://pyinstrument.readthedocs.io/en/latest/reference.html#pyinstrument.session.Session).
- Example code to print details of the gathered statistics:
```python
import os
from pathlib import Path

from pyinstrument.renderers import HTMLRenderer
from pyinstrument.session import Session

path = Path(os.environ.get("PALACE_PYINSTRUMENT"))
for file in path.glob("*.pyisession"):
session = Session.load(file)
renderer = HTMLRenderer()
renderer.open_in_browser(session)
```
- Example code to print details of the gathered statistics:

```python
import os
from pathlib import Path

from pyinstrument.renderers import HTMLRenderer
from pyinstrument.session import Session

path = Path(os.environ.get("PALACE_PYINSTRUMENT"))
for file in path.glob("*.pyisession"):
session = Session.load(file)
renderer = HTMLRenderer()
renderer.open_in_browser(session)
```

### Other Environment Variables

Expand Down
4 changes: 2 additions & 2 deletions api/admin/config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ class OperationalMode(str, Enum):


class Configuration(LoggerMixin):
APP_NAME = "Palace Collection Manager"
PACKAGE_NAME = "@thepalaceproject/circulation-admin"
APP_NAME = "E-kirjasto Collection Manager"
PACKAGE_NAME = "@natlibfi/ekirjasto-circulation-admin"
PACKAGE_VERSION = "1.11.0"

STATIC_ASSETS = {
Expand Down
Loading

0 comments on commit f8f1795

Please sign in to comment.