0.22.0

Changelog

0.22.0 - 22/7/2024

General System Refactoring

• Renamed NODE to WORKER and restructured WORKER packages (#477)
  • Worker is packaged by feature, including exareme2/flower packages.
  • Fixed version incompatibility issues between pre-commit config and poetry.
  • Resolved a bug in Kubernetes templates.
• System tables are loaded from SQLite (#488)
• Refactored Task Handlers (#476)
  • Restructured the tests to align with the project structure. (#479)
• Refactored Algorithm Specifications (#480)
  • Specifications are now loaded via JSON files.
  • Added type specifications to algorithms in preparation for integrating Flower.

Algorithms

• Dev/mip 897/add log transformation (#487)
  • Added PCA with transformations

Flower Integration

• Integrated Flower Federated Learning Framework (#478)

  • New Controller for Flower Execution:
    • Introduced a controller with modules for managing Flower workflow and algorithm execution.
  • Added Flower-Compatible Algorithms:
    • Logistic Regression with MIP Data: Utilizes MIP data with parameter customization.
    • Logistic Regression with MNIST Data: Tailored for MNIST data operations.
  • Robust Process Management Module:
    • Enhanced process control with functions for sending signals, checking status, managing zombie processes, and process termination with retry capabilities.
    • The controller uses this module to initiate, monitor, and safely terminate Flower execution processes, ensuring better oversight of Flower's client and server components.
    • Lowered local workers in production tests due to increased resource usage by Flower and controller.
    • Implemented process garbage collection at the start of each Flower algorithm.

• Error Handling: (#482)

• Dev/mip 902/flower logging (#486)

  • Added a Flower logger similar to the worker.

• Retry mechanism for Flower client-server connections. (#484)

• Implemented DataFrame filter for Flower inputdata processing. (#485)

• Test Data Loaded on GlobalWorker (#489)

  • Integrated Mipdb into GlobalWorker for dataset monitoring by Worker Landscape Aggregator (WLA).
  • Added test datasets, excluded from Exareme2 flow.

• Inputdata: Split datasets to training datasets and validation datasets. (#490)

• Updated deployment logic:

  • Kubernetes: Ensure GlobalWorker deployment even in single-worker setups.
  • Flower: Configure server to run on GlobalWorker.

0.21.2

Changlog:

• Relaxed the validation of the filter values.

0.21.1

Changelog:

• Upgraded monetdb version to 11.49.1

0.21.0

Release notes - MIP Development - 0.21.0 November 23 Release

Story

MIP-312 Container pod Healthcheck probs

Bug

MIP-807 Remove Celery APP Fix for RabbitMQ restart

Task

MIP-859 NODE pod liveness and startup probe

MIP-860 RABBIMTMQ pod startup probe

MIP-861 MonetDB pod startup probe

MIP-862 Controller pod liveness and readyness probe

0.20.0

Release notes - MIP Development - 0.20.0 October 23 Release

Story

MIP-358 Remote table access rights

Bug

MIP-817 Naive bayes tests missing from prod_env_tests

MIP-827 Validate all parameters passed exist in the specifications

MIP-838 Algorithm with same name should not be possible

Task

MIP-516 Flags in the algorithm specifications

MIP-819 Hard coded DP integration

MIP-826 k8s node specific database password

MIP-829 Add documentation for algorithm development process

MIP-831 On demand Federation Landscape aggregation

MIP-841 NODE and CONTROLLER refactoring to allow for different type of execution

0.19.1

Release notes - Exareme2 - 0.19.1 September 23 Bugfix Release

Story

MIP-358 Remote table access rights

Bug

MIP-817 Naive bayes tests missing from prod_env_tests

MIP-827 Validate all parameters passed exist in the specifications

MIP-838 Algorithm with same name should not be possible

Task

MIP-516 Flags in the algorithm specifications

MIP-826 k8s node specific database password

MIP-829 Add documentation for algorithm development process

MIP-831 On demand Federation Landscape aggregation

0.19.0

Release notes - Exareme2 - 0.19.0 September 23 Release

Story

MIP-382 Port algorithm k-Means

Bug

MIP-476 Quart default error handler doesn't contain error information

MIP-589 monetdb and rabbitmq images are not pulled in conftest.py

MIP-808 Stabilize algorithm validation tests

MIP-809 Controller logs do not provide necessary info in case of an error.

MIP-814 Longitudinal Transformation call fails

MIP-816 Longitudinal Transformer Specifications bad descs and labels

Task

MIP-611 NodeAlgorithmTasksHandler and request_id parameter

MIP-691 NODE tasks and algo_exec code cleanup

MIP-722 test_algorithm_executor should not be in algorithm_validation_tests folder

MIP-746 Remote tables must use the public db user

MIP-762 Remove insert_data_to_table task.

MIP-800 fed_average global UDF

MIP-802 MIP SVM scikit implementation

MIP-821 Document the observability addon setup for microk8s

MIP-825 get_table_data NODE task should be allowed only for "public" tables

MIP-828 Move and update algorithms documentation from exareme1 to exareme2.

0.18.1

Release notes - MIP-Engine - 0.18.1 June 23 Release

Bug

MIP-809 Controller logs do not provide necessary info in case of an error.
MIP-476 Quart default error handler doesn't contain error information
Fixed longitudinal transform bad merge causing internal errors.

0.18.0

Release notes - MIP-Engine - 0.18.0 June 23 Release

Story

MIP-379 Port algorithm Naive-Bayes

Bug

MIP-804 Celery APP is being reset after TimeoutError due to bug.

Epic

MIP-749 Longitudinal data transformation

Task

MIP-796 Address non deterministic tests related to cross-validation

0.17.1

Release notes - MIP-Engine - 0.17.1 April 23 Bugfix Release

Bug

k8s memory limits removed from monetdb container due to improper initialization of the available memory of the database.