Skip to content
This repository has been archived by the owner on Jan 13, 2020. It is now read-only.

Home

Jump to bottom
Grayson Hilliard edited this page Aug 2, 2016 · 9 revisions

HIVE-Master

High Incantation Velocity for Experiments

"Because whatever happens inside that container is black magic to me..."

Description

HIVE is a user-friendly master-slave batch-queue system designed to be attached to docker images containing Fermilab experiment software. One method of implementing this system is to put both the worker and master code into one image and selecting the corresponding startup script will assign the container a role, however this may not be feasible for some implementations.

Hive is built with:

+ Python-Flask
+ Docker
+ RabbitMQ
+ MongoDB
+ Kubernetes (optional)

Install

  1. Install the python dependecies

    pip install -r requirements.txt

  2. Modify the callback function in worker.py to perform your desired tasks.

  3. Ensure paths are correct for the DATA_PATH and LOG_PATH in worker.py.

  4. If the worker node is not on the same machine as the master, the auto-connect will fail. Therefore, use of the --masterip argument is neccessary when executing worker.py.

  5. Create a file called HIVE-Master/secret_key.py with the variable SECRET_KEY set to a string of super secret characters.

Resources

Slides: https://drive.google.com/open?id=1Zbezdlr00o3MSaXDqb9udY5M7igir-L30tGtF8w-eCI

Worker Github: https://github.com/wghilliard/HIVE-Worker

LARIATSoft Docker Image: https://hub.docker.com/r/wghilliard/jfw2/

Email: grsn.hilliard{at}gmail.com

Clone this wiki locally