Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting Up and End-to-End CI/CD Framework #68

Open
callahantiff opened this issue Dec 21, 2020 · 2 comments · Fixed by #85
Open

Setting Up and End-to-End CI/CD Framework #68

callahantiff opened this issue Dec 21, 2020 · 2 comments · Fixed by #85
Assignees
Labels
ISMB challenge release v2.0.0 noting work and issues related to release v2.0.0

Comments

@callahantiff
Copy link
Owner

callahantiff commented Dec 21, 2020

Task

Task Type: INFRASTRUCTURE
Determine which tools we will use in order to set-up an end-to-end CI/CD framework.

TODO

The requirements for this system include:

  • Leveraging GitHub Actions to:
    • Test the codebase
    • Downloaded needed resources and build the Docker Container
    • Deploy and run the Docker container via Google Cloud Run (one for each KG build type)
    • Generate baseline embeddings (Add Baseline KG Embeddings To CI/CD #71)
    • Returning all results
    • Pushing certain files to Neo4J instance and SPARQL Endpoint

Potential Configurations:

  • CI/CD with Serverless Containers on GCP - Described here
  • Consider using Google Cloud Composer to kick-off the first task of the monthly build process which downloads and preprocess the data used for each build (LOD and Ontology data)

Proposed Tasks for CI/CD

  • Download all LOD and Ontology data
  • Preprocess and Clean data
  • KG Build

Related GitHub Issues: #47, #49

@callahantiff
Copy link
Owner Author

callahantiff commented Dec 26, 2020

TODO

  • Script out data download and write to GCS (TASK 1)
    • If any failure in download, default to last build's version of downloaded data and log issue
  • Script out preprocessing of LOD and Ontology data (TASK 2)
    • Log any issues
    • Output updated resource_info.txt, edge_source_list.txt, and ontology_source_list.txt to Docker container
    • Decide if ontologies are merged in TASK 2 and then merged data is also sent to resources/knowledge_graphs in Docker container
  • Update Docker build trigger to pull from TASKS 1-2 (TASK 3)
  • Add script that runs after each successful build and copies data from release_v2.0.0/archived_builds/build_DDMMYYY to release_v2.0.0/current_build/
  • Update SPARQL Endpoint and Neo4J

@callahantiff callahantiff mentioned this issue Jan 19, 2021
5 tasks
@callahantiff callahantiff linked a pull request Jan 20, 2021 that will close this issue
5 tasks
@callahantiff callahantiff removed a link to a pull request Jan 31, 2021
5 tasks
@callahantiff callahantiff linked a pull request Jan 31, 2021 that will close this issue
5 tasks
@callahantiff
Copy link
Owner Author

Nearly done. The jobs are too long to use GitHub-hosted runners for GitHub Actions. Need to explore other options for self-hosted runner. Considering using Terraform.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ISMB challenge release v2.0.0 noting work and issues related to release v2.0.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants