Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up P&E's SQS process #68

Merged
merged 9 commits into from
Mar 27, 2024
Merged

Clean up P&E's SQS process #68

merged 9 commits into from
Mar 27, 2024

Conversation

aloftus23
Copy link
Contributor

@aloftus23 aloftus23 commented Mar 14, 2024

🗣 Description

Clean up the P&E SQS process. The old flow is described in this PR.

Instead of using ECS services to manage concurrent Fargate tasks, we can simply generate Fargate tasks through "ECS Run tasks". The issue with ECS services is that you can set a desired count of concurrent tasks (say 10), but if one finishes the script early, it will start a new one even though there's nothing left to run. We want tasks to complete and then shut down, which ECS run task ensures.

Also, this method removes the need of a "Control" SQS queue that parses messages into scan specific queues. Now we only need a scan specific queue.

💭 Motivation and context

Save costs and time running scans through SQS, lambda, and Fargate.

🧪 Testing

Tested locally and passes checks.

📷 Screenshots (if appropriate)

OLD:
Screenshot 2024-03-14 at 11 27 54 AM

NEW:
Screenshot 2024-03-14 at 11 28 27 AM

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced
    in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated
    to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

✅ Pre-merge checklist

  • Revert dependencies to default branches.
  • Finalize version.

✅ Post-merge checklist

  • Create a release.

backend/src/tools/run-scanExecution.ts Dismissed Show dismissed Hide dismissed
Copy link
Collaborator

@rapidray12 rapidray12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not a fan of magic numbers (e.g. 604200). We need to clean those up. Also we need to come up with a new acronym. Maybe switch from PE to XFD

@schmelz21 schmelz21 merged commit 7323ace into develop Mar 27, 2024
16 of 19 checks passed
@schmelz21 schmelz21 mentioned this pull request Mar 27, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants