Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: queue #430

Merged
merged 22 commits into from
Sep 12, 2023
Merged

feat: queue #430

merged 22 commits into from
Sep 12, 2023

Conversation

thantos
Copy link
Contributor

@thantos thantos commented Aug 11, 2023

Config:

  • Fifo: true/false
  • Encryption: yes/no
  • MessageGroupID

Client-side APIs:

  • SendMessage
  • SendMessageBatch
  • ChangeMessageVisibility
  • DeleteMessage
  • DeleteMessageBatch

Tasks:

  • Core
    • Queue Call
    • Queue Resource + Spec
    • Queue Physical Name Property
    • Queue Handler
  • Runtime
    • Queue Client
    • Workers
      • Queue Call Executor
      • Queue Physical Name Retriever
      • Queue Handler Worker
    • Workflow
      • Queue Workflow Events
      • Queue Call Workflow Executor
      • Queue Call Workflow Eventual Factory
      • Wire Up Factory, Executor, and QueueName property
    • Transaction Worker
      • Support new Properties
    • Replay
      • Wire up Properties
  • Environment
    • AWS
      • AWS Queue Client
      • Dep Injection
      • AWS Queue Worker
      • Update other workers
    • Local
      • Implement Local Queue Client
      • Local Queue Event
    • Test
  • Infra
    • CDK
      • Queue Service
      • Add Queue Service and things to Queue
      • Add Queue to other Services
  • Tests

@thantos thantos marked this pull request as draft August 11, 2023 20:09
handler: QueueHandler;
}

class Queue extends Construct implements IQueue {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see it implementing any of the IQueue methods?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IQueue is just properties (above)?

return call.arguments.length === 3 || call.arguments.length === 4;
}
}
return false;
}

/**
* A heuristic for identifying a {@link CallExpression} that is a call to an `on` handler.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious why you decided to allow multiple consumers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh, this is changed, forgot to change the infer.

@@ -38,6 +38,7 @@ export function createTransactionWorker(
const propertyRetriever = new AllPropertyRetriever({
BucketPhysicalName: unsupportedPropertyRetriever,
OpenSearchClient: unsupportedPropertyRetriever,
QueuePhysicalName: unsupportedPropertyRetriever,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this unsupported in this context?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason as bucket physical name, the current support for these are through the client, which we don't currently inject into the transaction worker. Would need to decouple the physical name calculation (and overrides) from the client first.

}

/**
* TODO: implement message deduplication
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue or do? Seems straightforward?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the hard part is the content based deduplication

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is the time window, but may not be hard to do. Save all of the values/hashes with a date, separated by group for fifo. Ignore messages with the same value/hash until the time is up.

@sam-goodwin sam-goodwin marked this pull request as ready for review September 12, 2023 18:56
@thantos thantos merged commit 6f0dfea into main Sep 12, 2023
3 checks passed
@thantos thantos deleted the sussman/feat/queue branch September 12, 2023 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants