Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Only log that the scan is complete one time for s3 scan #3168

Merged
merged 1 commit into from
Aug 16, 2023
Merged

Only log that the scan is complete one time for s3 scan #3168

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .../main/java/org/opensearch/dataprepper/plugins/source/S3ScanPartitionCreationSupplier.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ public class S3ScanPartitionCreationSupplier implements Function<Map<String, Obj
private static final String BUCKET_OBJECT_PARTITION_KEY_FORMAT = "%s|%s";
static final String SCAN_COUNT = "SCAN_COUNT";
static final String LAST_SCAN_TIME = "LAST_SCAN_TIME";
static final String SINGLE_SCAN_COMPLETE = "SINGLE_SCAN_COMPLETE";

private final S3Client s3Client;
private final BucketOwnerProvider bucketOwnerProvider;
Expand Down Expand Up @@ -143,6 +144,7 @@ private boolean isKeyMatchedBetweenTimeRange(final LocalDateTime lastModifiedTim

private void initializeGlobalStateMap(final Map<String, Object> globalStateMap) {
globalStateMap.put(SCAN_COUNT, 0);
globalStateMap.put(SINGLE_SCAN_COMPLETE, false);
}

private boolean isLastModifiedTimeAfterMostRecentScanForBucket(final String bucketName,
Expand Down Expand Up @@ -174,14 +176,22 @@ private Instant getMostRecentLastModifiedTimestamp(final ListObjectsV2Response l
}

private boolean shouldScanBeSkipped(final Map<String, Object> globalStateMap) {

if (Objects.isNull(schedulingOptions) && hasAlreadyBeenScanned(globalStateMap)) {
LOG.info("Skipping scan because the buckets have already been scanned once");

if (!(Boolean) globalStateMap.get(SINGLE_SCAN_COMPLETE)) {
LOG.info("Single S3 scan has already been completed");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why log only during the seconds attempt at scanning? Why not log once the first scan is complete?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could log after the first scan, but with source coordination this will only get run again after all the scanned objects have been processed, so as far as timing of the log it makes more since when trying to scan again

globalStateMap.put(SINGLE_SCAN_COMPLETE, true);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between SINGLE_SCAN_COMPLETE and SCAN_COUNT? can we just use SCAN_COUNT to determine if scan is complete?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCAN_COUNT could block scanning since the global state doesn't go away. so it's better to keep it separate for this log

}

return true;
}

if (Objects.nonNull(schedulingOptions) &&
(hasReachedMaxScanCount(globalStateMap) || !hasReachedScheduledScanTime(globalStateMap))) {



if (hasReachedMaxScanCount(globalStateMap)) {
LOG.info("Skipping scan as the max scan count {} has been reached", schedulingOptions.getCount());
} else {
Expand Down
Loading