Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] segment replication changes #531

Merged
merged 1 commit into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import java.util.function.Supplier;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.ActionRequest;
import org.opensearch.core.action.ActionResponse;
Expand Down Expand Up @@ -293,6 +294,7 @@
// Trigger initialization of prepackaged rules by calling SearchRule API
SearchRequest searchRequest = new SearchRequest(Rule.PRE_PACKAGED_RULES_INDEX);
searchRequest.source(new SearchSourceBuilder().query(QueryBuilders.matchAllQuery()).size(0));
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 297 in src/main/java/org/opensearch/securityanalytics/SecurityAnalyticsPlugin.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/SecurityAnalyticsPlugin.java#L297

Added line #L297 was not covered by tests
client.execute(
SearchRuleAction.INSTANCE,
new SearchRuleRequest(true, searchRequest),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import org.apache.logging.log4j.Logger;
import org.apache.lucene.search.join.ScoreMode;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.search.MultiSearchRequest;
import org.opensearch.action.search.MultiSearchResponse;
Expand Down Expand Up @@ -85,6 +86,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationRule.CORRELATION_RULE_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 89 in src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java#L89

Added line #L89 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down Expand Up @@ -144,6 +146,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(indices.toArray(new String[]{}));
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 149 in src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java#L149

Added line #L149 was not covered by tests

validCorrelationRules.add(rule);
mSearchRequest.add(searchRequest);
Expand Down Expand Up @@ -221,6 +224,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(DetectorMonitorConfig.getAllFindingsIndicesPattern(categoryToQueries.getKey()));
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 227 in src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java#L227

Added line #L227 was not covered by tests
mSearchRequest.add(searchRequest);
categoryToQueriesPairs.add(new Pair<>(categoryToQueries.getKey(), categoryToQueries.getValue()));
}
Expand Down Expand Up @@ -291,6 +295,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(docSearchCriteria.getValue().indices.toArray(new String[]{}));
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 298 in src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java#L298

Added line #L298 was not covered by tests

categories.add(docSearchCriteria.getKey());
mSearchRequest.add(searchRequest);
Expand Down Expand Up @@ -359,6 +364,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(DetectorMonitorConfig.getAllFindingsIndicesPattern(relatedDocIds.getKey()));
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 367 in src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/JoinEngine.java#L367

Added line #L367 was not covered by tests

categories.add(relatedDocIds.getKey());
mSearchRequest.add(searchRequest);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.bulk.BulkRequest;
import org.opensearch.action.bulk.BulkResponse;
Expand Down Expand Up @@ -74,6 +75,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationIndices.CORRELATION_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 78 in src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java#L78

Added line #L78 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down Expand Up @@ -103,6 +105,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationIndices.CORRELATION_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 108 in src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java#L108

Added line #L108 was not covered by tests

mSearchRequest.add(searchRequest);
}
Expand Down Expand Up @@ -240,6 +243,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationIndices.CORRELATION_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 246 in src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java#L246

Added line #L246 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down Expand Up @@ -415,6 +419,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationIndices.CORRELATION_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 422 in src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/correlation/VectorEmbeddingsEngine.java#L422

Added line #L422 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.opensearch.ExceptionsHelper;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.ResourceAlreadyExistsException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.DocWriteRequest;
import org.opensearch.action.admin.indices.create.CreateIndexRequest;
Expand Down Expand Up @@ -116,7 +117,7 @@ public void getAllLogTypes(ActionListener<List<String>> listener) {
.field(LOG_TYPES)
.size(MAX_LOG_TYPE_COUNT)
));
searchRequest.preference("_primary");
searchRequest.preference(Preference.PRIMARY_FIRST.type());
client.search(
searchRequest,
ActionListener.delegateFailure(
Expand Down Expand Up @@ -374,6 +375,7 @@ public void getAllFieldMappings(ActionListener<List<FieldMappingDoc>> listener)
SearchRequest searchRequest = new SearchRequest(LOG_TYPE_INDEX);
searchRequest.source(new SearchSourceBuilder().query(QueryBuilders.boolQuery()
.mustNot(QueryBuilders.existsQuery("source"))).size(10000));
searchRequest.preference(Preference.PRIMARY_FIRST.type());
client.search(
searchRequest,
ActionListener.delegateFailure(
Expand Down Expand Up @@ -407,6 +409,7 @@ public void getFieldMappingsByLogTypes(List<String> logTypes, ActionListener<Lis
QueryBuilders.termsQuery(LOG_TYPES, logTypes.toArray(new String[0])))
.size(10000)
);
searchRequest.preference(Preference.PRIMARY_FIRST.type());
client.search(
searchRequest,
ActionListener.delegateFailure(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import org.apache.logging.log4j.Logger;
import org.opensearch.action.search.SearchRequest;
import org.opensearch.client.node.NodeClient;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.index.query.QueryBuilder;
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.rest.BaseRestHandler;
Expand Down Expand Up @@ -53,7 +54,8 @@
.version(true);
SearchRequest searchRequest = new SearchRequest()
.source(searchSourceBuilder)
.indices(CorrelationRule.CORRELATION_RULE_INDEX);
.indices(CorrelationRule.CORRELATION_RULE_INDEX)
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 58 in src/main/java/org/opensearch/securityanalytics/resthandler/RestSearchCorrelationRuleAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/resthandler/RestSearchCorrelationRuleAction.java#L57-L58

Added lines #L57 - L58 were not covered by tests

SearchCorrelationRuleRequest searchCorrelationRuleRequest = new SearchCorrelationRuleRequest(searchRequest);
return channel -> client.execute(SearchCorrelationRuleAction.INSTANCE, searchCorrelationRuleRequest, new RestToXContentListener<>(channel));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import org.opensearch.action.search.SearchResponse;
import org.opensearch.action.search.SearchRequest;
import org.opensearch.client.node.NodeClient;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.common.bytes.BytesReference;
import org.opensearch.common.xcontent.LoggingDeprecationHandler;
import org.opensearch.common.xcontent.XContentFactory;
Expand Down Expand Up @@ -69,6 +70,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.source(searchSourceBuilder);
searchRequest.indices(Detector.DETECTORS_INDEX);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 73 in src/main/java/org/opensearch/securityanalytics/resthandler/RestSearchDetectorAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/resthandler/RestSearchDetectorAction.java#L73

Added line #L73 was not covered by tests

SearchDetectorRequest searchDetectorRequest = new SearchDetectorRequest(searchRequest);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import org.opensearch.action.search.SearchRequest;
import org.opensearch.action.search.SearchResponse;
import org.opensearch.client.node.NodeClient;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.common.bytes.BytesReference;
import org.opensearch.common.xcontent.LoggingDeprecationHandler;
import org.opensearch.common.xcontent.XContentFactory;
Expand Down Expand Up @@ -68,7 +69,8 @@
.version(true);
SearchRequest searchRequest = new SearchRequest()
.source(searchSourceBuilder)
.indices(isPrepackaged ? Rule.PRE_PACKAGED_RULES_INDEX: Rule.CUSTOM_RULES_INDEX);
.indices(isPrepackaged ? Rule.PRE_PACKAGED_RULES_INDEX: Rule.CUSTOM_RULES_INDEX)
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 73 in src/main/java/org/opensearch/securityanalytics/resthandler/RestSearchRuleAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/resthandler/RestSearchRuleAction.java#L73

Added line #L73 was not covered by tests

SearchRuleRequest searchRuleRequest = new SearchRuleRequest(isPrepackaged, searchRequest);
return channel -> client.execute(SearchRuleAction.INSTANCE, searchRuleRequest, searchRuleResponse(channel));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import org.apache.logging.log4j.Logger;
import org.apache.lucene.search.join.ScoreMode;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.ActionRequest;
import org.opensearch.action.ActionRunnable;
Expand Down Expand Up @@ -212,6 +213,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(Detector.DETECTORS_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 216 in src/main/java/org/opensearch/securityanalytics/transport/TransportCorrelateFindingAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportCorrelateFindingAction.java#L216

Added line #L216 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down Expand Up @@ -291,6 +293,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationIndices.CORRELATION_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 296 in src/main/java/org/opensearch/securityanalytics/transport/TransportCorrelateFindingAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportCorrelateFindingAction.java#L296

Added line #L296 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import org.apache.logging.log4j.Logger;
import org.apache.lucene.search.join.ScoreMode;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.ActionRunnable;
import org.opensearch.action.get.GetRequest;
Expand Down Expand Up @@ -144,7 +145,8 @@
.seqNoAndPrimaryTerm(true)
.version(true)
.query(queryBuilder)
.size(10000));
.size(10000))
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 149 in src/main/java/org/opensearch/securityanalytics/transport/TransportDeleteRuleAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportDeleteRuleAction.java#L148-L149

Added lines #L148 - L149 were not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import org.apache.logging.log4j.Logger;
import org.apache.lucene.search.join.ScoreMode;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.search.SearchRequest;
import org.opensearch.action.search.SearchResponse;
Expand Down Expand Up @@ -111,6 +112,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(Detector.DETECTORS_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 115 in src/main/java/org/opensearch/securityanalytics/transport/TransportGetAlertsAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportGetAlertsAction.java#L115

Added line #L115 was not covered by tests

transportSearchDetectorAction.execute(new SearchDetectorRequest(searchRequest), new ActionListener<>() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import org.apache.logging.log4j.Logger;
import org.apache.lucene.search.join.ScoreMode;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.search.SearchRequest;
import org.opensearch.action.search.SearchResponse;
Expand Down Expand Up @@ -128,6 +129,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(Detector.DETECTORS_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 132 in src/main/java/org/opensearch/securityanalytics/transport/TransportGetFindingsAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportGetFindingsAction.java#L132

Added line #L132 was not covered by tests

transportSearchDetectorAction.execute(new SearchDetectorRequest(searchRequest), new ActionListener<>() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import org.apache.lucene.search.join.ScoreMode;
import org.opensearch.common.SetOnce;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.ActionRunnable;
import org.opensearch.action.StepListener;
Expand Down Expand Up @@ -204,7 +205,9 @@
User user
) {
String [] detectorIndices = request.getDetector().getInputs().stream().flatMap(detectorInput -> detectorInput.getIndices().stream()).toArray(String[]::new);
SearchRequest searchRequest = new SearchRequest(detectorIndices).source(SearchSourceBuilder.searchSource().size(1).query(QueryBuilders.matchAllQuery()));;
SearchRequest searchRequest = new SearchRequest(detectorIndices)
.source(SearchSourceBuilder.searchSource().size(1).query(QueryBuilders.matchAllQuery()))
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 210 in src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java#L208-L210

Added lines #L208 - L210 were not covered by tests
client.search(searchRequest, new ActionListener<>() {
@Override
public void onResponse(SearchResponse searchResponse) {
Expand Down Expand Up @@ -1132,7 +1135,8 @@
.seqNoAndPrimaryTerm(true)
.version(true)
.query(queryBuilder)
.size(10000));
.size(10000))
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 1139 in src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java#L1138-L1139

Added lines #L1138 - L1139 were not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down Expand Up @@ -1191,7 +1195,8 @@
.seqNoAndPrimaryTerm(true)
.version(true)
.query(queryBuilder)
.size(10000));
.size(10000))
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 1199 in src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java#L1198-L1199

Added lines #L1198 - L1199 were not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import org.opensearch.action.support.HandledTransportAction;
import org.opensearch.action.support.master.AcknowledgedResponse;
import org.opensearch.client.Client;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.inject.Inject;
import org.opensearch.common.settings.Settings;
Expand Down Expand Up @@ -316,7 +317,8 @@
.seqNoAndPrimaryTerm(true)
.version(true)
.query(queryBuilder)
.size(10000));
.size(10000))
.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 321 in src/main/java/org/opensearch/securityanalytics/transport/TransportIndexRuleAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexRuleAction.java#L320-L321

Added lines #L320 - L321 were not covered by tests

client.search(searchRequest, listener);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.opensearch.OpenSearchStatusException;
import org.opensearch.cluster.routing.Preference;
import org.opensearch.core.action.ActionListener;
import org.opensearch.action.ActionRunnable;
import org.opensearch.action.search.SearchRequest;
Expand Down Expand Up @@ -112,6 +113,7 @@
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices(CorrelationIndices.CORRELATION_INDEX);
searchRequest.source(searchSourceBuilder);
searchRequest.preference(Preference.PRIMARY_FIRST.type());

Check warning on line 116 in src/main/java/org/opensearch/securityanalytics/transport/TransportListCorrelationAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/transport/TransportListCorrelationAction.java#L116

Added line #L116 was not covered by tests

client.search(searchRequest, new ActionListener<>() {
@Override
Expand Down
Loading
Loading