Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

threat intel monitor bug fixes #1317

Merged
merged 5 commits into from
Sep 24, 2024

Conversation

eirsep
Copy link
Member

@eirsep eirsep commented Sep 21, 2024

Description

handle exception arising from trying to search with sort on empty index: Sorting by _seq_no field fails on empty index as mapping isn't created yet. This failure is expected and needs to be handled to return empty list instead of throwing exception and calling it a monitor failure

Caused by: org.opensearch.index.query.QueryShardException: No mapping found for [_seq_no] in order to sort on
        at org.opensearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:564) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:411) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.SortBuilder.buildSort(SortBuilder.java:168) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.parseSource(SearchService.java:1268) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.createContext(SearchService.java:998) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:606) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring:
Use grouped listener to execute parallel calls instead of submitting calls with same listener

[2024-09-15T18:03:08,110][ERROR][o.o.s.t.i.s.IoCScanService] [384634db970ddbb1dc6211644bcfe4fe] Threat intel monitor oulm9JEBop9BFvxFVds2: Unexpected failure in running scan for 246 docs
java.lang.IllegalArgumentException: groupSize must be greater than 0 but was 0
        at org.opensearch.action.support.GroupedActionListener.<init>(GroupedActionListener.java:66)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.getGroupedListenerForIocScanFromAllIocTypes(SaIoCScanService.java:320)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.matchAgainstThreatIntelAndReturnMaliciousIocs(SaIoCScanService.java:254)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.IoCScanService.scanIoCs(IoCScanService.java:85)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$onGetIocTypeToIndices$7(TransportThreatIntelMonitorFanOutAction.java:186)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchDataFromShards$9(TransportThreatIntelMonitorFanOutAction.java:224)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:81)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.fetchLatestDocsFromShard(TransportThreatIntelMonitorFanOutAction.java:261)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchLatestDocsFromShard$11(TransportThreatIntelMonitorFanOutAction.java:291)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$searchShard$15(TransportThreatIntelMonitorFanOutAction.java:352)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:150)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Index out of bounds exception as wrong listener is used.

[2024-09-15T06:50:29,074][ERROR][o.o.s.t.i.s.SaIoCScanService] [efdc3b06f28dcf50f11616883b4a106e] Threat intel monitor oulm9JEBop9BFvxFVds2 scan with 48720 user data indicators failed for ioc Type ipv4-addr
Failed to execute phase [expand],
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:780)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:501)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:158)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 1
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:63)
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:60)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
        at java.base/java.lang.invoke.VarHandleReferences$Array.compareAndSet(VarHandleReferences.java:655)
        at java.base/java.util.concurrent.atomic.AtomicReferenceArray.compareAndSet(AtomicReferenceArray.java:153)
        at org.opensearch.common.util.concurrent.AtomicArray.setOnce(AtomicArray.java:79)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:75)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.lambda$performScanForMaliciousIocsPerIocType$24(SaIoCScanService.java:369)
        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        ... 14 more

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#1319

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@eirsep eirsep changed the title handle exception arising from trying to search with sort on empty index threat intel monitor bug fixes Sep 24, 2024
Signed-off-by: Surya Sashank Nistala <[email protected]>
@eirsep
Copy link
Member Author

eirsep commented Sep 24, 2024

- org.opensearch.securityanalytics.findings.FindingIT.testGetFindings_byDetectionType_success is a flaky test causing CI failure
It is unrelated to this PR's changes

@@ -281,6 +285,8 @@ public void testCreateThreatIntelMonitor() throws IOException {
String doc = String.format("{\"ip\":\"%s\", \"ip1\":\"%s\"}", val, val);
try {
indexDoc(index, "" + i++, doc);
indexDoc(index, "" + i++, String.format("{\"ip\":\"1.2.3.4\", \"ip1\":\"1.2.3.4\"}", val, val));
indexDoc(index, "" + i++, String.format("{\"random\":\"%s\", \"random1\":\"%s\"}", val, val));
} catch (IOException e) {
fail();
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test scenario that replicates the case where the index being queried doesn't have any docs and it correctly catches the all shards failed and the No mapping found for [_seq_no] in order to sort on exception?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@eirsep eirsep merged commit 39c29d4 into opensearch-project:main Sep 24, 2024
16 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 24, 2024
* handle exception arising from trying to search with sort on empty index

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring

Signed-off-by: Surya Sashank Nistala <[email protected]>

* remove unused variable

Signed-off-by: Surya Sashank Nistala <[email protected]>

* avoid grouped listener being initiated with size 0

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add verification that empty index scan is handled gracefully

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
(cherry picked from commit 39c29d4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 24, 2024
* handle exception arising from trying to search with sort on empty index

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring

Signed-off-by: Surya Sashank Nistala <[email protected]>

* remove unused variable

Signed-off-by: Surya Sashank Nistala <[email protected]>

* avoid grouped listener being initiated with size 0

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add verification that empty index scan is handled gracefully

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
(cherry picked from commit 39c29d4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 24, 2024
* handle exception arising from trying to search with sort on empty index

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring

Signed-off-by: Surya Sashank Nistala <[email protected]>

* remove unused variable

Signed-off-by: Surya Sashank Nistala <[email protected]>

* avoid grouped listener being initiated with size 0

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add verification that empty index scan is handled gracefully

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
(cherry picked from commit 39c29d4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 24, 2024
* handle exception arising from trying to search with sort on empty index

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring

Signed-off-by: Surya Sashank Nistala <[email protected]>

* remove unused variable

Signed-off-by: Surya Sashank Nistala <[email protected]>

* avoid grouped listener being initiated with size 0

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add verification that empty index scan is handled gracefully

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
(cherry picked from commit 39c29d4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
eirsep pushed a commit that referenced this pull request Sep 25, 2024
* handle exception arising from trying to search with sort on empty index



* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring



* remove unused variable



* avoid grouped listener being initiated with size 0



* add verification that empty index scan is handled gracefully



---------


(cherry picked from commit 39c29d4)

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
eirsep pushed a commit that referenced this pull request Sep 25, 2024
* handle exception arising from trying to search with sort on empty index



* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring



* remove unused variable



* avoid grouped listener being initiated with size 0



* add verification that empty index scan is handled gracefully



---------


(cherry picked from commit 39c29d4)

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
eirsep pushed a commit that referenced this pull request Sep 25, 2024
* handle exception arising from trying to search with sort on empty index



* add setting to test max term count in threat intel ioc scan terms query and verify grouped listener wiring



* remove unused variable



* avoid grouped listener being initiated with size 0



* add verification that empty index scan is handled gracefully



---------


(cherry picked from commit 39c29d4)

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants