[BUG] Issues in threat intel monitor scans

[eirsep]

Sorting by _seq_no field fails on empty index as mapping isn't created yet. This failure is expected and needs to be handled to return empty list instead of throwing exception and calling it a monitor failure

Caused by: org.opensearch.index.query.QueryShardException: No mapping found for [_seq_no] in order to sort on
        at org.opensearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:564) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:411) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.SortBuilder.buildSort(SortBuilder.java:168) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.parseSource(SearchService.java:1268) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.createContext(SearchService.java:998) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:606) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

There is an edge case scenario when there are hits in data but no iocs are actually present to scan. That case is causing creation of a grouped listener of size 0 which is throwing error

[2024-09-15T18:03:08,110][ERROR][o.o.s.t.i.s.IoCScanService] [384634db970ddbb1dc6211644bcfe4fe] Threat intel monitor oulm9JEBop9BFvxFVds2: Unexpected failure in running scan for 246 docs
java.lang.IllegalArgumentException: groupSize must be greater than 0 but was 0
        at org.opensearch.action.support.GroupedActionListener.<init>(GroupedActionListener.java:66)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.getGroupedListenerForIocScanFromAllIocTypes(SaIoCScanService.java:320)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.matchAgainstThreatIntelAndReturnMaliciousIocs(SaIoCScanService.java:254)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.IoCScanService.scanIoCs(IoCScanService.java:85)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$onGetIocTypeToIndices$7(TransportThreatIntelMonitorFanOutAction.java:186)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchDataFromShards$9(TransportThreatIntelMonitorFanOutAction.java:224)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:81)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.fetchLatestDocsFromShard(TransportThreatIntelMonitorFanOutAction.java:261)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchLatestDocsFromShard$11(TransportThreatIntelMonitorFanOutAction.java:291)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$searchShard$15(TransportThreatIntelMonitorFanOutAction.java:352)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:150)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Index out of bounds exception as wrong listener is used.

[2024-09-15T06:50:29,074][ERROR][o.o.s.t.i.s.SaIoCScanService] [efdc3b06f28dcf50f11616883b4a106e] Threat intel monitor oulm9JEBop9BFvxFVds2 scan with 48720 user data indicators failed for ioc Type ipv4-addr
Failed to execute phase [expand],
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:780)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:501)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:158)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 1
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:63)
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:60)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
        at java.base/java.lang.invoke.VarHandleReferences$Array.compareAndSet(VarHandleReferences.java:655)
        at java.base/java.util.concurrent.atomic.AtomicReferenceArray.compareAndSet(AtomicReferenceArray.java:153)
        at org.opensearch.common.util.concurrent.AtomicArray.setOnce(AtomicArray.java:79)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:75)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.lambda$performScanForMaliciousIocsPerIocType$24(SaIoCScanService.java:369)
        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        ... 14 more

[dblock]

Was this fixed in #1317? Close?

[Catch All Triage - 1, 2, 3, 4]