[Fetch Migration] Monitoring script for Data Prepper

[kartg]

This is a draft PR until opensearch-project#261 has been merged, at which point the base branch of this PR should be updated to the upstream main branch

Description

This change adds a monitor.py Python script that can monitor the running Data Prepper pipeline and shut it down once the target document count has been reached (as reported by Data Prepper's documentsSuccess Prometheus metric) and an idle pipeline is detected.

• Category: New feature

---

Testing

Unit Tests

$ python -m coverage report --omit "*/tests/*"
Name                  Stmts   Miss  Cover
-----------------------------------------
endpoint_info.py          6      0   100%
index_operations.py      36      0   100%
main.py                 117      0   100%
monitor.py               48      0   100%
utils.py                 13      0   100%
-----------------------------------------
TOTAL                   220      0   100%

Integration Test

1. Setup step (see the Index Configuration Tool readme):

python index_configuration_tool/main.py -r <input_pipeline_yaml> <output_yaml_path>

2. Data Prepper container kickoff:

docker run -p 4900:4900 -v <output_yaml_path>:/usr/share/data-prepper/pipelines/pipelines.yaml opensearch-data-prepper:2.4.0-SNAPSHOT

3. After Data Prepper server has started, monitoring process kickoff:

python index_configuration_tool/monitor.py https://localhost:4900 <target_doc_count_from_report>

4. Verified that the monitoring process invoked the /shutdown endpoint once the target doc count was reached

---

Limitations

• Though the monitoring script correctly stops the Data Prepper server, the Docker container does not terminate. The current theory is that this is due to [BUG] Docker container does not stop running after Data Prepper shuts down opensearch-project/data-prepper#3141
• Currently the endpoint auth information is hard-coded. The fix for this is tracked in JIRA task MIGRATIONS-1285
• The monitoring script only tracks Data Prepper metrics to identify shutdown state. It does not check the target cluster to validate that the target document count has been reached. Since index readers need to refresh to make the newly ingested documents searchable, there is often a delay (~5 minutes from my local testing) in the document counts reflecting in the _cat/indices count of the target cluster. Handling this scenario is tracked in JIRA task MIGRATIONS-1228

---

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kartg]

Closed in favor of opensearch-project#264