x-pack/filebeat/input/salesforce: Bug fixes and improvements

[shmsr]

Proposed commit message

Add more debug logs and for requesting log files do check for 200 status code and then only allow. More details here.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Connect with SF instance and collect documents. Samples docs here:

apex

{ "@timestamp": "2024-10-01T11:40:55.686Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "9.0.0", "pipeline": "filebeat-9.0.0-salesforce-apex-pipeline" }, "ecs": { "version": "8.0.0" }, "event": { "timezone": "+05:30", "provider": "EventLogFile", "dataset": "salesforce.apex", "module": "salesforce" }, "service": { "type": "salesforce" }, "input": { "type": "salesforce" }, "salesforce": { "instance_url": "" }, "message": "{\"CALLOUT_TIME\":\"0\",\"CLIENT_IP\":\"\",\"CPU_TIME\":\"58\",\"DB_TOTAL_TIME\":\"26\",\"ENTRY_POINT\":\"DNBConnect.DNB_LinkageOpportunity_Batch\",\"EVENT_TYPE\":\"ApexExecution\",\"EXEC_TIME\":\"90\",\"IS_LONG_RUNNING_REQUEST\":\"0\",\"LOGIN_KEY\":\"\",\"NUMBER_SOQL_QUERIES\":\"1\",\"ORGANIZATION_ID\":\"\",\"QUIDDITY\":\"BA\",\"REQUEST_ID\":\"-\",\"RUN_TIME\":\"127\",\"SESSION_KEY\":\"\",\"TIMESTAMP\":\"20240924140008.050\",\"TIMESTAMP_DERIVED\":\"2024-09-24T14:00:08.050Z\",\"URI\":\"\",\"URI_ID_DERIVED\":\"\",\"USER_ID\":\"\",\"USER_ID_DERIVED\":\"IAK\"}", "tags": [ "salesforce-apex", "forwarded" ], "fileset": { "name": "apex" }, "agent": { "version": "9.0.0", "ephemeral_id": "", "id": "", "name": "", "type": "filebeat" } }

login

{ "@timestamp": "2024-10-01T11:41:01.222Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "9.0.0", "pipeline": "filebeat-9.0.0-salesforce-login-default" }, "event": { "dataset": "salesforce.login", "module": "salesforce", "timezone": "+05:30", "provider": "EventLogFile" }, "tags": [ "salesforce-login", "forwarded" ], "fileset": { "name": "login" }, "service": { "type": "salesforce" }, "salesforce": { "instance_url": "" }, "agent": { "ephemeral_id": "", "id": "", "name": "", "type": "filebeat", "version": "9.0.0" }, "message": "{\"API_TYPE\":\"\",\"API_VERSION\":\"9998.0\",\"AUTHENTICATION_METHOD_REFERENCE\":\"\",\"AUTHENTICATION_SERVICE_ID\":\"\",\"BROWSER_TYPE\":\"\",\"CIPHER_SUITE\":\"TLS_AES_256_GCM_SHA384\",\"CLIENT_IP\":\"\",\"CPU_TIME\":\"62\",\"DB_TOTAL_TIME\":\"85692869\",\"EVENT_TYPE\":\"Login\",\"FORWARDED_FOR_IP\":\"\",\"LOGIN_KEY\":\"Am2nijOL2d98wLS3\",\"LOGIN_STATUS\":\"LOGIN_NO_ERROR\",\"LOGIN_SUB_TYPE\":\"oauthpassword\",\"LOGIN_TYPE\":\"A\",\"ORGANIZATION_ID\":\"\",\"REQUEST_ID\":\"SLB:\",\"REQUEST_STATUS\":\"S\",\"RUN_TIME\":\"182\",\"SESSION_KEY\":\"\",\"SOURCE_IP\":\"\",\"TIMESTAMP\":\"20240930031304.166\",\"TIMESTAMP_DERIVED\":\"2024-09-30T03:13:04.166Z\",\"TLS_PROTOCOL\":\"TLSv1.3\",\"URI\":\"/services/oauth2/token\",\"URI_ID_DERIVED\":\"\",\"USER_ID\":\"\",\"USER_ID_DERIVED\":\"IAK\",\"USER_NAME\":\"\",\"USER_TYPE\":\"Standard\"}", "input": { "type": "salesforce" }, "ecs": { "version": "8.0.0" } }

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @shmsr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[shmsr]

most important change. only resp with 200 status code should be allowed to go forward.

[kush-elastic]

Provided few minor comments. Otherwise LGTM.

[devamanv]

Just left a small suggestion, rest looks good to me.