[Backport 1.3] Allow TransportConfigUpdateAction when security config initialization has completed

[cwperks]

Backport #3810 to 1.3

Testing performed with the configuration below:

1.3-latest.zip

This uses a configuration similar to the demo configuration, but with the following modifications that make a reproduction of this error more pronounced:

# plugins.security.allow_default_init_securityindex: true
plugins.security.unsupported.restapi.allow_securityconfig_modification: true
plugins.security.unsupported.load_static_resources: false

Steps to test:

1. Start the 3 node cluster
2. Exec into one of the nodes: docker exec -it 13-latest-opensearch-node2-1 /bin/bash
3. Run securityadmin: cd plugins/opensearch-security/tools && ./securityadmin.sh -cd ../securityconfig/ -icl -nhnv \ -cacert ../../../config/root-ca.pem \ -cert ../../../config/kirk.pem \ -key ../../../config/kirk-key.pem
4. Exit the remote session: exit
5. Start infinite loop of config update in another terminal: while true;do curl -ai -u "admin:admin" -k -X PATCH https://localhost:9200/_opendistro/_security/api/securityconfig -H 'Content-Type: application/json' -d'[{"op": "replace", "path": "/config/dynamic/authc/basic_internal_auth_domain/transport_enabled", "value": "true"}]'; done
6. Restart one of the nodes

Query the rebooted node and you will not get a response:

> curl -XGET https://admin:admin@localhost:9201 -k
No response

If the cache is loaded correctly you will get a response, but access denied since static resource loading is disabled (this is expected and it means that internal users have been loaded into the cache):

> curl -XGET https://admin:admin@localhost:9200 -k
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}

[peternied]

@cwperks Why couldn't the integration tests be backported?

[cwperks]

@cwperks Why couldn't the integration tests be backported?

I ran into issues backporting to 1.3 with the integrationTest framework. I may have misremembered which integration tests are running on the 1.3 line and thought it was only the ResourceFocusedTests of the integrationTest framework.

I'm taking another look at it now and currently running into an issue with the TransportClient in SecurityAdmin.

[cwperks]

@peternied I raised a PR for adding the integ tests to 1.3: #4134

The biggest issue was the port for the security admin tests. In 1.3 its the transport port, but in 2.x its the http port.