Internal Server Error During Schema Registration

[radhikari-3]

What happened?

I attempted to register a new schema with Karapace using a CURL command. The schema subject is test-key and the schema definition is:

{ "type": "record", "name": "Obj", "fields": [{"name": "age", "type": "int"}] }

The CURL command used for registration is:

curl --location 'https://karapace.banking-ci4.cvp-fabric-experiments.net/subjects/test-key/versions' \
--header 'Content-Type: application/vnd.schemaregistry.v1+json' \
--header 'Authorization: Basic YW1hemluZy1kdWNrbGluZzpTS0lLa2Z1NGFOcUw3MGV6' \
--data '{
  "schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"
}'

However, the registration failed with an Internal Server Error message in the Karapace pod logs. The logs indicate a timeout error during message sending.

What did you expect to happen?

I expected the schema to be successfully registered with Karapace and a new schema ID to be returned in the response.

What else do we need to know?

Platform: AWS EKS Cluster
Karapace Image: ghcr.io/aiven-open/karapace:3.10.6
Schema Registry Backend: AWS Managed Streaming for Kafka (MSK) with SASL authentication
Pod Logs:

karapace.schema_registry MainThread INFO Writing new schema locally since we're the master
karapace MainThread ERROR Internal server error
Traceback (most recent call last):
File "/venv/lib/python3.10/site-packages/karapace/rapu.py", line 329, in _handle_request
data = await callback(**callback_kwargs)
File "/venv/lib/python3.10/site-packages/karapace/schema_registry_apis.py", line 1240, in subject_post
raise xx
File "/venv/lib/python3.10/site-packages/karapace/schema_registry_apis.py", line 1207, in subject_post
schema_id = await self.schema_registry.write_new_schema_local(subject, new_schema, references)
File "/venv/lib/python3.10/site-packages/karapace/schema_registry.py", line 419, in write_new_schema_local
self.send_schema_message(
File "/venv/lib/python3.10/site-packages/karapace/schema_registry.py", line 467, in send_schema_message
self.producer.send_message(key=key, value=value)
File "/venv/lib/python3.10/site-packages/karapace/messaging.py", line 111, in send_message
self._send_kafka_message(key=key_bytes, value=value_bytes)
File "/venv/lib/python3.10/site-packages/karapace/messaging.py", line 79, in _send_kafka_message
msg = future.result(self._kafka_timeout)
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 460, in result
raise TimeoutError()
concurrent.futures._base.TimeoutError

Access Log:
aiohttp.access MainThread INFO 20.011972s - "POST /subjects/test-key/versions HTTP/1.1" 500 "PostmanRuntime/7.32.3" response=358b request_body=116b

Request for Help:

I would appreciate assistance in diagnosing the cause of the timeout error and fixing the issue with schema registration. Any suggestions or insights into potential configuration problems with Karapace or the MSK integration would be helpful. Let me know if you need any more information regading the setup.