Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/rabbitmq] RabbitMQ is asked to stop #29644

Open
BenoitJimini opened this issue Sep 27, 2024 · 1 comment
Open

[bitnami/rabbitmq] RabbitMQ is asked to stop #29644

BenoitJimini opened this issue Sep 27, 2024 · 1 comment
Assignees
@javsalgar
Labels
rabbitmq tech-issues The user has a technical issue about an application triage Triage is needed

Comments

@BenoitJimini
Copy link

BenoitJimini commented Sep 27, 2024 •
edited
Loading

Name and Version

bitnami/rabbitmq 14.7.0

What architecture are you using?

amd64

What steps will reproduce the bug?

I am using rabbitmq's chart in my application.
Sometimes, it restarts. When I have a look at the logs, I don't see much except RabbitMQ is asked to stop...
To get the following logs, I enabled the bitnami debug. I copy-pasted the logs from rabbitmq startup signal to RabbitMQ is asked to stop.
I have trouble understanding why it stops from the logs.
The startup and connection are fine, the memory watermark (which is set to 0.7) is not raising any notice, the applications can use rabbitmq normaly.
A few minutes later, a log about RABBITMQ_SYNC_TIMEOUT appears, the config appears, and then rabbitmq is asked to stop.

2024-09-27 11:09:44.473688+00:00 [info] <0.633.0> Server startup complete; 8 plugins started.
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_prometheus
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_federation
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_auth_backend_ldap
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_peer_discovery_k8s
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_peer_discovery_common
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_management
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_management_agent
2024-09-27 11:09:44.473688+00:00 [info] <0.633.0>  * rabbitmq_web_dispatch
2024-09-27 09:09:44.472452+00:00 [info] <0.9.0> Time to start RabbitMQ: 15381 ms
2024-09-27 11:10:00.282626+00:00 [info] <0.837.0> accepting AMQP connection <0.837.0> (ip-address-1:52898 -> rabbitmq-ip-address:5672)
2024-09-27 11:10:00.287476+00:00 [info] <0.837.0> connection <0.837.0> (ip-address-1:52898 -> rabbitmq-ip-address:5672): user 'application-user' authenticated and granted access to vhost '/'
2024-09-27 11:10:03.700636+00:00 [info] <0.851.0> accepting AMQP connection <0.851.0> (ip-address-2:55950 -> rabbitmq-ip-address:5672)
2024-09-27 11:10:03.708965+00:00 [info] <0.851.0> connection <0.851.0> (ip-address-2:55950 -> rabbitmq-ip-address:5672): user 'application-user' authenticated and granted access to vhost '/'
2024-09-27 11:15:11.313 09:15:12.51 DEBUG ==> RABBITMQ_SYNC_TIMEOUT is 110
2024-09-27 11:15:12.515 Cluster status of node rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local ...
2024-09-27 11:15:14.287 Basics
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Cluster name: rabbitmq
2024-09-27 11:15:14.348 Total CPU cores available cluster-wide: 4
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Disk Nodes
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Running Nodes
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Versions
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local: RabbitMQ 3.13.7 on Erlang 26.2.5
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 CPU Cores
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Node: rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local, available CPU cores: 4
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Maintenance status
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Node: rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local, status: not under maintenance
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Alarms
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 (none)
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Network Partitions
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 (none)
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Listeners
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Node: rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
2024-09-27 11:15:14.348 Node: rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local, interface: [::], port: 9419, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
2024-09-27 11:15:14.348 Node: rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
2024-09-27 11:15:14.348 Node: rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Feature flags
2024-09-27 11:15:14.348
2024-09-27 11:15:14.348 Flag: classic_mirrored_queue_version, state: enabled
2024-09-27 11:15:14.348 Flag: classic_queue_type_delivery_support, state: enabled
2024-09-27 11:15:14.348 Flag: detailed_queues_endpoint, state: enabled
2024-09-27 11:15:14.348 Flag: direct_exchange_routing_v2, state: enabled
2024-09-27 11:15:14.348 Flag: drop_unroutable_metric, state: enabled
2024-09-27 11:15:14.348 Flag: empty_basic_get_metric, state: enabled
2024-09-27 11:15:14.348 Flag: feature_flags_v2, state: enabled
2024-09-27 11:15:14.348 Flag: implicit_default_bindings, state: enabled
2024-09-27 11:15:14.348 Flag: khepri_db, state: disabled
2024-09-27 11:15:14.348 Flag: listener_records_in_ets, state: enabled
2024-09-27 11:15:14.348 Flag: maintenance_mode_status, state: enabled
2024-09-27 11:15:14.348 Flag: message_containers, state: enabled
2024-09-27 11:15:14.348 Flag: message_containers_deaths_v2, state: enabled
2024-09-27 11:15:14.348 Flag: quorum_queue, state: enabled
2024-09-27 11:15:14.348 Flag: quorum_queue_non_voters, state: enabled
2024-09-27 11:15:14.348 Flag: restart_streams, state: enabled
2024-09-27 11:15:14.348 Flag: stream_filtering, state: enabled
2024-09-27 11:15:14.348 Flag: stream_queue, state: enabled
2024-09-27 11:15:14.348 Flag: stream_sac_coordinator_unblock_group, state: enabled
2024-09-27 11:15:14.348 Flag: stream_single_active_consumer, state: enabled
2024-09-27 11:15:14.348 Flag: stream_update_config_command, state: enabled
2024-09-27 11:15:14.348 Flag: tracking_records_in_ets, state: enabled
2024-09-27 11:15:14.348 Flag: user_limits, state: enabled
2024-09-27 11:15:14.348 Flag: virtual_host_metadata, state: enabled
2024-09-27 11:15:14.348  09:15:14.36 DEBUG ==> Will wait up to 110 seconds for node to make sure cluster is healthy after node shutdown
2024-09-27 11:15:14.364 Node rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local seems to be the only one in a single node cluster, the check does not apply
2024-09-27 11:15:16.301 Node rabbit@rabbitmq-pod-0.rabbitmq-headless.test-namespace.svc.cluster.local seems to be the only one in a single node cluster, the check does not apply
2024-09-27 11:15:18.396
2024-09-27 11:15:20.396272+00:00 [info] <0.1130.0> RabbitMQ is asked to stop...
2024-09-27 09:15:20.471304+00:00 [info] <0.1130.0> Stopping RabbitMQ applications and their dependencies in the following order:

Are you using any custom parameters or values?

rabbitmq:
  loadDefinition:
    enabled: true
    file: "/app/definitions.json"
    existingSecret: "rabbitmq-definitions-secret"
  extraConfiguration: |-
    consumer_timeout = 3600000
  memoryHighWatermark:
    enabled: true
    type: "relative"
    value: 0.7
  resources:
    requests:
      cpu: 500m
      memory: 8Gi
      ephemeral-storage: "10Gi"
    limits:
      cpu: 750m
      memory: 12Gi
      ephemeral-storage: "15Gi"
  image:
    debug: true

What is the expected behavior?

Not to restart / not to be Asked to stop

What do you see instead?

RabbitMQ is asked to stop...

Additional information

No response
@BenoitJimini BenoitJimini added the tech-issues The user has a technical issue about an application label Sep 27, 2024
@github-actions github-actions bot added the triage Triage is needed label Sep 27, 2024
@javsalgar
Copy link
Contributor

Hi,

Does kubectl describe pod provide any meaningful information? Just to understand if it is because of the liveness probes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@javsalgar javsalgar

Labels
rabbitmq tech-issues The user has a technical issue about an application triage Triage is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@javsalgar @BenoitJimini