Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory while synchronizing the node #2624

Open
SixoneKui opened this issue Sep 30, 2023 · 25 comments
Open

Out of memory while synchronizing the node #2624

SixoneKui opened this issue Sep 30, 2023 · 25 comments

Comments

@SixoneKui
Copy link

I tried run a node on a mounted volume using:
docker run -it -v /mnt/HC_Volume_37317682:/acala/data acala/acala-node:2.21.0 --base-path=/acala/data --chain=acala

the above is got from another question.
But I see acala-node requires a large RAM, sometimes over 32GB and then causes out of memory issue (My could server has only 32GB ).
Is this normal ?

Thanks a lot.

@xlc
Copy link
Member

xlc commented Sep 30, 2023

Do you have logs, memory usage matrices etc from the server?

@SixoneKui
Copy link
Author

Capture

Please see this. It has used 49GB and this number will increase till out of memory (64GB).

@xlc
Copy link
Member

xlc commented Sep 30, 2023

This is not enough for us to diagnose the issue.
I need logs and memory usage growth matrix.
Also what’s the Acala node version, OS version?

@SixoneKui
Copy link
Author

ubuntu 22.04.3 LTS
acala node version 2.21.0 (should I use 2.20.0?)

Logs:
acala2

acala1

My RAM is 64GB
Ram
PID USER PR NI VIRT RES SHR S CPU MEM TIME+ COMMAND
20605 1000 20 0 252.2g 38.5g 47512 D 43.8 62.7 6:43.89 acala
20605 1000 20 0 252.2g 38.8g 47512 D 33.9 63.3 6:47.28 acala
20605 1000 20 0 252.2g 39.1g 47512 R 34.2 63.8 6:50.71 acala
20605 1000 20 0 252.2g 39.5g 47512 R 32.3 64.4 6:53.95 acala
20605 1000 20 0 252.2g 39.8g 47512 D 33.4 64.9 6:57.30 acala
20605 1000 20 0 252.2g 40.1g 47512 D 32.4 65.5 7:00.54 acala
20605 1000 20 0 252.2g 40.5g 47512 D 31.4 66.0 7:03.69 acala
20605 1000 20 0 258.2g 40.8g 47512 R 32.1 66.5 7:06.91 acala
20605 1000 20 0 258.2g 41.1g 47512 R 32.5 67.1 7:10.17 acala
20605 1000 20 0 258.2g 41.5g 47512 D 32.5 67.6 7:13.43 acala
20605 1000 20 0 258.2g 41.8g 47512 D 33.1 68.2 7:16.75 acala
20605 1000 20 0 258.2g 42.1g 47512 D 33.1 68.7 7:20.06 acala
20605 1000 20 0 258.2g 42.5g 47512 D 33.2 69.3 7:23.39 acala
20605 1000 20 0 258.2g 42.8g 47512 D 31.7 69.8 7:26.57 acala
20605 1000 20 0 258.2g 43.1g 47512 D 33.9 70.4 7:29.97 acala
20605 1000 20 0 258.2g 43.5g 47512 R 33.3 70.9 7:33.31 acala
20605 1000 20 0 258.2g 43.8g 47512 D 31.7 71.4 7:36.48 acala
20605 1000 20 0 258.2g 44.1g 47512 D 33.4 72.0 7:39.83 acala
20605 1000 20 0 258.2g 44.5g 47496 R 32.4 72.5 7:43.08 acala
20605 1000 20 0 258.2g 44.8g 47496 R 32.3 73.0 7:46.31 acala
20605 1000 20 0 258.3g 45.1g 47496 D 32.8 73.6 7:49.60 acala
20605 1000 20 0 258.3g 45.5g 47496 R 31.3 74.1 7:52.74 acala
20605 1000 20 0 258.3g 45.8g 47496 R 32.6 74.7 7:56.01 acala
20605 1000 20 0 258.3g 46.1g 47496 D 33.5 75.2 7:59.37 acala
20605 1000 20 0 258.3g 46.4g 47496 R 32.3 75.7 8:02.60 acala
20605 1000 20 0 265.3g 46.8g 47496 D 34.0 76.3 8:06.01 acala
20605 1000 20 0 265.3g 47.1g 47496 D 33.3 76.8 8:09.35 acala
20605 1000 20 0 265.3g 47.4g 47496 R 32.8 77.3 8:12.64 acala
20605 1000 20 0 265.3g 47.7g 47496 R 32.6 77.9 8:15.91 acala
20605 1000 20 0 265.3g 48.1g 47496 R 32.7 78.4 8:19.18 acala
20605 1000 20 0 265.3g 48.4g 47496 D 32.8 78.9 8:22.47 acala
20605 1000 20 0 265.3g 48.7g 47496 R 33.1 79.5 8:25.79 acala
20605 1000 20 0 265.3g 49.0g 47496 R 32.8 80.0 8:29.08 acala
20605 1000 20 0 265.3g 49.4g 47496 D 34.0 80.5 8:32.48 acala
20605 1000 20 0 265.3g 49.7g 47496 R 31.9 81.1 8:35.68 acala
20605 1000 20 0 265.3g 50.0g 47496 D 33.9 81.6 8:39.07 acala
20605 1000 20 0 265.3g 50.4g 47496 D 32.9 82.1 8:42.37 acala
20605 1000 20 0 265.3g 50.7g 47496 D 33.8 82.7 8:45.76 acala
20605 1000 20 0 265.3g 51.0g 47496 D 34.2 83.3 8:49.19 acala
20605 1000 20 0 265.3g 51.4g 47496 D 34.4 83.8 8:52.64 acala
20605 1000 20 0 265.3g 51.7g 47496 D 35.9 84.4 8:56.23 acala
20605 1000 20 0 265.3g 52.1g 47496 D 35.4 84.9 8:59.78 acala
20605 1000 20 0 265.3g 52.4g 47496 D 33.8 85.5 9:03.17 acala
20605 1000 20 0 265.3g 52.7g 24672 D 36.1 86.0 9:06.79 acala
20605 1000 20 0 654.2g 53.5g 54700 S 208.0 87.3 9:27.61 acala
20605 1000 20 0 654.6g 53.6g 54740 S 236.1 87.5 9:51.27 acala
20605 1000 20 0 654.8g 53.5g 54676 S 256.0 87.2 10:16.92 acala
20605 1000 20 0 654.9g 53.2g 54712 S 262.3 86.8 10:43.20 acala
20605 1000 20 0 655.0g 53.0g 54712 S 257.3 86.5 11:08.98 acala
20605 1000 20 0 655.1g 52.8g 54712 S 247.5 86.0 11:33.75 acala
20605 1000 20 0 655.2g 52.1g 54712 S 229.5 85.0 11:56.75 acala
20605 1000 20 0 655.2g 51.7g 54716 S 106.5 84.3 12:07.42 acala
20605 1000 20 0 655.2g 51.3g 54716 S 104.3 83.6 12:17.87 acala
20605 1000 20 0 655.2g 50.9g 54716 S 102.8 83.0 12:28.17 acala
20605 1000 20 0 655.2g 50.5g 54716 S 102.5 82.4 12:38.43 acala
20605 1000 20 0 655.2g 50.2g 54716 S 102.6 81.8 12:48.71 acala
20605 1000 20 0 655.2g 49.8g 54716 S 102.2 81.2 12:58.95 acala
20605 1000 20 0 669.2g 49.6g 54716 S 102.4 80.8 13:09.20 acala
20605 1000 20 0 669.2g 49.2g 54780 S 102.9 80.3 13:19.51 acala
20605 1000 20 0 669.2g 48.9g 54780 S 102.1 79.7 13:29.73 acala
20605 1000 20 0 669.2g 48.5g 54844 S 102.5 79.2 13:40.00 acala
20605 1000 20 0 669.2g 49.2g 55012 S 112.6 80.2 13:51.28 acala
20605 1000 20 0 685.2g 53.6g 55012 S 117.0 87.5 14:03.00 acala

@xlc
Copy link
Member

xlc commented Oct 3, 2023

Your screenshot is unreadable. Please just copy & paste the logs instead of screenshots.
We don't have any known issue on 2.21. But you could try downgrade 2.20/2.19 to see if it helps.

@SixoneKui
Copy link
Author

I have switched to 2.20. This issue still occurs .
See the log below:

2023-10-05 02:29:42 [Parachain] ✨ Imported #4600692 (0xc7d6…cd25)
2023-10-05 02:39:28 Acala Node
2023-10-05 02:39:28 ✌️ version 2.20.0-unknown
2023-10-05 02:39:28 ❤️ by Acala Developers, 2019-2023
2023-10-05 02:39:28 📋 Chain specification: Acala
2023-10-05 02:39:28 🏷 Node name: kkc
2023-10-05 02:39:28 👤 Role: FULL
2023-10-05 02:39:28 💾 Database: RocksDb at /acala/data/chains/acala/db/full
2023-10-05 02:39:28 Parachain id: Id(2000)
2023-10-05 02:39:28 Parachain id: Id(2000)
2023-10-05 02:39:28 Is collating: no
2023-10-05 02:45:55 [Relaychain] 🏷 Local node identity is: 12D3KooWFu3t8kw9PEefyAcz1oAndfits5nsLmHkeNxk5QP2JjYB
2023-10-05 02:45:56 [Relaychain] 💻 Operating system: linux
2023-10-05 02:45:56 [Relaychain] 💻 CPU architecture: x86_64
2023-10-05 02:45:56 [Relaychain] 💻 Target environment: gnu
2023-10-05 02:45:56 [Relaychain] 💻 CPU: Intel Xeon Processor (Skylake, IBRS)
2023-10-05 02:45:56 [Relaychain] 💻 CPU cores: 4
2023-10-05 02:45:56 [Relaychain] 💻 Memory: 15611MB
2023-10-05 02:45:56 [Relaychain] 💻 Kernel: 5.15.0-84-generic
2023-10-05 02:45:56 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.4 LTS
2023-10-05 02:45:56 [Relaychain] 💻 Virtual machine: yes
2023-10-05 02:45:56 [Relaychain] 📦 Highest known block at #8274944
2023-10-05 02:45:56 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616
2023-10-05 02:45:56 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost:", "http://127.0.0.1:", "https://localhost:", "https://127.0.0.1:", "https://polkadot.js.org"]
2023-10-05 02:45:56 [Relaychain] Starting with an empty approval vote DB.
2023-10-05 02:45:56 [Parachain] 🏷 Local node identity is: 12D3KooWRmbcLPAirR1eVvvUxg7xaeGRiiPY2FRYSjB2qevmRRFR
2023-10-05 02:45:56 [Parachain] 💻 Operating system: linux
2023-10-05 02:45:56 [Parachain] 💻 CPU architecture: x86_64
2023-10-05 02:45:56 [Parachain] 💻 Target environment: gnu
2023-10-05 02:45:56 [Parachain] 💻 CPU: Intel Xeon Processor (Skylake, IBRS)
2023-10-05 02:45:56 [Parachain] 💻 CPU cores: 4
2023-10-05 02:45:56 [Parachain] 💻 Memory: 15611MB
2023-10-05 02:45:56 [Parachain] 💻 Kernel: 5.15.0-84-generic
2023-10-05 02:45:56 [Parachain] 💻 Linux distribution: Ubuntu 20.04.4 LTS
2023-10-05 02:45:56 [Parachain] 💻 Virtual machine: yes
2023-10-05 02:45:56 [Parachain] 📦 Highest known block at #4600688
2023-10-05 02:45:56 [Parachain] Running JSON-RPC server: addr=0.0.0.0:9933, allowed origins=["*"]
2023-10-05 02:45:56 [Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615
2023-10-05 02:45:56 [Relaychain] di Prometheus exporter started at 127.0.0.1:9615
2023-10-05 02:45:56 [Parachain] discovered: 12D3KooWFu3t8kw9PEefyAcz1oAndfits5nsLmHkeNxk5QP2JjYB /ip4/172.17.0.2/tcp/30334/ws
2023-10-05 02:45:56 [Parachain] ✨ Imported #4600693 (0x69e4…d78c)
2023-10-05 02:45:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/95.216.220.103/tcp/30334/ws/p2p/12D3KooWFu3t8kw9PEefyAcz1oAndfits5nsLmHkeNxk5QP2JjYB
2023-10-05 02:45:56 [Parachain] 🔍 Discovered new external address for our node: /ip4/95.216.220.103/tcp/30333/ws/p2p/12D3KooWRmbcLPAirR1eVvvUxg7xaeGRiiPY2FRYSjB2qevmRRFR
2023-10-05 02:46:00 [Parachain] ✨ Imported #4600770 (0x9220…6988)
2023-10-05 02:46:01 [Relaychain] ⚙️ Syncing, target=#17583135 (8 peers), best: #8275064 (0x4d4b…971f), finalized #8274432 (0x651c…8f51), ⬇ 7.7MiB/s ⬆ 67.2kiB/s
2023-10-05 02:46:01 [Parachain] 💤 Idle (8 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 244.9kiB/s ⬆ 8.1kiB/s
2023-10-05 02:46:02 [Parachain] ✨ Imported #4600771 (0xfe34…77c4)
2023-10-05 02:46:06 [Relaychain] ⚙️ Syncing 31.6 bps, target=#17583135 (9 peers), best: #8275222 (0xa1fd…00c1), finalized #8274432 (0x651c…8f51), ⬇ 778.3kiB/s ⬆ 50.2kiB/s
2023-10-05 02:46:06 [Parachain] 💤 Idle (8 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 11.8kiB/s ⬆ 3.4kiB/s
2023-10-05 02:46:11 [Relaychain] ⚙️ Syncing 32.2 bps, target=#17583135 (11 peers), best: #8275383 (0x9e2a…c5d1), finalized #8274432 (0x651c…8f51), ⬇ 182.1kiB/s ⬆ 82.6kiB/s
2023-10-05 02:46:11 [Parachain] 💤 Idle (9 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 10.4kiB/s ⬆ 2.5kiB/s
2023-10-05 02:46:13 [Parachain] ✨ Imported #4600772 (0x62fa…29c6)
2023-10-05 02:46:16 [Relaychain] ⚙️ Syncing 14.5 bps, target=#17583135 (13 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 89.7kiB/s ⬆ 52.5kiB/s
2023-10-05 02:46:16 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 12.2kiB/s ⬆ 12.0kiB/s
2023-10-05 02:46:21 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583135 (17 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 129.5kiB/s ⬆ 53.8kiB/s
2023-10-05 02:46:21 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0 ⬆ 0 202
3-10-05 02:46:26 [Parachain] ✨ Imported #4600773 (0x5c59…127f) 2023-10-05 02:46:26 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583136 (21 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 94.7kiB/s ⬆ 50.3kiB/s 2023-10-05 02:46:26 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.5kiB/s ⬆ 0.8kiB/s 2023-10-05 02:46:26 [Parachain] ✨ Imported #4600773 (0xa24f…1c07) 2023-10-05 02:46:31 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583137 (25 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 165.0kiB/s ⬆ 66.2 2kiB/s 2023-10-05 02:46:31 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.8kiB/s ⬆ 6.4kiB/s 2023-10-05 02:46:36 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583138 (29 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 141.6kiB/s ⬆ 59.3kiB B/s 2023-10-05 02:46:36 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 69 B/s ⬆ 5.4kiB/s 2023-10-05 02:46:36 [Parachain] ✨ Imported #4600774 (0xe280…6094) 2023-10-05 02:446:41 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583139 (37 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 117.2kiB/s ⬆ 49.5kiB/s 2023-10-05 02:46:41 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 14.6kiB/s ⬆ 11.6kiB/s 2023-10-05 02:46:46 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583139 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 94.2kiB/s ⬆ 42.2kiB/s
2023-10-05 02:46:46 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
2023-10-05 02:46:48 [Parachain] ✨ Imported #4600775 (0x6fc1…6340)
2023-10-05 02:46:48 [Parachain] ✨ Imported #4600775 (0xd526…a2d5)
2023-10-05 02:46:51 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583141 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 124.2kiB/s ⬆ 69.8kiB/s
2023-10-05 02:46:51 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 20.8kiB/s ⬆ 15.3kiB/s
2023-10-05 02:46:56 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 155.8kiB/s ⬆ 119.5kiB/s
2023-10-05 02:46:56 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.1kiB/s ⬆ 0.1kiB/s
2023-10-05 02:47:01 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 331.9kiB/s ⬆ 209.6kiB/s
2023-10-05 02:47:01 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 7.5kiB/s ⬆ 2.6kiB/s
2023-10-05 02:47:02 [Parachain] ✨ Imported #4600776 (0x9c67…c1b3)
2023-10-05 02:47:06 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 225.0kiB/s ⬆ 210.5kiB/s
2023-10-05 02:47:06 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.3kiB/s ⬆ 6.2kiB/s
2023-10-05 02:47:11 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 453.6kiB/s ⬆ 331.9kiB/s
2023-10-05 02:47:11 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 12 B/s ⬆ 31 B/s
2023-10-05 02:47:12 [Parachain] ✨ Imported #4600777 (0x9bcc…b8f3)
2023-10-05 02:47:16 [Relaychain] ⚙️ Syncing 3.7 bps, target=#17583142 (40 peers), best: #8275475 (0xc218…3723), finalized #8275456 (0xf374…335f), ⬇ 1.2MiB/s ⬆ 285.1kiB/s
2023-10-05 02:47:16 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 31.6kiB/s ⬆ 17.9kiB/s
2023-10-05 02:47:21 [Relaychain] ⚙️ Syncing 30.0 bps, target=#17583143 (40 peers), best: #8275625 (0x9db1…21de), finalized #8275456 (0xf374…335f), ⬇ 910.9kiB/s ⬆ 346.3kiB/s
2023-10-05 02:47:21 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.1kiB/s ⬆ 0.1kiB/s
2023-10-05 02:47:26 [Relaychain] ⚙️ Syncing 33.2 bps, target=#17583143 (40 peers), best: #8275791 (0x0547…9a7e), finalized #8275456 (0xf374…335f), ⬇ 329.8kiB/s ⬆ 319.8kiB/s
2023-10-05 02:47:26 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.8kiB/s ⬆ 0.2kiB/s
2023-10-05 02:47:31 [Relaychain] ⚙️ Syncing 33.2 bps, target=#17583144 (40 peers), best: #8275957 (0x24ea…4287), finalized #8275456 (0xf374…335f), ⬇ 396.9kiB/s ⬆ 320.8kiB/s
2023-10-05 02:47:31 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.5kiB/s ⬆ 0.5kiB/s
2023-10-05 02:47:36 [Relaychain] ⚙️ Syncing 10.8 bps, target=#17583144 (40 peers), best: #8276011 (0x499e…7ad2), finalized #8276005 (0xc890…fa1d), ⬇ 326.3kiB/s ⬆ 308.0kiB/s
2023-10-05 02:47:36 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 12 B/s ⬆ 12 B/s

.... .... the RAM usage keeps increasing..... (the capacity is 16GB )

2023-10-05 05:45:07 [Relaychain] ⚙️ Syncing 37.5 bps, target=#17584918 (36 peers), best: #8648566 (0x47ee…85fd), finalized #8648192 (0xd17f…ce76), ⬇ 732.9kiB/s ⬆ 23.1kiB/s
2023-10-05 05:45:08 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 51 B/s ⬆ 51 B/s
2023-10-05 05:45:12 [Relaychain] ⚙️ Syncing 32.3 bps, target=#17584918 (35 peers), best: #8648728 (0x3c2d…347d), finalized #8648704 (0x958b…965a), ⬇ 532.0kiB/s ⬆ 36.0kiB/s
2023-10-05 05:45:13 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 51 B/s ⬆ 51 B/s
2023-10-05 05:45:17 [Relaychain] ⚙️ Syncing 35.7 bps, target=#17584919 (39 peers), best: #8648907 (0x80e7…1ab7), finalized #8648704 (0x958b…965a), ⬇ 881.9kiB/s ⬆ 37.2kiB/s
2023-10-05 05:45:18 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 23.8kiB/s ⬆ 0.3kiB/s
2023-10-05 05:45:22 [Relaychain] ⚙️ Syncing 37.2 bps, target=#17584920 (37 peers), best: #8649093 (0x6140…4b83), finalized #8648704 (0x958b…965a), ⬇ 618.9kiB/s ⬆ 29.2kiB/s
2023-10-05 05:45:23 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 51 B/s ⬆ 51 B/s
2023-10-05 05:45:27 [Relaychain] ⚙️ Syncing 30.2 bps, target=#17584921 (30 peers), best: #8649244 (0xc189…7beb), finalized #8649216 (0x6017…599c), ⬇ 528.7kiB/s ⬆ 37.7kiB/s
2023-10-05 05:45:28 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 18.1kiB/s ⬆ 0.2kiB/s
2023-10-05 05:45:32 [Relaychain] ⚙️ Syncing 35.5 bps, target=#17584922 (31 peers), best: #8649422 (0xde64…6078), finalized #8649216 (0x6017…599c), ⬇ 1023.3kiB/s ⬆ 47.8kiB/s
2023-10-05 05:45:33 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 70 B/s ⬆ 64 B/s
2023-10-05 05:45:37 [Relaychain] ⚙️ Syncing 37.1 bps, target=#17584923 (30 peers), best: #8649608 (0xd02a…1bdb), finalized #8649216 (0x6017…599c), ⬇ 535.2kiB/s ⬆ 43.8kiB/s
2023-10-05 05:45:38 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 18.7kiB/s ⬆ 0.7kiB/s
2023-10-05 05:45:42 [Relaychain] ⚙️ Syncing 29.3 bps, target=#17584923 (35 peers), best: #8649755 (0xd6b5…eaff), finalized #8649728 (0x6963…9fab), ⬇ 741.0kiB/s ⬆ 25.1kiB/s
2023-10-05 05:45:43 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 44 B/s ⬆ 44 B/s
2023-10-05 05:45:47 [Relaychain] ⚙️ Syncing 36.1 bps, target=#17584924 (32 peers), best: #8649936 (0x2e21…4b3a), finalized #8649728 (0x6963…9fab), ⬇ 1008.9kiB/s ⬆ 44.4kiB/s
2023-10-05 05:45:48 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 70 B/s ⬆ 70 B/s
2023-10-05 05:45:52 [Relaychain] ⚙️ Syncing 36.3 bps, target=#17584925 (32 peers), best: #8650118 (0x5810…bcba), finalized #8649728 (0x6963…9fab), ⬇ 538.1kiB/s ⬆ 41.3kiB/s
2023-10-05 05:45:53 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 35.0kiB/s ⬆ 0.4kiB/s
2023-10-05 05:45:57 [Relaychain] ⚙️ Syncing 24.3 bps, target=#17584926 (36 peers), best: #8650240 (0x717c…1837), finalized #8649728 (0x6963…9fab), ⬇ 543.7kiB/s ⬆ 38.4kiB/s
2023-10-05 05:45:58 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 1.2kiB/s ⬆ 1.1kiB/s
2023-10-05 05:46:02 [Relaychain] ⚙️ Syncing 5.1 bps, target=#17584927 (35 peers), best: #8650266 (0x212c…6a77), finalized #8650256 (0xa7a7…c90a), ⬇ 332.9kiB/s ⬆ 44.3kiB/s
2023-10-05 05:46:03 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 57.2kiB/s ⬆ 3.3kiB/s
2023-10-05 05:46:07 [Relaychain] ⚙️ Syncing 6.5 bps, target=#17584928 (37 peers), best: #8650299 (0x206e…0382), finalized #8650256 (0xa7a7…c90a), ⬇ 123.3kiB/s ⬆ 60.3kiB/s
2023-10-05 05:46:08 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
2023-10-05 05:46:12 [Relaychain] ⚙️ Syncing 28.7 bps, target=#17584928 (40 peers), best: #8650443 (0x993e…bad7), finalized #8650256 (0xa7a7…c90a), ⬇ 599.3kiB/s ⬆ 53.4kiB/s
2023-10-05 05:46:13 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 6.3kiB/s ⬆ 2.0kiB/s
2023-10-05 05:46:17 [Relaychain] ⚙️ Syncing 13.1 bps, target=#17584929 (40 peers), best: #8650509 (0x224d…cbd2), finalized #8650256 (0xa7a7…c90a), ⬇ 501.6kiB/s ⬆ 56.0kiB/s
2023-10-05 05:46:20 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 10.7kiB/s ⬆ 0.4kiB/s
2023-10-05 05:46:22 [Relaychain] ⚙️ Syncing 3.1 bps, target=#17584929 (40 peers), best: #8650525 (0x2ea6…487b), finalized #8650256 (0xa7a7…c90a), ⬇ 59.3kiB/s ⬆ 36.9kiB/s
2023-10-05 05:46:25 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 76 B/s ⬆ 70 B/s
2023-10-05 05:46:35 [Relaychain] ⚙️ Syncing 1.2 bps, target=#17584931 (38 peers), best: #8650541 (0x9912…7c21), finalized #8650256 (0xa7a7…c90a), ⬇ 17.4kiB/s ⬆ 9.9kiB/s
2023-10-05 05:46:35 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 1.0kiB/s ⬆ 0.2kiB/s
2023-10-05 05:46:40 [Relaychain] ⚙️ Syncing 13.3 bps, target=#17584933 (40 peers), best: #8650608 (0x3629…7e29), finalized #8650256 (0xa7a7…c90a), ⬇ 346.5kiB/s ⬆ 92.9kiB/s
2023-10-05 05:46:40 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 53.6kiB/s ⬆ 1.2kiB/s
2023-10-05 05:46:45 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 76 B/s ⬆ 76 B/s
2023-10-05 05:46:45 [Relaychain] ⚙️ Syncing 23.7 bps, target=#17584934 (38 peers), best: #8650727 (0xd3c8…29de), finalized #8650256 (0xa7a7…c90a), ⬇ 473.6kiB/s ⬆ 11.1kiB/s
[11719.707590] Out of memory: killed process 1196 (acala) total-vm:641883904KB, anon-rss:15479608KB, file-rss:0KB, shmem-rss:0KB, UID:1000pgtables:52380KB, oom_score_adj:0

@xlc
Copy link
Member

xlc commented Oct 5, 2023

Can you try a different server? Your report looks similar to this one #2576 and it is the server issue.

@SixoneKui
Copy link
Author

Thanks, I will try. Now I am using Hetzner Cloud .

@SixoneKui
Copy link
Author

SixoneKui commented Nov 4, 2023

this memory issue seems still there with other Sever (digital ocean)
every time the process reaches 8551424 then the ram usages goes high, above 16GB, and then the procedure is killed.
I did not run it on a mounted disk.
This is the command i used.
docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933

23.3kiB/s ⬆ 0.4kiB/s
2023-11-04 02:35:04 [Relaychain] ⚙️ Syncing 0.0 bps, target=#18014281 (32 peers), best: #8551424 (0x8867…262f), finalized #8550400 (0x940d…516a), ⬇ 51.3kiB/s ⬆ 23.8kiB/s
2023-11-04 02:35:04 [Parachain] 💤 Idle (9 peers), best: #4815393 (0x31d5…5d57), finalized #31174 (0x9caf…18b8), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
2023-11-04 02:35:07 [Parachain] ✨ Imported #4815397 (0x096e…59e9)
2023-11-04 02:35:07 [Parachain] ✨ Imported #4815397 (0x793c…ce3d)
2023-11-04 02:35:09 [Relaychain] ⚙️ Syncing 0.0 bps, target=#18014283 (38 peers), best: #8551424 (0x8867…262f), finalized #8550400 (0x940d…516a), ⬇ 59.7kiB/s ⬆ 28.7kiB/s
2023-11-04 02:35:09 [Parachain] 💤 Idle (10 peers), best: #4815393 (0x31d5…5d57), finalized #31174 (0x9caf…18b8), ⬇ 21.4kiB/s ⬆ 9.6kiB/s
2023-11-04 02:35:14 [Relaychain] ⚙️ Syncing 0.0 bps, target=#18014285 (36 peers), best: #8551424 (0x8867…262f), finalized #8550400 (0x940d…516a), ⬇ 82.2kiB/s ⬆ 26.3kiB/s
2023-11-04 02:35:14 [Parachain] 💤 Idle (10 peers), best: #4815393 (0x31d5…5d57), finalized #31174 (0x9caf…18b8), ⬇ 84 B/s ⬆ 88 B/s

info:

2023-11-04 02:30:22 Acala Node
2023-11-04 02:30:22 ✌️ version 2.20.0-unknown
2023-11-04 02:30:22 ❤️ by Acala Developers, 2019-2023
2023-11-04 02:30:22 📋 Chain specification: Acala
2023-11-04 02:30:22 🏷 Node name: aaaa
2023-11-04 02:30:22 👤 Role: FULL
2023-11-04 02:30:22 💾 Database: RocksDb at /acala/data/chains/acala/db/full
2023-11-04 02:30:22 Parachain id: Id(2000)
2023-11-04 02:30:22 Is collating: no
2023-11-04 02:34:18 [Relaychain] 🏷 Local node identity is: 12D3KooWC7TL1xHcXm9kDmRH4taWCqXkA8XrRe15sjcNxtDLHFuv

2023-11-04 02:34:19 [Relaychain] 💻 Operating system: linux
2023-11-04 02:34:19 [Relaychain] 💻 CPU architecture: x86_64
2023-11-04 02:34:19 [Relaychain] 💻 Target environment: gnu
2023-11-04 02:34:19 [Relaychain] 💻 CPU: DO-Premium-Intel
2023-11-04 02:34:19 [Relaychain] 💻 CPU cores: 8
2023-11-04 02:34:19 [Relaychain] 💻 Memory: 15983MB
2023-11-04 02:34:19 [Relaychain] 💻 Kernel: 6.5.0-9-generic
2023-11-04 02:34:19 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.4 LTS
2023-11-04 02:34:19 [Relaychain] 💻 Virtual machine: yes
2023-11-04 02:34:19 [Relaychain] 📦 Highest known block at #8550912
2023-11-04 02:34:19 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost
:", "http://127.0.0.1:", "https://localhost:", "https://127.0.0.1:", "https://polkadot.js.org"]
2023-11-04 02:34:19 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616
2023-11-04 02:34:19 [Relaychain] Starting with an empty approval vote DB.
2023-11-04 02:34:19 [Parachain] 🏷 Local node identity is: 12D3KooWMTYcaYnW4QiY8fjev87LUXT6j4abZ9yPNSy8rhARrg1r

2023-11-04 02:34:19 [Parachain] 💻 Operating system: linux
2023-11-04 02:34:19 [Parachain] 💻 CPU architecture: x86_64
2023-11-04 02:34:19 [Parachain] 💻 Target environment: gnu
2023-11-04 02:34:19 [Parachain] 💻 CPU: DO-Premium-Intel

Could you please check the reason ?

@xlc
Copy link
Member

xlc commented Nov 4, 2023

If you are not build an archive mode, I will suggest enable wrap sync

docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

@SixoneKui
Copy link
Author

If you are not build an archive mode, I will suggest enable wrap sync

docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

thanks I just tried. Let's see how it works

docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

"-- --sync=warp" is part of the command, right?
thanks again

@xlc
Copy link
Member

xlc commented Nov 4, 2023

yes. the flags before double dash is for parachain and after is for relaychain. in this case, we want to enable warp sync for both relaychain and parachain and therefore two of it

@SixoneKui
Copy link
Author

Hi, this command works. Thank you.

But I met another problem.
I tried the command below. I remember --ws-port=9944 works before. But it now gives "error: unexpected argument '--ws-external' found"

root@acala-2:~/kkc/acala# sudo docker run -it --privileged=true -v /mnt/volume_fra1_01:/acala/data acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --ws-external --rpc-external --rpc-cors=all --ws-port=9944 --rpc-port=9933 --sync=warp -- --sync=warp
error: unexpected argument '--ws-external' found

tip: a similar argument exists: '--rpc-external'

Usage: acala --name --rpc-external <--chain <CHAIN_SPEC>|--dev|--base-path |--log <LOG_PATTERN>...|--detailed-log-output|--disable-log-color|--enable-log-reloading|--tracing-targets |--tracing-receiver > [-- <RELAY_CHAIN_ARGS>...]

For more information, try '--help'.
root@acala-2:~/kkc/acala#

Now I have a node running with

sudo docker run -it --privileged=true -v /mnt/volume_fra1_01:/acala/data acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

I tried ws://172.17.0.2:9944 to connect but it does not work.

How do I connect to WS?

THank you again.

@xlc
Copy link
Member

xlc commented Nov 6, 2023

the http port and ws port are merged so you only need one specified with --rpc-port. please check the release note https://github.com/AcalaNetwork/Acala/releases/tag/2.20.0

@qezz
Copy link

qezz commented Apr 23, 2024

We also observe an OOM while syncing a node.

image

More over, the acala's finalized block went to 0, and its catching up speed doesn't look good.
image

We run it as follows:

ExecStart=/opt/validators/acala/bin/2.24.0/acala \
        --base-path=/opt/validators/acala/data \
        --chain=acala \
        --name=chorusone-rpc-1 \
        --pruning=archive \
# workaround for substrate bug, see https://github.com/paritytech/substrate/issues/9697
        --state-cache-size 0 \
        --rpc-external \
        --rpc-cors=all \
        --rpc-port=9944 \
        --rpc-max-connections=2000 \
        --execution=wasm \
        --prometheus-port 10140 \
        --prometheus-external \
        --listen-addr=/ip4/0.0.0.0/tcp/30333 \
        --listen-addr=/ip4/0.0.0.0/tcp/30335 \
        -- \
        --chain=polkadot \
        --prometheus-external \
        --prometheus-port 9616

We also run the eth-rpc-adapter alongside, but I believe since Relaychain didn't sync yet, it wasn't doing anything useful.

@xlc
Copy link
Member

xlc commented Apr 24, 2024

@qezz it is expected the finalized block to be 0 while the relaychain part is syncing
can you share some logs

@qezz
Copy link

qezz commented Apr 24, 2024

it is expected the finalized block to be 0 while the relaychain part is syncing

It synced from 0 to (current head) in a day, and then after another day dropped to 0. Is this expected?

can you share some logs

Will the logs for the OOM loop work? I stopped the node for now, so it doesn't cause issues for collocated processes.
It was OOMing in a loop within ~1h, i.e. it crashed on OOM, restarted, and then reached OOM within an hour, and restarted again etc. etc.
I don't think I have logs before that, as they are probably rotated by this point.

@qezz
Copy link

qezz commented Apr 24, 2024

it is expected the finalized block to be 0 while the relaychain part is syncing
It synced from 0 to (current head) in a day, and then after another day dropped to 0. Is this expected?

Sorry, I'm confusing things here. I mean finalized height for Parachain, and that's what shown on the screenshot

Edit:

Relaychain's finalized height was indeed 0, and that's OK. It has started growing after ~2 days of syncing, and then it went OOM after several hours

@xlc
Copy link
Member

xlc commented Apr 24, 2024

ok it is not expected for the parachain finalized height goes from a non zero value to zero

Will the logs for the OOM loop work?

that will be useful

@SixoneKui
Copy link
Author

Hi,

I tried to enable wrap sync with:
--sync=warp -- --sync=warp
But I see below

Error: Service(Other("Warp sync enabled, but no warp sync provider configured."))

how to solve this ? or is the sync provider configured on my side ?

(this is not form acala node but from parallel, could please also give some clues)

Many thanks

@xlc
Copy link
Member

xlc commented Apr 26, 2024

@SixoneKui this is not related to the original issue and it is not even related to this repo so please avoid posting unrelated content in the issue next time.

for this particular question, my guess is that warp sync is not enabled in the client but I did not check their code and can't really tell

@qezz
Copy link

qezz commented Apr 30, 2024

The logs are basically empty on the OOM loop

# journalctl -alxfu acala --no-hostname
-- Logs begin at Tue 2024-04-30 04:06:54 UTC. --
Apr 30 12:53:52 acala[933902]: CLI parameter `--execution` has no effect anymore and will be removed in the future!
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 Acala Node
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 ✌️  version 2.24.0-unknown
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 ❤️  by Acala Developers, 2019-2024
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 📋 Chain specification: Acala
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 🏷  Node name: chorusone-rpc-1
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 👤 Role: FULL
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 💾 Database: RocksDb at /opt/validators/acala/data/chains/acala/db/full
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 Parachain id: Id(2000)
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 Is collating: no

no more logs after this ^, and it's already going OOM

image

@xlc
Copy link
Member

xlc commented May 1, 2024

This issue is very unlikely caused by Acala specific code as we don't really have any non standard code compare to other parachains.

I've made a new release branch prepare for 2.25.0 release: https://github.com/AcalaNetwork/Acala/tree/release-acala-2.25.0
Could you give it a try and see if it helps? It includes a new version of polkadot-sdk.

The docker image can be found here: https://github.com/AcalaNetwork/Acala/pkgs/container/acala-node/210655705?tag=2.25.0

@bkchr
Copy link

bkchr commented May 1, 2024

Please also run with -ltrace to get more logs.

@qezz
Copy link

qezz commented Jun 21, 2024

I didn't manage to make it sync from genesis. The solution for me was to rsync a datadir from one node to another

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants