Re-Creating node from scratch does not copy tables for the Postgres and Kafka engines #1455
Comments
|
@Hubbitus , have you used latest 0.23.6 or earlier release? |
|
@alex-zaitsev, thank you for the response. That was in older version. Now we have updated operator. What is a correct way to re-init node? Is it enough to just delete PVC of failed node and delete POD? |
|
@Hubbitus , if you want to re-init the existing node, delete STS, PVC, PV and start re-concile. Do you have multiple replicas? |
|
@alex-zaitsev, thank you for the reply. I understand how to delete objects. But what you are meant under "start re-concile"? I have two replicas
|
|
@Hubbitus , we have released 0.23.7 that is more aggressive re-creating the schema. So you may try to delete PVC/PV completely, and let it to re-create the objects. |
|
@alex-zaitsev, thank you very much! And doing in ArgoCD:
Then I see pod is up and running.
SELECT hostname() as node, COUNT(*)
FROM clusterAllReplicas('{cluster}', system.tables)
WHERE database NOT IN ('INFORMATION_SCHEMA', 'information_schema', 'system')
GROUP BY node
And also error in log like: So, I see only tables in |
|
Notes:
Note, the order is important, but local_directory may be skipped if you are not using it. Keep it, if there are users defined with CREATE USER already, otherwise they disappear at all.
Others should work, so operator log is needed to check what went wrong. The correct PVC recovery sequence is:
Looks like since you have deleted PVC and Pod, the recovery has been handled by Kubernetes (STS), and Operator even did not know that PVC has been recreated. So make sure you delete STS as well. Also consider using operator managed persistance: |
|
@alex-zaitsev, very thank you for the answer. First I would like to recover my tables, then I will try to deal with users. Today, I eventfully receive rights to see operator pod in kube-system namespace. |
|
As we are speaking, I have tried to reconcile cluster by providing: spec:
taskID: "click-reconcile-1"Indeed, that looks like triggering reconcile. Logs of operator pod: Not sure what going wrong, but on host |
|
@alex-zaitsev, could you please look on it? |
@Hubbitus is your cluster have 2 shards with only 1 replica inside shard? Could you share: |
|
@Slach, thanks to response. Output of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We use your Operator to manage Clickhouse cluster. Thank you.
After some hardware failure we reset PVC (and zookeeper namespace) to re-create one clickhouse node.
Most of metadata like views, materialized views and tables with most engines (
MergeTree,ReplicatedMergeTreeetc.) was successfully re-created on the node and replication was started.Meantime none of Postgres and Kafka based engines tables was recreated.
Is it a bug, or we need to use some commands or hacks to sync all metadata across the cluster?
The text was updated successfully, but these errors were encountered: