Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CN-1152]: add localConfig to persistence #171

Merged
merged 20 commits into from
Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
name: hazelcast <1>
spec:
clusterSize: 3
repository: 'docker.io/hazelcast/hazelcast-enterprise'
licenseKeySecretName: hazelcast-license-key
customConfigCmName: mapCustomConfig <2>
persistence:
pvc:
accessModes: ["ReadWriteOnce"]
requestStorage: 20Gi
restore:
localConfig:
pvcNamePrefix: "hot-restart-persistence" <3>
baseDir: "base" <4>
backupDir: "backup" <5>
backupFolder: "backup-1709045128548" <6>
16 changes: 16 additions & 0 deletions docs/modules/ROOT/examples/pod-local-pvc-content.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
apiVersion: v1
kind: Pod
metadata:
name: list-content-pod
spec:
containers:
- name: list-content-container
image: busybox
command: ["/bin/sh", "-c", "tree /data/persistence"] <1>
volumeMounts:
- name: pv-storage
mountPath: /data/persistence <2>
volumes:
- name: pv-storage
persistentVolumeClaim:
claimName: hot-restart-persistence-hazelcast-0 <3>
53 changes: 43 additions & 10 deletions docs/modules/ROOT/pages/backup-restore.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@

This topic assumes that you know about Persistence in Hazelcast. To learn about Persistence, see the xref:hazelcast:storage:persistence.adoc[Platform documentation].

There are two options for a backup: local and external. Local backups are kept in volume and never moved anywhere whereas external backups are moved into buckets provided by the user.
Backups can be either of the following:

- Local: Local backups are kept in volume and never moved anywhere.
- External: External backups are moved into buckets provided by the user.

TIP: For a working example, see link:https://docs.hazelcast.com/tutorials/hazelcast-platform-operator-external-backup-restore[this tutorial].

Expand All @@ -23,8 +26,8 @@ include::ROOT:example$/hazelcast-persistence.yaml[]
<1> <<recovery-policy, Cluster recovery policy>>.
<2> Size of the PersistentVolumeClaim (PVC) where Hazelcast data is persisted.
<3> Agent responsible for moving data from the local storage to external buckets.
The agent configuration is optional. If you enable `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.
The agent configuration is optional. If you enable `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.

NOTE: Make sure to calculate the total disk space that you will use. The total used disk space may be larger than the size of in-memory data, depending on how many <<local-backup, backups>> you take.

Expand Down Expand Up @@ -153,9 +156,11 @@ hot-backup Success

== Restoring from Local Backups

=== Restoring from Local Backups Created by Hot Backup Custom Resource

To restore a cluster from local backups, you can directly reapply the `Hazelcast` resource, which gives the cluster access to the PVCs that contain the persisted data. This will restore the Hazelcast cluster from existing `hot-restart` folders.

Or, to restore from local backups that you have taken using the `HotBackup` resource, give the `HotBackup` resource name in the restore configuration. For the restore to work correctly, make sure the status of the `HotBackup` resource is `Success`.
Or, to restore from local backups that you have taken using the `HotBackup` resource, give the `HotBackup` resource name in the restore configuration. For the restore to work correctly, make sure the status of the `HotBackup` resource is `Success`.

When this restore mechanism is used, the Restore Agent container is
deployed with the Hazelcast container in the same Pod. The agent starts as an initContainer before the Hazelcast container.
Expand All @@ -168,8 +173,36 @@ include::ROOT:example$/hazelcast-persistence-restore-cr-name.yaml[]

<1> `HotBackup` resource name used for restore. The backup folder name will be taken from the `HotBackup` resource.
<2> Agent responsible for restoring data from the local storage.
The agent configuration is optional. If you give `restore` under `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.
The agent configuration is optional. If you give `restore` under `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.

=== Restoring from Local Backups Created with other Methods

To restore from local backups that were not created by the HotBackup resource, configure `localConfig`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have titles or at least listing approach to emphasize the two different restore scenario when we have local backup. I don't know if it makes sense for you but I suggest something like this.

  • Local Backups taken with the Operator.
  • Local Backups taken with the Helm Chart

While thinking about their difference, I bring another idea. How about creating new page just for the newly added restore support? Since backup and restore features are expected as one of the main features which will be used always by the all customers who are using operator. However, newly supported restore is not actually a feature which will be used continuously. Instead, it's kinda migration solution for older operator versions and helm chart users. Also in the future we may decide to support fully custom backups. Are we going to put it also in this page? I don't think so, it'll be a migration solution as well so I suggest to separate new restore option from the existing one. Sorry for late feedback, I just got into this idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such a nice a feedback Cagri, let's talk with the team and decide together. I am not sure %100 but it might be a good idea to make it a part of migration documents.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better idea would be to put the documentation about localConfig in here, and reference it from migration documentations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we provide examples of both methods, and some guidance on when to use Operator and when to use Helm?

All information on restore ought to be in a single place; preferably with headings that clearly indicate what the section covers. For example, in this section it could be Use Restore for Migration; in the migration section it could be Migrate using a Backup.

The required configuration is best with the content it will be used with.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I committed more changes to improve the docs: 32bb2dd. I didn't mention about the helm and migration words explicitly, because in the feature we will have specific guides for those scenarios like hazelcast/hz-docs#1052. We will refer to this document from them.


[source,yaml,subs="attributes+"]
----
include::ROOT:example$/hazelcast-persistence-restore-local.yaml[]
----

<1> The name of the Hazelcast custom resource for both backup and restore must be the same. If the Hazelcast custom resource name is different, the restore fails.
<2> If you configured maps with persistence, you must create them while creating the Hazelcast CR. To create them while creating the Hazelcast CR, you must use xref:custom-config.adoc[Custom Config]. You cannot apply the Hazelcast CR and then the Map CRs.
<3> `pvcNamePrefix` is the prefix of the existing PVCs. It can be set to `persistence` or `hot-restart-persistence` depending on the installation method of the old cluster. In different versions of Hazelcast Helm Chart and Hazelcast Platform Operator, these values are used by default. Run the `kubect get pvc` command to decide which setting best suits your requirements.
<4> `baseDir` is the root directory for persistence.
<5> `backupDir` is the directory that contains a backupFolder for each available backup.
<6> `backupFolder` is the directory containing the specific backup for the restore.

TIP: To find the `backupFolder` value, you can run `kubectl exec -it <hazelcast-custom-resource-name> -c hazelcast -- /bin/bash` and list the contents of your in your existing installation. If you already deleted your installation, you can run a simple pod that lists the contents of the PVC and checks its logs. This lists the folder structure of the specified PVC. For example:

[source,yaml,subs="attributes+"]
----
include::ROOT:example$/pod-local-pvc-content.yaml[]
----
<1> Replace `/data/persistence` with the path to your PV, which is mounted inside the container.
<2> Replace `/data/persistence` with the correct mountPath specified in PV.
<3> Replace with the name of the one of the PVCs, which is mounted to the cluster from which the backup is taken.

NOTE: Agent copies the backup to be restored from `{baseDir}/{backupDir}/{backupFolder}` to `/data/persistence/base-dir`.

== Restoring from External Backups

Expand All @@ -192,8 +225,8 @@ include::ROOT:example$/hazelcast-persistence-restore-external.yaml[]
<1> Bucket URI where backup data will be restored from.
<2> Name of the secret with credentials for accessing the given bucket URI.
<3> Agent which is responsible for restoring data from the external storage.
The agent configuration is optional. If you provide `restore` under `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.
The agent configuration is optional. If you provide `restore` under `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.
--

HotBackup resource name::
Expand All @@ -206,8 +239,8 @@ include::ROOT:example$/hazelcast-persistence-restore-cr-name.yaml[]

<1> `HotBackup` resource name used for the restore. The bucket URI and secret are taken from the `HotBackup` resource.
<2> Agent responsible for restoring data from external storage.
The agent configuration is optional. If you provide `restore` under `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.
The agent configuration is optional. If you provide `restore` under `persistence` and do not pass the agent configuration, Hazelcast Platform Operator
uses the latest agent version that is compatible with its version.
--

====
Expand Down
Loading