How to securely decommission a MicroCloud deployment

Important

This process will erase all data associated with your MicroCloud deployment. Make copies of any data that you need to preserve before proceeding. Refer to How to back up instances and How to back up custom storage volumes for relevant details.

This guide walks you through the steps to decommission an entire MicroCloud cluster.

If you only need to decommission a single cluster member, first remove the member from the cluster. After removing the member, update the certificate on the cluster remaining in production. Then, return to this guide, skip ahead to Remove snaps, and follow the instructions in the sections that follow.

Some commands used to decommission MicroCloud are LXD or MicroCeph commands. Refer to the LXD decommissioning guide and the MicroCeph guide to removing disks for related information.

Remove offline cluster members

Use the --force flag to remove any offline cluster members (you will remove online cluster members later in the process):

sudo microcloud remove --force <offline_member_name>

Revoke remote access

List all identities that have access to LXD, then delete each identity:

lxc auth identity list
lxc auth identity delete <type>/<name_or_identifier>

List projects

Instances, profiles, and custom volumes are scoped by project. For deployments with more than one project, you must repeat some steps for each project, each time using the --project flag. You do not need to use the --project flag to decommission deployments with only one project.

Run this command to get a list of all projects:

lxc project list

Note

You can also delete a project (except the default project) and all of its project-level entities with:

lxc project delete <project_name> --force

Delete data

You can run the commands in this section on any online cluster member to delete data.

Important

Data deleted by LXD physically remains on disks and can be recovered by users with access to the disks. To prevent unauthorized data recovery, you must destroy and sanitize your data.

Stop and delete instances

For each project, stop all instances:

lxc stop --all --project <project_name>

Next, for each project, list all instances, then delete each instance:

lxc list --project <project_name>
lxc delete <instance_name> --project <project_name>

If you are unable to stop or delete an instance, use the --force flag:

lxc stop --force <instance_name> --project <project_name>
lxc delete --force <instance_name> --project <project_name>

Delete profiles

For each project, list all profiles:

lxc profile list --project <project_name>

Each project has a default profile that cannot be deleted. Delete all other profiles:

lxc profile delete <profile_name> --project <project_name>

Remove disk devices from default profiles

You cannot delete a storage pool used by an instance, profile, or custom volume. You must, therefore, remove any disk devices used by the default profiles in order to delete any storage pools or custom volumes referenced by those devices.

At a minimum, the default profile of the default project has a disk device named root that references a storage pool. Remove this device with:

lxc profile device remove default root --project default

To check for additional disk devices, view information about the default profile of each project:

lxc profile show default --project <project_name>

Remove any remaining disk devices that reference storage pools:

lxc profile device remove default <device_name> --project <project_name>

Delete custom volumes

To delete custom volumes, you must specify the storage pools used by the volumes. First, list all storage pools across projects:

lxc storage list

Next, for each storage pool, list the custom volumes. Use the --all-projects flag to view all custom volumes across projects:

lxc storage volume list <pool_name> type=custom --all-projects

Use the PROJECT column in the output to identify the project associated with each custom volume. Then delete each custom volume, specifying both the storage pool and the project:

lxc storage volume delete <pool_name> <volume_name> --project <project_name>

Delete storage pools

Note

Storage pools are not scoped by project, so you do not need to use the --project flag with lxc storage commands.

List all storage pools, then delete each one:

lxc storage list
lxc storage delete <pool_name>

Delete monitoring data

Delete data from any external systems that you used to monitor LXD events, LXD metrics, or Ceph logging, such as Loki, Prometheus, or Grafana. Refer to the documentation for those systems for details.

Remove MicroCeph OSDs

To remove MicroCeph OSDs, list all disks, then remove each one:

microceph disk list
sudo microceph disk remove <osd_id>

Finally, verify that the OSDs have been removed:

microceph disk list

Note

If you are unable to remove an OSD, use the --bypass-safety-checks flag:

sudo microceph disk remove <osd_id> --bypass-safety-checks

Remove remaining cluster members

After deleting data, you can remove the online cluster members from the cluster. First, list all cluster members:

microcloud cluster list

You can then remove most cluster members with:

sudo microcloud remove <member_name>

However, before reducing the cluster from two members to one member, you must clean up the Ceph monitor map.

Note

As you remove each member, you can run microcloud status on the remaining cluster members to verify the removal.

Remove snaps

Important

Run these commands on every machine that you decommission.

Removing MicroCloud does not erase ZFS pools (zpools) or dedicated disks used by MicroCeph as Ceph object storage daemons (OSDs). To securely decommission MicroCloud, you must destroy and sanitize your data.

Remove the MicroCloud, LXD, MicroCeph, and MicroOVN snaps. Use the --purge flag, or a snapshot of your data will be preserved:

sudo snap remove microcloud --purge
sudo snap remove lxd --purge
sudo snap remove microceph --purge
sudo snap remove microovn --purge

Note

The MicroCeph and MicroOVN snaps may not be installed if you deployed a MicroCloud without those components.

Verify that the snaps and associated data were removed. The following commands should report that none of these snaps are installed and that the /var/snap/microcloud/, /var/snap/lxd/, /var/snap/microceph/, and /var/snap/microovn directories do not exist:

snap list microcloud lxd microceph microovn
ls /var/snap/microcloud/ /var/snap/lxd/ /var/snap/microceph/ /var/snap/microovn/

Destroy and sanitize data

Data deleted with MicroCloud, LXD, or MicroCeph commands remains readable and can be recovered by users with access to disks used in your deployment. To prevent unauthorized recovery, you must physically overwrite the data. Follow your data destruction policy to securely erase or destroy the disks that you are decommissioning.

If you are decommissioning an entire MicroCloud, apply your data destruction policy to any machines used to monitor events, logs, or metrics. For clusters configured with OIDC, consult your OIDC identity provider for the steps to remove any data associated with your profile. Likewise, if you used ACME services to issue server certificates, refer to the service provider for the steps to remove any associated data.

Important

Sanitized data is irreversibly destroyed and cannot be recovered.