The following options are available on pool creation, and partially also when editing a pool.

Further information on Ceph pool handling can be found in the Ceph pool operation
[Ceph pool operation https://docs.ceph.com/en/tentacle/rados/operations/pools/]
manual.

Erasure coded (EC) pools can be created with the pveceph CLI tooling. Planning an EC pool needs to account for the fact, that they work differently than replicated pools.

The default min_size of an EC pool depends on the m parameter. If m = 1, the min_size of the EC pool will be k. The min_size will be k + 1 if m > 1. The Ceph documentation recommends a conservative min_size of k + 2
[Ceph Erasure Coded Pool Recovery https://docs.ceph.com/en/tentacle/rados/operations/erasure-code/#erasure-coded-pool-recovery]
.

If there are less than min_size OSDs available, any IO to the pool will be blocked until there are enough OSDs available again.

For example, an EC pool with k = 2 and m = 1 will have size = 3, min_size = 2 and will stay operational if one OSD fails. If the pool is configured with k = 2, m = 2, it will have a size = 4 and min_size = 3 and stay operational if one OSD is lost.

To create a new EC pool, run the following command:

Optional parameters are failure-domain and device-class. If you need to change any EC profile settings used by the pool, you will have to create a new pool with a new profile.

This will create a new EC pool plus the needed replicated pool to store the RBD omap and other metadata. In the end, there will be a <pool name>-data and <pool name>-metadata pool. The default behavior is to create a matching storage configuration as well. If that behavior is not wanted, you can disable it by providing the --add_storages 0 parameter. When configuring the storage configuration manually, keep in mind that the data-pool parameter needs to be set. Only then will the EC pool be used to store the data objects. For example:

If there is a need to further customize the EC profile, you can do so by creating it with the Ceph tools directly
[Ceph Erasure Code Profile https://docs.ceph.com/en/tentacle/rados/operations/erasure-code/#erasure-code-profiles]
, and specify the profile to use with the profile parameter.

For example:

Ceph Tentacle introduces a new erasure coding I/O path called FastEC, enabled per pool via the allow_ec_optimizations flag. FastEC adds partial reads and partial writes on EC pools, substantially improving small random I/O performance — the dominant access pattern of virtual machine disk I/O. Upstream recommends it for RBD and CephFS backed storage
[Ceph Erasure Code https://docs.ceph.com/en/tentacle/rados/operations/erasure-code/]
.

FastEC has four requirements that the Ceph monitor enforces:

If any precondition is not met, ceph osd pool set <pool>-data allow_ec_optimizations 1 fails with a descriptive error — no silent degradation.

pveceph does not set the plugin or technique explicitly when creating an EC profile — both are seeded from the monitor’s osd_pool_default_erasure_code_profile option (reed_sol_van by default on both Squid and Tentacle). The technique requirement is therefore satisfied out of the box unless the operator has overridden this setting. Verify with:

If the output shows a technique other than reed_sol_van, either update the cluster-wide default or create a compatible profile explicitly and pass it via the profile property when creating the pool (see above).

Check the cluster release with:

If this reports a release older than tentacle, run ceph osd require-osd-release tentacle after all OSDs have been upgraded.

You can add an already existing EC pool as storage to Proxmox VE. It works the same way as adding an RBD pool but requires the extra data-pool option.

Ceph Squid 19.2.6 and Ceph Tentacle 20.2.4 added a key cipher called aes256k, which fixes CVE-2025-30156, an authentication bypass caused by a misuse of AES-CBC. Whether a key can use the new cipher depends on the client that reads it.

The userspace clients ship with Ceph itself, so librados, librbd, and libcephfs understand aes256k as soon as the packages are upgraded. Virtual machines normally use librbd.

The Linux kernel clients are separate. They support aes256k starting with kernel 7.0. A node must run a supported kernel, not merely have one installed. Proxmox VE 9 can use such a key while running kernel 7.0 or newer, but its 6.14 and 6.17 kernels cannot. Proxmox VE 8 does not provide a kernel that supports the new cipher.

Migrate a client key only if every client that uses it supports aes256k. Every node that can use the key through an in-kernel RBD or CephFS client must run kernel 7.0 or newer. Consider these Proxmox VE consumers:

Also check external hosts that map RBD images or mount CephFS with the same key.

First decide which cipher the monitors should use for newly created keys. This choice applies to client and service keys, so the helper does not change the long-term setting. During a live key swap, the helper temporarily sets auth_preferred_cipher to aes256k, because Ceph cannot select a cipher explicitly when it creates a pending key. The helper restores the previous value when it finishes. If a run leaves the temporary value in place, the next run with --apply attempts to restore it before continuing.

Set it if every client of this cluster can already use the new cipher. A manager or metadata server created afterwards then gets the new cipher too, instead of raising the error again. Leave it at aes if you still have to create keys for kernel clients that do not support aes256k yet, and rotate any daemon that is added during the migration by hand. Either way you can pass --key-type on a single ceph auth command to override the default for that one key. OSDs are unaffected, as ceph-volume generates their keys locally and already uses the new cipher.

Then use the helper that Proxmox VE ships for the rotation itself. It checks that the monitors accept the new cipher, rotates the key of every MGR, MDS, and OSD one daemon at a time, writes the new key where that daemon reads it from, and switches the service tickets over at the end. It runs as a dry run and shows you the plan, unless you pass --apply:

Run the helper on one node only. It reaches the other nodes over SSH and drives the whole cluster from that node. For each daemon type it touches, the helper takes the same lock as the rolling restart in the web interface. This lock is advisory: Ceph’s configuration store has no compare-and-swap operation, so two jobs started at the same time can both acquire it. Do not start another helper or a rolling restart until the current run finishes.

A daemon that is stopped is migrated as well. Ceph does not list it, so the helper takes it from what each node reports about itself, writes the key where that daemon reads it, and leaves the daemon stopped. Otherwise a single stopped daemon would block the service ticket switch at the end of the migration.

It also repairs a stale mon. key in /etc/pve/priv/ceph.mon.keyring on its own, without --rotate-mon-key and without restarting anything, since that copy only matters for monitors created later.

Pass --help for the full list of options. Some are worth knowing about here:

After a successful unrestricted run, ceph -s no longer reports AUTH_INSECURE_SERVICE_KEY_TYPE or AUTH_INSECURE_SERVICE_TICKETS.

The helper records pre-rotation daemon keys and migration progress in /etc/pve/priv/cephx-key-migration.json. It uses this information to continue recorded work after an interruption. For example, it can recover a daemon that was left stopped after Ceph stopped listing it. It also clears only the noout flags that it set. If it finds a pending key that it cannot account for on an entity it would touch, it stops and explains how to commit or clear that key. A pending key on an unrelated client entity is only reported.

That file is the only copy of the pre-rotation keys, so keep it until you are satisfied that the cluster is healthy, then delete it, as it holds them in plain text. A file left over from a different cluster makes the helper refuse to start.

If you cannot use the helper, Ceph documents the whole procedure for a cluster that is not managed by cephadm
[Upgrading and rotating cephx keys https://docs.ceph.com/en/tentacle/rados/configuration/auth-config-ref/#cephx-upgrade]
. Two of those steps need different handling on Proxmox VE, and both fail silently if you miss them.

Upstream’s last steps cover the client keys as well. Read Rotate the Client Keys below before you follow them, in particular the warning about client.osd-lockbox keys, which must never be rotated.

The shared mon. key has to be merged into /etc/pve/priv/ceph.mon.keyring as well, because pveceph mon create feeds that file to ceph-mon --mkfs. Without it, every monitor created later cannot join the cluster:

An OSD keyring lives on a tmpfs that ceph-volume rebuilds on every boot, priming it from the osd_key entry in the bluestore label of the OSD’s block device. Writing only the keyring file therefore works until the next reboot, after which the OSD comes back with its old key and can no longer authenticate. The upstream procedure covers the label as an optional step and locates the device with ceph-volume lvm list. On Proxmox VE the block symlink in the OSD’s data directory already points at it, and priming the data directory from the label afterwards makes the new key effective without writing it twice.

Set the noout flag before you stop the first OSD, or the monitors start marking a stopped OSD out after at least ten minutes and the cluster begins to rebalance:

Then, for the OSD with the ID 3, on its node:

Check the label before you rebuild the data directory from it. whoami has to be the ID of this OSD and ceph_fsid the FSID of this cluster. If they are not, the device belongs to another OSD, and priming it would overwrite that OSD’s identity as well as its key:

Then rebuild the data directory and start the OSD again:

Once every OSD is done, unset the flag again:

Remove the keyring files that the rotations leave in /root afterwards, as they hold the keys of your cluster in plain text.

Migrate a client key only if every client that uses it supports aes256k. See Key Cipher Support for the userspace and kernel requirements. If a client does not meet them, leave its key on the old cipher. AUTH_INSECURE_CLIENT_KEY_TYPE then remains as a warning, but the cluster returns to HEALTH_WARN after the service keys are migrated.

Rotating client.admin makes the command-line interface unavailable until the new key is imported. To recover from an interrupted helper run, use the independent mon. credential described above. For a manual rotation, create and test a temporary admin identity instead:

Then rotate client.admin and import the new key into both keyrings on the configuration file system that hold it:

Every node where pveceph init ran also has a copy in /etc/ceph/ceph.client.admin.keyring. Nothing reads it, as ceph.conf points at the one on the configuration file system, but it holds the most privileged key in the cluster, so replace it there too, on each node that has one.

Check that the cluster is still reachable, then remove the temporary key:

Rotate the keys of all other clients the same way and import each new key into the keyring that the client uses. Do not forget client.crash, whose keyring is at /etc/pve/ceph/ceph.client.crash.keyring, as AUTH_INSECURE_CLIENT_KEY_TYPE does not clear while it is left out.

Storages need care of their own. Every RBD and CephFS storage keeps its own copy of the key it uses, whether it points at an external cluster or at the hyper-converged one on the node. A storage created without an explicit key holds a copy of client.admin, so rotating client.admin alone already cuts that storage off. The two storage types keep the key in a different format. An RBD storage holds a full keyring:

A CephFS storage holds only the bare key, so write that one with ceph auth get-key:

Two warnings stay as long as the monitors still allow the old cipher, AUTH_INSECURE_KEYS_ALLOWED and AUTH_INSECURE_KEYS_CREATABLE. Once every service and client key is migrated, drop the old cipher to clear both:

AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE needs no action. The monitors keep three rotating service keys and hand out fresh ones by themselves, so it clears about three hours after the service tickets were switched over, or three times auth_service_ticket_ttl if you raised it. To clear it at once instead, run ceph auth wipe-rotating-service-keys. Ceph does not recommend that, as every client then has to fetch a new ticket and only clients running 19.2.6 or newer can. Sessions that are already established are not affected.

Finally, remove the keyring files that the rotations left in /root, they hold the keys of your cluster in plain text.

If kernel clients keep you from migrating the client keys, you cannot drop the old cipher either, so AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED, and AUTH_INSECURE_KEYS_CREATABLE all stay.

You can clear the last one outright, by telling the monitors to stop creating keys with the old cipher. Only do this once auth_preferred_cipher is aes256k, as every command that creates a key without an explicit --key-type fails otherwise:

Mute the two that remain, as Mute a Health Check describes. Use a sticky mute for both: neither clears until the clients are upgraded.

The Ceph Public network is the main communication channel in a Ceph cluster between the different services and clients (for example, a VM). Changing it to a different network is not as simple as changing the Ceph Cluster network. The main reason is that besides the configuration in the ceph.conf file, the Ceph MONs (monitors) have an internal configuration where they keep track of all the other MONs that are part of the cluster, the monmap.

Therefore, the procedure to change the Ceph Public network is a bit more involved:

You first need to edit the /etc/pve/ceph.conf file. Change the public_network line to match the new subnet.

After you have changed the network, you need to restart the non MON services in the cluster for the changes to take effect. Do so one node at a time! To restart all non MON services on one node, you can use the following commands on that node. Ceph has systemd targets for each type of service.

To verify that the services are listening on the new network, you can run the following command on each node:

Once the Ceph cluster is back in a fully healthy state (HEALTH_OK), and the services are listening on the new network, continue to restart the services on the host.

The last services that need to be moved to the new network are the Ceph MONs themselves. The easiest way is to destroy and recreate each monitor one by one. This way, any mention of it in the ceph.conf and the monitor internal monmap is handled automatically.

Destroy the first MON and create it again. Wait a few moments before you continue on to the next MON in the cluster, and make sure the cluster reports HEALTH_OK before proceeding.

Once all MONs are recreated, you can verify that any mention of MONs in the ceph.conf file references the new network. That means mainly the mon_host line and the [mon.HOSTNAME] sections.

One final ss -tulpn | grep ceph should show that the old network is not used by any Ceph service anymore.

The Ceph Cluster network is used for the replication traffic between the OSDs. Therefore, it can be beneficial to place it on its own fast physical network.

The overall procedure is:

You first need to edit the /etc/pve/ceph.conf file. Change the cluster_network line to match the new subnet.

After you have changed the network, you need to restart the OSDs in the cluster for the changes to take effect. Do so one node at a time! To restart all OSDs on one node, you can use the following command on the CLI on that node:

To verify that the OSD services are listening on the new network, you can either check the OSD Details → Network tab in the Ceph → OSD panel or by running the following command on the host:

Once the Ceph cluster is back in a fully healthy state (HEALTH_OK), and the OSDs are listening on the new network, continue to restart the OSDs on the next host.