Protect Vault cluster from data loss with backups

6min
|
Vault
Consul

The objective of this document is to provide a set of standard operating procedures (SOP) for backing up a Vault cluster. It protects your Vault cluster against data corruption or sabotage of which Disaster Recovery Replication might not be able to protect against.

Vault supports a number of storage backend types. Therefore, the exact steps to backup Vault will depend on your selected storage backend. The two recommended storage backend types are Consul and Integrated Storage (also known as Raft), and so this document assumes either of these storage backends is being used.

Personas

This standard operating procedures is primarily aimed at operations personnel.

Prerequisites

The following prerequisite steps and knowledge are required in order to backup a Vault cluster. All of the following are required to understand or carry out before attempting to a backup or restore of Vault.

Working Knowledge of Vault: Some working knowledge of Vault is required in order to follow these Standard Operating Procedures.
Vault cluster configuration is defined: Vault (and Consul, where using it as a storage backend) infrastructure configured as per Vault Reference Architecture.
A cluster configuration as defined in either our Vault with Integrated Storage Reference Architecture is required.
Vault is initialised: This SOP assumes you have already initialised Vault, keyholders are available with access to the unseal keys for each, that you have access to tokens with sufficient privileges for both clusters and encrypted data is stored in the storage backend.

Manual backup procedures

Follow these steps to backup Vault manually (compatible with Consul and Vault Community Edition and Enterprise). Note that the exact steps needed to be undertaken differs dependent upon your Vault architecture.

Single Vault cluster

Take a snapshot from the leader node using the following command

$ vault operator raft snapshot save backup.snap

Store these snapshots in a secure place away from the Vault clusters.

Take a snapshot from any Consul cluster member using the following command.

$ consul snapshot save backup.snap

This will take the snapshot using a consistent mode that forwards the request to the cluster leader, and the leader will verify it is still in power before taking the snapshot.

Alternatively, if the underlying Consul cluster is in a degraded state, use the following command to create a potentially stale snapshot from any available server:

$ consul snapshot save -stale backup.snap

Store these snapshots in a secure place away from the Vault/Consul clusters.

Vault with Disaster Recovery Replication enabled

Take snapshots from the Vault Integrated Storage (Raft) cluster members on the Vault DR Primary cluster using the following command.

$ vault operator raft snapshot save primary.snap

Store these snapshots in a secure place away from the Vault clusters.

Take separate snapshots from the Consul cluster members supporting both the Vault DR Primary cluster and DR Secondary cluster using the following commands.

First, take a snapshot on the primary cluster.

$ consul snapshot save primary.snap

Take a snapshot on the secondary cluster.

$ consul snapshot save secondary.snap

Store these snapshots in a secure place away from the Vault or Consul clusters.

Vault with Performance Replication enabled

Take separate snapshots from the Vault Integrated Storage (Raft) cluster members supporting both the Performance Primary cluster and Performance Secondary cluster using the following commands.

First, take a snapshot on the primary cluster.

$ vault operator raft snapshot save primary.snap

Take a snapshot on the secondary cluster.

$ vault operator raft snapshot save secondary.snap

Store these snapshots in a secure place away from the Vault clusters.

Take separate snapshots from the Consul cluster members supporting both the Performance Primary cluster and Performance Secondary cluster using the following commands.

First, take a snapshot on the primary cluster.

$ consul snapshot save primary.snap

Take a snapshot on the secondary cluster.

$ consul snapshot save secondary.snap

Store these snapshots in a secure place away from the Vault/Consul clusters.

Automated backup procedures

Follow these steps to backup Vault using the Automated Snapshots feature (available only in Vault Enterprise).

Single Vault cluster

Enable Automated Snapshots with the following command, following the instructions on the Automated Integrated Storage Snapshots and using the options specified on the API Documentation.

$ vault write sys/storage/raft/snapshot-auto/config/[:name] [options]

Example:

The following command creates a automatic snapshot configuration named, testsnap with a user defined location and interval. It retains up to 7 snapshots before deleting any old snapshot.

$ vault write sys/storage/raft/snapshot-auto/config/testsnap \     storage_type=local \     file_prefix=testsnappy \     interval=120m \     retain=7 \     local_max_space=1000000 \     path_prefix=/opt/vault/

Tutorial

Refer to the Vault HA Cluster with Integrated Storage tutorial.

Vault with Disaster Recovery Replication enabled

Enable Automated Snapshots on the primary cluster with the following command, following the instructions on the Raft Integrated Storage Documentation and using the options specified on the API Documentation.

$ vault write sys/storage/raft/snapshot-auto/config/[:name] [options]

Vault with Performance Replication enabled

Enable Automated Snapshots with the following command, following the instructions on the Raft Integrated Storage Documentation and using the options specified on the API Documentation. Each Vault cluster (forming the Primary and Secondary Vault Performance Replicas) should be configured to create separate automated snapshots.

$ vault write sys/storage/raft/snapshot-auto/config/[:name] [options]

Follow these steps to backup Vault using the Automated Backups feature (available only in Consul Enterprise).

Single Vault cluster

Enable the Consul Snapshot agent with the following command, following the instructions on the Consul Agent Documentation

$ consul snapshot agent [options]

Vault with Disaster Recovery Replication enabled

Enable the Consul Snapshot agent with the following command, following the instructions on the Consul Agent Documentation. Each Consul cluster (servicing the Primary and Secondary DR Vault Replicas) should be configured to create separate automated snapshots.

$ consul snapshot agent [options]

Vault with Performance Replication enabled

Enable the Consul Snapshot agent with the following command, following the instructions on the Consul Agent Documentation. Each Consul cluster (servicing the Primary and Secondary Vault Performance Replicas) should be configured to create separate automated snapshots.

$ consul snapshot agent [options]

Next steps

This standard operating procedure has discussed backing up Vault data. To learn about restoring a Vault backup, read the Vault Restore Standard Procedure guide.

If you backup the data in preparation for Vault upgrade, read the Vault Upgrade Standard Procedure guide.

Collection Overview

Standard procedures

Ugrade a Vault cluster