Configuring Backups to Amazon S3

Vertica supports the creation of backups on Amazon S3 Standard cloud storage. You can create these backups from your local cluster or from Amazon EC2 virtual servers.

Creating an S3 Configuration File

To back up to Amazon S3, you must add an [S3] section to your backup configuration file. For more information, refer to [S3]. Vertica also provides a sample S3 configuration file that you can copy and edit.

Configuring Amazon S3 Storage for Backup

Vertica supports using Amazon S3 cloud storage as a backup location. As with any storage location, you must initialize an S3 storage location with the vbr task init.

Because S3 storage does not support file locking, Vertica uses either your local file system or an Amazon EC2 file system to handle file locks during a backup. You identify this location using the s3_backup_file_system_path parameter in your vbr configuration file. During a backup, Vertica creates a locked identity file on your local or EC2 instance, and a duplicate file in your S3 backup location. As long at the files match, Vertica proceeds with the backup, releasing the lock when the backup is complete. As long as the files remain identical, you can use the S3 location for backup and restore tasks.

If the files in your locking location become out of sync with the files in your backup location, backup and restore tasks fail with an error message. You can resolve locking inconsistencies by rerunning the init task with the --s3-force-init parameter.

A typical S3 locking file reset command takes the following form:

/opt/vertica/bin/vbr --task init --s3-force-init -c filename.ini 

Configuring EC2 Authentication for Amazon S3

If you are backing up to S3 from an EC2-based cluster, you must provide authentication to your S3 host. Regardless of the authentication type you choose, your credentials do not leave your EC2 cluster. Vertica supports the following authentication types:

• IAM role
• AWS credential file
• Environment variables

IAM role - (Recommended) Create an AWS IAM role and grant that role permission to access your EC2 cluster and S3 resources. For more information, refer to the Amazon Web Services documentation.

AWS credential file - You can manually create a configuration file on your EC2 initiator host at ~/.aws/credentials.

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

For more information on credential files, refer to the Amazon Web Services documentation.

Environment variables - Amazon Web Services provides the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

Use these variables on your initiator to provide authentication to your S3 host. When your session ends, AWS deletes these variables. For more information, refer to the AWS documentation.

Encrypting Backups

Backups made to Amazon S3 can be encrypted using native server-side S3 encryption capability. For more information on Amazon S3 encryption, refer to Amazon documentation.

Vertica supports the following forms of S3 encryption:

• Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)

  • Encrypts backups with AES-256
  • Amazon manages encryption keys

• Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)

  • Encrypts backups with AES-256
  • Requires an encryption key from Amazon Key Management Service
  • Your S3 bucket must be from the same region as your encryption key
  • Allows auditing of user activity

When you enable encryption of your backups, Vertica encrypts backups as it creates them. If you enable encryption after creating an initial backup, only increments added after you enabled encryption are encrypted. To ensure that your backup is entirely encrypted, create new backups after enabling encryption.

To enable encryption, add the following settings to your configuration file:

• s3_encrypt_transport - Encrypts your backups during transmission. You must enable this parameter if you are using SSE-KMS encryption.
• s3_encrypt_at_rest - Enables encryption of your backups. If you enable encryption and do not provide a KMS key, Vertica uses SSE-S3 encryption.
• s3_sse_kms_key_id - If you are using KMS encryption, use this parameter to provide your key ID.

For more information on these settings, refer to S3 configuration settings.

The following example shows a typical configuration for KMS encryption of backups.

[S3]
s3_encrypt_transport = True
s3_encrypt_at_rest = sse				
s3_sse_kms_key_id = 6785f412-1234-4321-8888-6a774ba2aaaa