Migration from the production environment to AWS EC2 raises a lot of questions, ranging from pricing to services availability. One of the critical aspects of migration for us was deployment security and, to a big extent, confidentiality of the critical data that we move to AWS.

Once moved outside of our premises, the data must be encrypted in transit and at rest. An analysis of AWS EC2 encrypted volumes led us to the conclusion that any encrypted volume and its snapshot can be decrypted in the EC2 environment as long as the decryption key can be accessed by the account used to log in. So, if the account is compromised, an attacker would be able to access data on encrypted volume(s), and even detached volumes and volume snapshots. This was a deal breaker.

Full disk encryption within AWS EC2

After looking at our options, we have selected the approach with the full disk (partition) encryption within AWS EC2 instance relying on Linux encryption techniques, namely cryptsetup facility with LUKS key management. This gives us the following benefits:

  1. More control over who can decrypt the volumes (even though you would need to unlock them manually on instance maintenance and reboot)
  2. The ability to revoke compromised and retired passwords
  3. Rebooted instance, detached volume, and any volume backup would not be decrypted
  4. Higher chance that even Amazon itself would not be able to eavesdrop on data at rest

We cannot encrypt everything including the boot volume in this manner since the lack of console access would not allow entering a password at boot time. But this approach is well suited for separate disks/partitions with critical data. As a side note, there is still a chance to decrypt the data inside a running EC2 instance, but that is another story of “physical” instance protection.

File Recovery

So what about file recovery in such a configuration? We are using our own product, NAKIVO Backup & Replication, to back up our environment at AWS and we have the file recovery feature, that enables restoring files from “cold backups”. However, file recovery from encrypted disks must fail (and actually does) since the data is encrypted and no “default” passwords/keys are available.

As in many other cases - if something cannot be done directly in the product interface, it is still possible for an administrator to solve the task at hand. Let's see how:

  1. Open a file recovery session in NAKIVO for the backup and recovery point of interest
  2. Log in via SSH into the Transporter that is assigned to the repository
  3. Identify the attached encrypted disk/partitions to recover from
  4. Open the encrypted disk/partition by providing the right passwords
  5. Mount the decrypted volume
  6. Find and recover files you need
  7. Unmount the decrypted volume
  8. Close the encrypted disk/partition
  9. Close the file recovery session

File Recovery Steps

So let’s quickly walk through the steps with a bit more detail and open a backup recovery point for file recovery:

File Recovery from Encrypted Volumes in AWS EC2 illustration 1

Log in via SSH (either with an ssh client or the putty utility, depending on your preferences).

>ssh root@TRANSPORTER_IP_ADDRESS

Check that disks from backup are mounted:

>iscsiadm -m session

Identify disk and partition within the system view (via dmesg or lsblk):

>dmesg | tail -n 50

Where you should see a disk label (like sdc) and its partitions (like sdc1, sdc2, …) with sizes. Additionally, the parted tool can be used for additional verification that the right disk/partition is selected:

>parted /dev/sdc print

File Recovery from Encrypted Volumes in AWS EC2 illustration 2

Once you know which disk/partition you need to decrypt and mount, run the following command and enter a valid password when prompted (this should create a new decrypted device entry point under /dev/mapper with the name that you specified as a third argument to the command):

>cryptsetup luskOpen /dev/sdc2 recover

Create (or use an existing) folder and mount the decrypted drive into it:

>mkdir 1; mount /dev/mapper/recover 1

Find and grab the files you need!

When you’re done, do the proper teardown:

> umount /dev/mapper/recover

> cryptsetup luskClose recover

File Recovery from Encrypted Volumes in AWS EC2 illustration 3

Now you can close the file recovery wizard so the software would do its own cleanup steps.

Thoughts

The same technique can be applied to other encryption types and features of file systems not directly available from NAKIVO (like BitLocker, TrueCrypt, etc.). Just remember that the Transporter OS should be capable of supporting the feature and be brave to try.