Re-attachable EBS volumes and Terraform

I’ve recently been migrating my company infrastructure from Rackspace to AWS. Part of that process has been setting up servers for my private Git source code and my Jenkins deployment pipeline. My new infrastructure is fully software defined and uses boto3 and Terraform. Part of my recent struggles have been about preparing an EBS volume to store data for Jenkins and Git that can simply be detached and reattached to the updated instance.

To use the drive it needs to be formatted, my initial attempts involved conditionally partitioning and formatting the device as part of unit files using lsblk, parted and mkfs. However, the results appeared to be indeterministic, which, although my data is backed up as part of the process makes it annoying when an update wipes out the drive. This isn’t really acceptable.

It would seem that, because EBS volumes attached after the instance is created (there is no way to attach an existing EBS volume while creating instance), the checks prior to partitioning and formatting suffer race conditions which give indeterministic results.

One alternative is to preparing the volume ahead of time using a tool such as Packer, but this requires actually requires a bit of infrastructure, VPC, subnet, instance and temporary key pairs to be created to run remote commands on the instance. This removes some of the simplicity and provides a bit surface area for things to go wrong.

Realistically the drive only needs formatting the very first time the instance is booted.

1. Create format service.

data "template_file" "unit_service_prepare_dev_xvdf" {
  template = <<EOF
[Unit]
Requires=dev-xvdf.device
After=dev-xvdf.device
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -xc "\
  parted /dev/${volume} mklabel gpt mkpart primary 0%% 100%% && \
  mkfs.ext4 /dev/${volume}1"
[Install]
WantedBy=multi-user.target
EOF
}

2. Mount service requires device.

data "template_file" "unit_mount_home_git" {
  template = <<EOF
[Unit]
Requires=dev-xvdf1.device
After=dev-xvdf1.device
[Mount]
What=/dev/xvdf1
Where=/home/git
Type=ext4
[Install]
WantedBy=multi-user.target
EOF
}

If we were to leave the unit file as is, as soon as the device is partitioned we will attempt to mount the device, however, formatting must occur before mounting, to do this we can use After to hint to SystemD that our format service should be completed first if enabled before running the mount unit. We cannot use Requires because in most cases the format service will be disabled.

data "template_file" "unit_mount_home_git" {
  template = <<EOF
[Unit]
Requires=dev-xvdf1.device
After=dev-xvdf1.device prepare-dev-xvdf.service
[Mount]
What=/dev/xvdf1
Where=/home/git
Type=ext4
[Install]
WantedBy=multi-user.target
EOF
}

Terraform variables enable us to pass a conditional flag through.

data "template_file" "ignition" {
  template = <<EOF
{
  ...
  "systemd":{
    "units":[
      {"name":"prepare-dev-xvdf.service","enable":${should_prepare_volume},"contents":${unit_service_prepare_dev_xvdf}},
      {"name":"home-git.mount","enable":true,"contents":${unit_mount_home_git}},
      ...
    ]
  }
}
EOF
  var {
    should_prepare_volume = "${var.should_prepare_volume == true}"
    unit_service_prepare_dev_xvdf = "${jsonencode(data.template_file.unit_service_prepare_dev_xvdf.rendered)}"
    unit_mount_home_git = "${jsonencode(data.template_file.unit_mount_home_git.rendered)}"
  }
}

Now when we run terraform, the first time we want to bring up everything we can pass through the variable should_prepare_volume=true.

You can see the full code here.