{
  "$type": "site.standard.document",
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreigvtmthn2yoff2aey643qznyzvdsqa52k2lgahh3xl3gqe5aaw7au"
    },
    "mimeType": "image/png",
    "size": 59690
  },
  "description": "The original deploy strategy for SongRender involved using Ansible to provision and build the application servers. I was unhappy with that process for a few reasons.",
  "path": "/words/blue-green-deploys-and-immutable-infrastructure-with-terraform/",
  "publishedAt": "2020-08-22T00:00:00Z",
  "site": "at://did:plc:vrrdgcidwpvn4omvn7uuufoo/site.standard.publication/3mmyfl3pxzi2a",
  "tags": [
    "songrender",
    "terraform"
  ],
  "textContent": "The original deploy strategy for SongRender involved using Ansible to provision and build the application servers. I was unhappy with that process for a few reasons.\n\nThe application was built on production servers at the same time as they were serving live requests.\n\nOld versions of the application would stay on the servers, taking up disk space. I had set up a cron job to clean these up, but cron jobs are fickle and it would still run out of space every so often.\n\nSide effects of repeated updates and deploys (or worse, failed updates and deploys) could slowly accrue on the servers, a phenomenon known as configuration drift.\n\nI decided to solve these issues using immutable infrastructure: a strategy in which servers are provisioned once and then never updated. If a server’s configuration needs to change or a new version of the application needs to be deployed, an entirely new server is spun up to replace the old one.\n\nBlue/green deployment is a way of deploying applications that dovetails nicely with immutable infrastructure. Instead of maintaining one set of servers, you maintain two — only one of which is live at any given time. To deploy a new version of the application, you first deploy it to the backup set of servers, then make those servers live.\n\nWe’ll implement both of these practices using three technologies:\n\nPacker lets you run shell scripts to make changes to a server, then saves the entire disk — including the operating system and whatever changes you made — into a \"machine image\" that you can use as a base for other servers. We’ll use it to provision our servers, as well as to install and configure our application.\n\nTerraform is a \"configuration-as-code\" tool — you describe your desired infrastructure in a declarative language, and then Terraform diffs it with the infrastructure that actually exists. It’s kind of like React, but instead of modifying a web page it configures your infrastructure.\n\nnginx is a popular web server and load balancer. I chose it because I was already familiar with it, but it’s not particularly important that you use nginx here specifically; HAProxy or any other application load balancer should work equally well.\n\nSongRender’s infrastructure is hosted on DigitalOcean, but you can use any cloud provider that supports Terraform. This article will focus mainly on the Terraform configuration files, and the points at which they integrate with Packer and nginx.\n\nThe TL;DR of how this works: one color — let’s say blue — will be live. First, Packer builds a new image in the backup color green. Terraform creates a new set of green servers, using the image Packer just built. Finally, our nginx load balancer swaps the live color with the backup color, sending requests to the newly-created green set of servers instead of the previously live blue ones. To deploy the application again, the process is repeated with the colors reversed.\n\nSince in a given deploy both sets of servers will have gone through both live and backup states, let’s use some more specific names. The “promoted” set will be the servers that begin in the backup state and become live, while the “demoted” set will be the servers that begin live and are changed to backup.\n\nLet’s get to the code. Here’s an abbreviated Packer configuration file server.json for building the application server image:\n\nThe provisioners array should contain whatever provisioning code you need to create your application server. The important bits here are the color variable — the color that will be promoted, which we’ll pass in from the command line — and the builder’s snapshot_name, which is the name of the machine image Packer will create. This name is mostly arbitrary, but it does need to identify the color of the image. When we get to Terraform, we’ll use a regular expression to retrieve the most recent image for which the name matches each color.\n\nWe can build now the image with this command:\n\nTo get started, run this once with color=green as well so you have one image for each color.\n\nNow that we’ve created our images, we need to create the Terraform configuration. First, let’s set up our DigitalOcean provider, which will let us create and modify resources on DigitalOcean:\n\nNext, let’s create the application server. Note that the second two blocks are the same as the first two, but with green instead of blue:\n\nThe digitalocean_droplet_snapshot.blue data block retrieves the most recent machine image (like the one we just created with Packer) whose name starts with blue-. Then, the digitalocean_droplet.blue resource creates a server using that image.\n\nIf we run Terraform, it should spin up two servers with the images Packer just created:\n\nNow we should have two application servers online: one with the blue image and one with the green image. At this point, we’ve already accomplished our immutable infrastructure goal. Creating a new image for a color with Packer and then running Terraform will entirely replace the server of that color.\n\nFor the blue/green deployment, Terraform needs to know which color should be promoted. We can pass that in from the command line when we run Terraform:\n\nThe variable.color block takes either blue or green as input. Terraform configuration is declarative, so the color we pass determines which set will be promoted.\n\nThe locals block below it creates two groups: one for the promoted servers, and one for the demoted servers. The color that makes up the promoted_servers group is the same one we passed in as variable.color; it contains the servers that will be promoted. The demoted_servers group, then, contains the current set of live servers that will be demoted.\n\nNow we need to point our nginx load balancer at the correct set of servers. But before we can do that, we need to create the load balancer in the first place:\n\nI use Packer to provision the load balancer as well (which is why it uses a digitalocean_droplet_snapshot as its image) but you can provision it however you’d like.\n\nFor the purposes of this tutorial, we’ll assume your load balancer is running nginx. Here’s the configuration:\n\nThe relevant portion is that upstream.backend block up top. We use Terraform’s templating capabilities to enumerate both sets of servers. The demoted set is marked with the backup directive, which means it will be passed requests only when the promoted set is unavailable. Listing both sets of upstream servers this way prevents requests from being dropped in case the promoted set isn’t immediately ready to begin serving requests.\n\nUnfortunately, Terraform doesn’t have any built-in mechanisms for keeping this file up–to–date. That brings us to the final piece of the puzzle: the null resource.\n\nA null resource is an escape hatch within Terraform — a way to execute arbitrary code. We’ll use it to update our load balancer with an nginx configuration containing the correct server IP addresses on every deploy. Here’s what it looks like:\n\nThe top block, triggers, tells Terraform when it needs to execute the resource again. In our case, that should happen when the load balancer ID or any of the server IDs change.\n\nAfter that, there are two provisioners. The first, file, generates the nginx configuration with the correct IP addresses and uploads it to a temp folder on the load balancer’s filesystem. The next moves it to the nginx configuration folder for our site and reloads nginx.\n\nThat’s blue/green deployment! Now we can spin up a whole new set of servers and cut over to them with this pair of commands:\n\nWe’ve also gained the ability to easily roll back if we discover we’ve introduced a bug. Without building a new image, we can simply run terraform apply and flip the color back:\n\nThere’s something we can do to make this even better. With our current configuration, we’re always running two sets of servers — even when we’re only using one. That costs money!\n\nLet’s introduce another variable:\n\nWhen it’s set to true, both sets of servers will remain up. When it’s false, running terraform apply will destroy the demoted set.\n\nWe can accomplish this by slightly modifying our server configuration:\n\nThe important new line is the count property, which tells Terraform how many servers we want. Instead of always creating each set of servers, we look to var.color and var.cautious. If the server’s color matches var.color, or if var.cautious is true, we create our normal number of servers. Otherwise, we destroy all of them.\n\nOur deploy process is now two steps. First, the normal command to deploy:\n\nThis is where we check that everything is working properly. If it’s not, we flip the color and run terraform apply to roll back. If things are looking good, we run terraform apply with the same color and var.cautious set to false:\n\nYou can find the source for everything here on GitLab.",
  "title": "Blue/Green Deploys and Immutable Infrastructure with Terraform"
}