Hello World CI/CD Pipeline for Infrastructure As Code in Azure DevOps

When we think of automating any process end-to-end, we think of automating each and every end of that particular process. A term pops up into our mind is that we need “No Human Intervention” in the whole process or pipeline. As a DevOps and Cloud enthusiast let’s think of deploying an Infrastructure either on any cloud provider (AWS, Google Cloud Platform or Microsoft Azure) or on any cluster of bare-metal servers or machines. There are a lot of tools and technologies currently for automation and provisioning of infrastructure neatly and in an optimized manner.

When we talk about automating the process of code integration, its testing and its deployment over a cloud based environment, the first term came to our mind is a “pipeline”, which is basically a process that drives software development through a path of building, testing, and deploying code, also known as CI/CD. The tools which are used in those pipeline can compile code, perform unit tests, perform code analysis, security and vulnerabilities assessment, and binaries creation.

In this blog, we will be looking at the process of creation of a CI/CD pipeline in Azure DevOps environment, for the terraform code which is also hosted in the same Azure DevOps environment.

Tools and Components to be used:

The following tools and components are to be used in order to setup a basic project and understand the usage of Azure DevOps and CI/CD.

• Azure DevOps as the version control system to host code repositories.
• Azure DevOps Pipeline as the CI/CD pipeline tool.
• Terraform as the Infrastructure as Code (IaC) tool.
• Azure Cloud as the cloud service provider for the services.
• Azure Subscription: If we don’t have an Azure subscription, we can create a free account at https://azure.microsoft.com before we start.
• Azure Service Principal: is an identity used to authenticate to Azure. Below are the instructions to create one.
• Azure Remote Backend for Terraform: we will store our Terraform state file in a remote backend location. We will need a Resource Group, Azure Storage Account and a Container.

Deploying Terraform using Azure DevOps

Setting up the Azure DevOps to deploy your Terraform into Azure. The pipeline will be consisting of the following steps or stages;

1. Validate: Azure DevOps pipeline job that initialize our code and identify if there is any drift configuration between what’s in the code and what’s really deployed. Here we will execute “Terraform init” and “Terraform validate”.
2. Plan: Here we will execute “Terraform plan”. If Infrastructure changes are noticed this build will publish an Artifact.
3. Build/Artifact: It will be generated by our pipeline whenever the pipeline status is successful and when any Infrastructure changes are identified. After you run a Terraform plan against the environment, you can .zip these contents along with the additional configurations stored in git into a build artifact on that pipeline. The build artifact is stored within the pipeline, so when Terraform Apply is ran, it will download and unzip this artifact — using this Terraform configuration rather than pulling directly from master branch for the newer version of the Terraform code.
4. Deployment/Release: Azure DevOps pipeline job is triggered each time an Artifact is published. In this context we will use the “Terraform apply” to create or update our Infrastructure resources as per their description done in the Terraform code. Before this step we can also add a manual approval in place.

Steps involved in CI/CD Pipeline

In order to setup a pipeline for deployment of terraform code, following steps needs to be taken;

1. Sign-in to your Azure DevOps organization and go to your project.
2. Go to Pipelines, and then select New pipeline.
3. Do the steps of the wizard by first selecting Azure DevOps Repository as the location of your source code.
4. When you see the list of repositories, select your repository.
5. Azure Pipelines will analyze your repository and recommend the pre-build pipeline template. Here you can use the terraform extension as well. but I have used the bash scripts in order to execute the pipeline and perform all the jobs.
6. The folder structure will look something like this;

Folder structure for Azure DevOps terraform project

The code for the script terraform-download.sh is provided here

#!/usr/bin/env bash
set -euo pipefail
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform
terraform --version

Here I have also provided the yml configuration for the pipeline used to deploy a simple terraform code;

Azure DevOps pipeline code for deployment of terraform code.

Note that the variables I have used in the pipeline like;

$(AZURE_CLIENT_ID), $(AZURE_TENANT_ID), $(STORAGE_ACCOUNT_NAME)...

There are multiple ways to use them in your pipeline code, One of them is that you can use Azure Key-Vault in order to store them as a secrets and retrieve them here in your pipeline code. The other way is to store them in Library inside the Azure DevOps environment.

Flow Diagram of the CI/CD pipeline in Azure DevOps

Flow Diagram for the CI/CD Pipeline

In a Nutshell

Throughout this tutorial, you have learned how to integrate Terraform into your Azure DevOps project to automate the management of your infrastructure. The pipeline I showed was a simple execution, you can configure this further depending on your requirements and you can also use the built-in terraform extension from Azure marketplace and perform the terraform init, plan and apply process.

Here I have put the basic project on github as well;

https://github.com/syedsaadahmed/Azure-DevOps-Terraform-CI-CD