Load Terraform Backend Output Into Ansible


During my experience with both Terrform and Ansible it became apparent to me that all available Ansible modules for Terraform do not include functionality to combine two the two automation tools together. We produce Terraform infrastructure on AWS. We try to automate absolutely everything from EC2 instances to DNS records and security groups using Terraform and then provision the servers using Ansible with required software our application code, NGINX virtual hosts and other.


The problem that we ran into was that we automated everything, but we still wrote variables from Terraform into Ansible manually, variables such as size of EBS volume sizes, EFS ids, DNS names and many more. The variables were required by our in house software or successful provision of our log collection e.g. DNS name of the central log server in each region. For larger infrastructures where many engineers are working, this cloud be an absolute nightmare, since hostnames could change or be added or AWS ARN can change for resources if rebuilt, and many more changes that would be difficult to update Ansible playbooks / roles with.



Get the module here!



Solving the Problem

I wrote Ansible module that I call fetch_terraform_backend_outputs to help us load Terraform output variables into Ansible and get rid of our manual process. Currently the module only supports s3 Terraform backend but I am planning to include Consul backend also. The module itself is very simple, it utilises boto3 python library to load Terraform tfstate file from an s3 bucket as a JSON object, then parse it to filter out outputs section. The Terraform output section is then outputed as exit_json from Ansible module, allowing it to be used in a playbook.




For this example, I’ll present my own Terraform s3 backend with Ansible debug turned on, with the following playbook. What this playbook tries to achieve is to obtain my VPC ID from my Terraform backend arn:aws:s3:::terraform-state-repository/ireland/katapult_cloud_networking.tfstate.


#!/usr/bin/env ansible-playbook
- hosts: localhost
  become: false
    - name: get Terraform networking variables
        bucket: "terraform-state-repository"
        object: "eu-west-1/networking.tfstate"
      register: networking

    - name: filter vpc id
        msg: "{{ networking.vars.katapult_cloud_vpc_id }}"



What Following playbook gives me this debug output. You can see all Terraform output variables available from tfstate file.


PLAYBOOK: terraform_output.yml **********************************************************************************************************************************************
1 plays in ./terraform_output.yml

PLAY [localhost] ************************************************************************************************************************************************************

TASK [damn san] *************************************************************************************************************************************************************
task path: ~/ansible/terraform_output.yml:5
Using module file ~/ansible/library/fetch_terraform_backend_outputs.py
<> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
ok: [localhost] => {
    "changed": false,
    "failed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": "",
            "aws_profile": "default",
            "aws_region": "us-east-1",
            "aws_secret_access_key": "",
            "bucket": "terraform-state-repository",
            "object": "eu-west-1/networking.tfstate"
    "vars": {
        "katapult_cloud_private_backup_subnet_az": "eu-west-1c",
        "katapult_cloud_private_backup_subnet_id": "subnet-24382ffe",
        "katapult_cloud_private_subnet_az": "eu-west-1a",
        "katapult_cloud_private_subnet_id": "subnet-df23b966",
        "katapult_cloud_public_subnet_az": "eu-west-1b",
        "katapult_cloud_public_subnet_id": "subnet-3ac55975f",
        "katapult_cloud_vpc_default_security_group_id": "sg-acfd436",
        "katapult_cloud_vpc_id": "vpc-21faace"

TASK [set vpc id] ***********************************************************************************************************************************************************
task path: ~/ansible/terraform_output.yml:11
ok: [localhost] => {
    "ansible_facts": {
        "vpc_id": "vpc-21faace"
    "changed": false,
    "failed": false
META: ran handlers
META: ran handlers

PLAY RECAP ******************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0
Katapult Cloud
No Comments

Post a Comment