Using pre-commit framework to keep your terraform in check

Introduction

In this post I’ll give a high level overview of the pre-commit framework and how to get up and running with it. I’ll cover the traditional use case of local development, and how to also use it in a pipeline to re-use the same config and framework to enforce standards.

Finally I’ll run through some scenario’s using an example repo I’ve created.

What is the pre-commit framework?

Pre-commit is a framework for defining, distributing, installing, and running “hooks” these “hooks” can check for, or enforce conditions. An example would be to run terraform validate to ensure code syntax is correct, or run terraform fmt to format code where needed.

The framework allows you to use these pre-made hooks, and in your own repo(s) define which ones you want to use. Contributors of your repo(s) can then install them as a “git hook” on their device so they can get a quicker feedback loop on any potential issues with code in their commits.

So what are “git hooks” and how do they relate to the pre-commit framework? Git hooks are a functionality provided by git, in the basic sense they allow you to call a script or program during a git event like a commit event. This is where the pre-commit framework comes in. The intended use case is for every commit you do locally, the pre-commit framework will run any hooks you have defined.

When committing changes with these hooks installed one of the following will happen :

• Pre-commit framework runs and finds no issues. In this case the commit happens as usual.
• Pre-commit framework runs and finds issues like syntax validation issue. In this case the commit fails and git will throw an error detailing which hook has failed and why. Once the underlying issue is solved you can attempt to commit again.
• Pre-commit framework runs and finds an issue but corrects it, an example would be its formatted whitespace using terraform fmt. In this case the commit fails and there will be un-staged changes for the corrections that you can review, and will need to stage before committing again.

How to get up and running locally

As pre-requisites, you will need git, and python installed.

Install pre-commit

Install is straight forward with the pip command. If you are having issues running pip, ensure Python is in your path environment variable.

pip install pre-commit
Using cached pre_commit-3.6.2-py2.py3-none-any.whl (204 kB)
Installing collected packages: pre-commit
Successfully installed pre-commit-3.6.2

WSL considerations

If you are on Windows and have WSL installed, you may run in to issues when using these hooks. Some these hooks are based on bash scripts, and WSL bash will cause issues as it doesn’t understands windows based paths.

To work around this issue, in your system path environment variable, ensure the first path is the path to directory where git bash exists (usually c:\Program Files\git\bin\).

Using pre-commit

Defining which hooks to use in a repo

To define which hooks to use in one of your repo’s just create a file at the root of the repo called .pre-commit-config.yaml and list the repo’s and which hooks to use, along with any optional arguments / config. An example is below:

repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: "v1.88.0"
    hooks:
      - id: terraform_validate
      - id: terraform_fmt
      - id: terraform_trivy
  - repo: https://github.com/terraform-docs/terraform-docs
    rev: "v0.17.0"
    hooks:
      - id: terraform-docs-go
        args: ["--lockfile=false", "markdown", "table", "--output-file", "README.md", "./"]

A great resource for terraform specific hooks is https://github.com/antonbabenko/pre-commit-terraform
The pre-commit website also has a massive list of all types: https://pre-commit.com/hooks.html

Running hooks

When in a repo that has a .pre-commit-config.yaml file you can run pre-commit run -a to have all hooks ran against all files. Something to keep in mind is that hooks can have dependencies. E.g. the hooks for terraform validate, terraform fmt etc, have a dependency on you having terraform installed.

Repo’s that publish the hooks will usually define what the dependencies are. When making use of hooks in one of your repo’s it is a good idea to include in the CONRIBUTING.md how the hooks are used, and what dependencies the hooks have.

I have an example repo you can clone to try this out – jamesw4/tf-precommit-example (github.com). In its current state it passes all hooks. The hooks have a dependency on you having terraform and trivy installed.

Example run:

pwsh> pre-commit run -a
Terraform validate.......................................................Passed
Terraform fmt............................................................Passed
Terraform validate with trivy............................................Passed
terraform-docs...........................................................Passed

Lets try and stage some issues and see what happens. In main.tf I will mess up the white space like below.

resource "random_pet" "pet" {
  count  = var.pet_config.count
  length    = var.pet_config.length
  separator = var.pet_config.separator
}

When I rerun pre-commit I now get the following:

pwsh> pre-commit run -a
Terraform validate.......................................................Passed
Terraform fmt............................................................Failed
- hook id: terraform_fmt
- files were modified by this hook

main.tf

Terraform validate with trivy............................................Passed
terraform-docs...........................................................Passed

When checking the file again I can see the whitespace has been corrected by the hook.

resource "random_pet" "pet" {
  count     = var.pet_config.count
  length    = var.pet_config.length
  separator = var.pet_config.separator
}

Lets see what we get for a syntax error, lets pass in an invalid parameter for the random_pet resource, it doesnt support a parameter called colour:

resource "random_pet" "pet" {
  count     = var.pet_config.count
  length    = var.pet_config.length
  separator = var.pet_config.separator
  colour    = "red"
}

And now when running pre-commit I get:

pwsh> pre-commit run -a
Terraform validate.......................................................Failed
- hook id: terraform_validate
- exit code: 1

Command 'terraform init' successfully done: .
Validation failed: .
╷
│ Error: Unsupported argument
│
│   on main.tf line 5, in resource "random_pet" "pet":
│    5:   colour    = "red"
│
│ An argument named "colour" is not expected here.
╵

Terraform fmt............................................................Passed
Terraform validate with trivy............................................Passed
terraform-docs...........................................................Passed

Some of these hooks are really powerful. There are hooks to check for things like private keys, do static code analysis, and terraform-docs can keep your README’s up to date on things like inputs, outputs, dependencies etc.

Installing hooks as a git hook

So far we have been manually triggering the pre-commit command, which in day to day use isn’t ideal. You can use the command pre-commit install to install a githook in your local copy of that repo. This command needs to be ran for each local repo you have, that contains githooks, otherwise the hooks wont trigger on commit for that particular repo (you could still run pre-commit run -a manually in them though.

pwsh> pre-commit install
pre-commit installed at .git\hooks\pre-commit

Now whenever you attempt a commit to this repo, pre-commit will automatically run on changed files. No file have been checked in the example as no files have changed.

pwsh> git commit -m "test"
Terraform validate...................................(no files to check)Skipped
Terraform fmt........................................(no files to check)Skipped
Terraform validate with trivy........................(no files to check)Skipped
terraform-docs.......................................(no files to check)Skipped
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

VSCode Experience

I’ll go ahead and stage another failure so we can see what the experience is like in VSCode since this is popular IDE people use for Terraform.

I’ll go add a new Azure Key Vault resource to my main.tf – I already know the hook for terraform-docs is going to want to update the readme to include this new resource, and I know the trivy hook won’t be happy about some missing best practices.

resource "random_pet" "pet" {
  count     = var.pet_config.count
  length    = var.pet_config.length
  separator = var.pet_config.separator
}

resource "azurerm_key_vault" "bad_example" {
  name                = "bad-example"
  resource_group_name = "test"
  sku_name            = "standard"
  location            = "uksouth"
  tenant_id           = "fbb13389-e951-44f2-9957-12075dd99086"
}

In my attempt at committing I get an error, Its clearly an error from the icon, but the text is just showing me the hook for terraform validate passed? This message most of the time is useless as you only get the first line of output from pre-commit.

All is not lost, hit the button for Show Command Output to get the full output from pre-commit, the following is what I got back:

Terraform validate.......................................................Passed
Terraform fmt............................................................Passed
Terraform validate with trivy............................................Failed
- hook id: terraform_trivy
- exit code: 1

2024-02-29T19:58:54.693Z	INFO	Misconfiguration scanning is enabled
2024-02-29T19:58:56.142Z	INFO	Detected config files: 2

main.tf (terraform)
===================
Tests: 2 (SUCCESSES: 0, FAILURES: 2, EXCEPTIONS: 0)
Failures: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 1)

CRITICAL: Vault network ACL does not block access by default.
════════════════════════════════════════
Network ACLs allow you to reduce your exposure to risk by limiting what can access your key vault. 

The default action of the Network ACL should be set to deny for when IPs are not matched. Azure services can be allowed to bypass.

See https://avd.aquasec.com/misconfig/avd-azu-0013
────────────────────────────────────────
 main.tf:7-13
────────────────────────────────────────
   7 ┌ resource "azurerm_key_vault" "name" {
   8 │   name                = "bad-example"
   9 │   resource_group_name = "test"
  10 │   sku_name            = "standard"
  11 │   location            = "uksouth"
  12 │   tenant_id           = "fbb13389-e951-44f2-9957-12075dd99086"
  13 └ }
────────────────────────────────────────


MEDIUM: Vault does not have purge protection enabled.
════════════════════════════════════════
Purge protection is an optional Key Vault behavior and is not enabled by default.

Purge protection can only be enabled once soft-delete is enabled. It can be turned on via CLI or PowerShell.

See https://avd.aquasec.com/misconfig/avd-azu-0016
────────────────────────────────────────
 main.tf:7-13
────────────────────────────────────────
   7 ┌ resource "azurerm_key_vault" "name" {
   8 │   name                = "bad-example"
   9 │   resource_group_name = "test"
  10 │   sku_name            = "standard"
  11 │   location            = "uksouth"
  12 │   tenant_id           = "fbb13389-e951-44f2-9957-12075dd99086"
  13 └ }
────────────────────────────────────────

terraform-docs...........................................................Failed
- hook id: terraform-docs-go
- files were modified by this hook

README.md updated successfully

As expected, trivy has flagged one critical issue, one medium issue, and we can also see that terraform-docs has updated the README to include this new resource. From here I would either need to follow the advice from trivy so the code is compliant, or add the required comments so trivy knows to ignore these occurrences.

VSCode Extension

It’s worth giving a shoutout to this VSCode extension – pre-commit – Visual Studio Marketplace

This adds commands to the command pallet to run pre-commit on all or the current file, and also has an option to run pre-commit on saving of a file. The hooks get executed as VSCode tasks so are visible in the terminal pane. Additionally its smart enough to run hooks that don’t modify files in parallel which will speed up things if you have multiple hooks.

How to use pre-commit in a pipeline to enforce standards

Leveraging git hooks for pre-commit is a very much opt in scenario. The person contributing needs the framework installed, any dependencies for the hooks you are using installed, and will need to remember to have ran pre-commit install in the local copy of their repo.

Even then, its possible to use the switch --no-verify when committing e.g. git commit --no-verify -m "commit message" this commits without running any hooks. While this might sound counter intuitive, there are genuine use cases. Consider the scenario you have someone with something half written that wont pass a terraform validate that your hook runs, but they are about to go on leave for a week, do you really want to point-blank block them from committing and pushing?

In short, a pre-commit git hook is not the stage to try and enforce or police things, it is there to make contributors lives easier if they wish to install the hooks to get any feedback on expected standards etc earlier. To enforce the same standards and checks, the checks should also be done in a CI pipeline so validation can be enforced during a PR.

If you don’t already have CI pipelines to run all of the checks, you can re-use what you have defined and are using with pre-commit. I’ve included a few example pipelines below. The basic idea is, the pipeline just runs pre-commit on all files and shows on diff for a failure.

Extending beyond the example, you need to consider hook dependencies. In my example the only external dependency I have is trivy. When adding hooks check if there are dependencies you may need to add additional install steps for. Below are some links to show pre-installed software per image for Azure DevOps agents and GitHub runners, so you can decide if you need to install anything beyond that to cater for hook dependencies.

Closing thoughts

The pre-commit framework and git hooks are a great way to offer quicker feedback on desired code standards, for those that want to make use of them. While the initial setup for a contributor of getting the framework up and running locally and sorting any dependencies can seem like a chore, in my opinion it is a worthwhile exercise.

I say the above from the perspective of a contributor as well, once I got the framework installed, and all dependencies that common repo’s I contribute towards installed, its a breeze.

With that said, for enforcing the same sort of standards and validation, stick to a pipeline triggered by PR.