Improve documentation for terraform test

[novekm]

Terraform Version

Terraform v1.10.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.94.0

Your version of Terraform is out of date! The latest version
is 1.11.3. You can update by downloading from https://www.terraform.io/downloads.html

Affected Pages

https://developer.hashicorp.com/terraform/cli/commands/test#terraform-test-command

What is the docs issue?

The current documentation for terraform test is not explicit enough with the example usage, especially for the optional parameters such as -filter. The docs mention:

-filter=testfile - Limits the terraform test operation to the specified test files.

However the above doesn't work. With a set of tests within a recommended /tests directory with filenames such as 01_test.tftest.hcl, 02_test.tftest.hcl etc, running terraform test -filter=02_test.tftest.hcl or even terraform test -filter="02_test.tftest.hcl" does not work:

❯ terraform test -filter=02_test.tftest.hcl

Success! 0 passed, 0 failed.

❯ terraform test -filter="02_test.tftest.hcl"

Success! 0 passed, 0 failed.

Instead, you must write the filter as a string, and include the full path to the file, including the /tests directory.

❯ terraform test -filter="tests/02_test.tftest.hcl"
tests/02_test.tftest.hcl... in progress
  run "unit_test"... pass
  run "e2e_test"... pass
tests/02_test.tftest.hcl... tearing down
tests/02_test.tftest.hcl... pass

I could understand this if using a non-standard directory name for your tests, however this is not expected if using the default recommended naming of /tests. One would assume that if terraform test automatically uses test files (*.tftest.hcl) that are contained in a directory with this name, then the simple filter should function the same way.

Proposal

Please update the docs with more examples and explicitly mention the way you must use filters to reference specific test files, and call out that it's required to directly reference the directory name, even if it is the default value. In the longterm, perhaps it would be possible to omit having to pass in the directory name in the filter if using the default naming of /tests 🙂

References

terraform test documentation

[liamcervante]

Hi @novekm, thanks for filing this. We do have examples of using the -filter option within Test Directory section. The docs team might take a look at your feedback and iterate on things anyway, but I did want to highlight that.

One thing to consider is that you can also always write tests directly within the root configuration directory. Terraform will pick up tests both within the tests directory and the root directory. This is why the full path to the test file is required within the -filter option, so that is unlikely to change in future iterations.

[novekm]

Thanks Liam, that makes sense. Related to some of the other flags for Terraform test - are there any examples for targeting speficic directories for tests beyond the default /tests directory, and any limitations on naming/nesting?

I've been trying terraform test -test-directory=<my-desired-folder> but for some reason when using this I can't successful run terraform init. Rather, I run into a cycle where I receive a success message about the initialization, but then with a subsequent run of terraform test am prompted to initlize again.

Additional challenges arose when trying to have a folder named modules within a /tests directory. This was done to allow for a single tests directory and test a variety of modules, as well as larger solutions that use a collection of modules. This was the folder structue:

─ cloud-game-development-toolkit/
    ├── modules
    └── tests/
        ├── modules/
        │   ├── jenkins/
        │   │   ├── 01_mandatory_test.tftest.hcl
        │   │   └── 02_custom_test.tftest.hcl
        │   ├── perforce/
        │   │   ├── 01_mandatory_test.tftest.hcl
        │   │   └── 02_custom_test.tftest.hcl
        │   └── unreal/
        │       └── horde/
        │           ├── 01_mandatory_test.tftest.hcl
        │           └── 02_custom_test.tftest.hcl
        └── solutions/
            ├── simple-build-pipeline/
            │   ├── 01_mandatory_test.tftest.hcl
            │   └── 02_custom_test.tftest.hcl
            └── audio-pipeline/
                ├── 01_mandatory_test.tftest.hcl
                └── 02_custom_test.tftest.hcl

The goal was to be able to group these tests and with a GitHub action (comment on PR), trigger tests within specific sub-directories for the modules with something like "/test-perforce" (during updates to the module) or "/test-all-modules" (when cutting a new version release for the entire project). Running terraform test and trying to target specific directories when having the name of modules within the test directory threw this error:

This persisted even after renaming the sub directory and re-initializing Terraform. I ended up having to delete the entire .Terraform directory and the lock file, restart my IDE, and re-run Terraform init to resolve. I'm assuming modules is a reserved name and made Terraform expect to see a module configuration instead of tests for the modules.

[liamcervante]

One thing to bear in mind is you need to specify the test directory when initialising as well: terraform init -test-directory=<whatever>.

The init command can't discover all the required dependencies for tests that aren't in the default locations without that extra hint.

There should be no restrictions on naming or structure other than the test directory has to be "beneath" the configuration directory. It's also worth noting that Terraform can't load tests from multiple directories at a time and it doesn't load directories recursively.

[novekm]

Hi @liamcervante, unfortunately I am still having some issues trying to use the -test-directory flag. This is what my folder structure looks like (condensed to relevant directories/files):

cloud-game-development-toolkit
├── modules
│   ├── README.md
│   ├── jenkins
│   │   ├── README.md
│   │   ├── alb.tf
│   │   ├── asg.tf
│   │   ├── data.tf
│   │   ├── ecs.tf
│   │   ├── efs.tf
│   │   ├── examples
│   │   │   └── complete
│   │   ├── fsxz.tf
│   │   ├── iam.tf
│   │   ├── local.tf
│   │   ├── outputs.tf
│   │   ├── s3.tf
│   │   ├── sg.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── perforce
│   │   ├── examples
│   │   │   └── complete
│   │   ├── helix-authentication-service
│   │   │   ├── README.md
│   │   │   ├── alb.tf
│   │   │   ├── data.tf
│   │   │   ├── iam.tf
│   │   │   ├── local.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   ├── helix-core
│   │   │   ├── README.md
│   │   │   ├── data.tf
│   │   │   ├── iam.tf
│   │   │   ├── locals.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   └── helix-swarm
│   │       ├── README.md
│   │       ├── alb.tf
│   │       ├── data.tf
│   │       ├── elasticache.tf
│   │       ├── iam.tf
│   │       ├── locals.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── sg.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── s3
│   │   ├── examples
│   │   │   ├── advanced
│   │   │   └── basic
│   │   ├── main.tf
│   │   ├── tests
│   │   │   ├── 01_mandatory_basic.tftest.hcl
│   │   │   └── 02_mandatory_advanced.tftest.hcl
│   │   └── variables.tf
│   ├── teamcity
│   │   ├── README.md
│   │   ├── examples
│   │   │   └── simple
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   └── unreal
│       ├── horde
│       │   ├── README.md
│       │   ├── alb.tf
│       │   ├── asg.tf
│       │   ├── config
│       │   ├── docdb.tf
│       │   ├── ecs.tf
│       │   ├── elasticache.tf
│       │   ├── examples
│       │   ├── iam.tf
│       │   ├── local.tf
│       │   ├── outputs.tf
│       │   ├── sg.tf
│       │   ├── variables.tf
│       │   └── versions.tf
│       └── unreal-cloud-ddc
│           ├── unreal-cloud-ddc-infra
│           └── unreal-cloud-ddc-intra-cluster

The s3 module/directory is what I'm currently trying to test:

├── examples
│   ├── advanced
│   │   └── main.tf
│   └── basic
│       └── main.tf
├── main.tf
├── tests
│   ├── 01_mandatory_basic.tftest.hcl
│   └── 02_mandatory_advanced.tftest.hcl
└── variables.tf

The s3 module and related tests are very basic currently, just to try to test the correct usage of the flags with terraform test. All the module does is create an S3 bucket with a supplied variable for the bucket prefix.

basic example (main.tf)

module "s3_bucket" {
  source = "../../"

  bucket_prefix = "kevon-test-basic"
}

For the tests, they simply reference the examples in the examples directories and use that for 2 run blocks:

01_mandatory_basic.tftest.hcl

run "unit_test" {
  command = plan
  module {
    source = "./examples/basic"
  }
}

run "e2e_test" {
  command = apply
  module {
    source = "./examples/basic"
  }
}

02_mandatory_advanced.tftest.hcl

run "unit_test" {
  command = plan
  module {
    source = "./examples/advanced"
  }
}

run "e2e_test" {
  command = apply
  module {
    source = "./examples/advanced"
  }
}

If I understand what you're saying, to run a test against that specific /tests directory, I need to do the following (run from the root of the project directory (cloud-game-development-toolkit):

1. Initialize Terraform with context to the module you want to test

terraform init -test-directory=modules/s3/tests

2. Run all terraform tests that are present in the target directory

terraform test -test-directory=modules/s3/tests

However, when I try to run the first command, I get a response that the Terraform was initialized in an empty directory:

❯ terraform init -test-directory=modules/s3/tests
Terraform initialized in an empty directory!

To get this to work, I instead had to use the -chdir flag in terraform and do the following:

terraform -chdir=modules/s3 init

Initializing the backend...
Initializing modules...
- test.tests.02_mandatory_advanced.unit_test in examples/advanced
- test.tests.02_mandatory_advanced.unit_test.s3_bucket in .
- test.tests.02_mandatory_advanced.e2e_test in examples/advanced
- test.tests.02_mandatory_advanced.e2e_test.s3_bucket in .
- test.tests.01_mandatory_basic.unit_test in examples/basic
- test.tests.01_mandatory_basic.unit_test.s3_bucket in .
- test.tests.01_mandatory_basic.e2e_test in examples/basic
- test.tests.01_mandatory_basic.e2e_test.s3_bucket in .
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.94.1...
- Installed hashicorp/aws v5.94.1 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Then I didn't even have to use the -test-directory flag since the s3 module is using the default /tests naming. I just had to run:

terraform -chdir=modules/s3 test

tests/01_mandatory_basic.tftest.hcl... in progress
  run "unit_test"... pass
  run "e2e_test"... pass
tests/01_mandatory_basic.tftest.hcl... tearing down
tests/01_mandatory_basic.tftest.hcl... pass
tests/02_mandatory_advanced.tftest.hcl... in progress
  run "unit_test"... pass
  run "e2e_test"... pass
tests/02_mandatory_advanced.tftest.hcl... tearing down
tests/02_mandatory_advanced.tftest.hcl... pass

Success! 4 passed, 0 failed.

I'm pretty confused on which should be used since I was never able to get the -test-directory flag to work.

[liamcervante]

Hi @novekm, sorry for the delayed response - I was out of office last week.

This behaviour is expected. You should run terraform test from the directory of the configuration you want to test. So, in this case, you want test the configuration in modules/s3, hence why the -chdir flag is necessary. You could also run a normal cd command into the correct config directory, and then run terraform test without the -chdir command.

Once you are executing from the required configuration directory, terraform will automatically detect the tests directory as the test directory unless it is overridden.

This command: terraform init -test-directory=modules/s3/tests, assumes you have configuration in your current directory and looks in the specified directory for tests against the current directory. It doesn't know that the configuration you want to test is actually within modules/s3. You achieve this with the -chdir flag as you discovered.

Hopefully that makes more sense? Thanks for your feedback here though, it is valuable for us to know which parts are confusing for users so we can update the docs to clear things up.

[novekm]

Thanks @liamcervante! No problem at all, hope you had a nice time off! Ahh thanks for the clarification. In terms of the -chdir / cd thing, this was only because the project I'm working on currently has multiple submodules so I was trying to ensure with GitHub Actions I could easily target the specific directories to run tests just for the individual modules desired. However I think I may be able to set the working-directory on the GitHub actions side to reference the specific module desired for the specific runs. So essentially, since in the above directory structure I mentioned, there's no actual root main.tf or any other *.tf files at the root level (instead these are just in the submodules and the examples within the submodules` and that's why it wasn't working. That makes sense.

One last question about terraform test in the event that a test fails in a CI/CD pipeline and for some reason terraform test couldn't successfully delete resources (for example, during race conditions where you have to run a subsequent terraform destroy) how should you best clean up resources since state is managed on your behalf during a run terraform test? Is it a best practice to somehow output the results of the test and all logs to a file, that way manual deletion of hanging resources can be done as a worst case scenario? Or is there any retry logic that can be used?

[liamcervante]

The cleanup story is a bit lacking at the moment - we have #33786 which is tracking improvements to this.

The current solution to a failed cleanup operation is that the test command will fail, and Terraform will print out a list of resources that were not tidied up before exiting. This should hopefully be reflected in your CI/CD pipeline as a failed stage, then checking the logs for that stage should have included Terraform's output. Unfortunately, this doesn't include attributes about the resources themselves so it can be quite difficult to work out exactly which resources have been left behind.

We are actively working on improving this though. The branch https://github.com/hashicorp/terraform/tree/f-controlling-destroys contains our in progress solution to this, which includes capturing the state files of any run blocks that weren't successfully tidied and the introduction of a dedicated cleanup command which will allow users to retry the destroy stages directly should things fail. This should be released as part of v1.13 and should greatly improve the failed destroy story.

[novekm]

Thanks Liam! Very helpful context. Looking to v1.13 and this functionality!