Terraform not allowing to pass nested values (like assume_role.role_arn) to the backend.s3 configuration using cli (-backend-config="KEY=VALUE") and config file (-backend-config=PATH)

[Humeid-Ussene-Jocordasse]

Terraform Version

1.11.4

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.33.0"
    }
  }
  
  backend "s3" {
    bucket = ""
    use_lockfile=""
    key=""
    region=""
    assume_role = {
      role_arn = ""
      external_id = ""
    }
  }
    required_version = ">= 0.14.9"
}
...terraform config...

Debug Output

not applicable

Expected Behavior

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Actual Behavior

Initializing the backend...
╷
│ Error: Invalid backend configuration argument
│ 
│ The backend configuration argument "assume_role.role_arn" given on the command line is not expected for the selected backend type.
╵
╷
│ Error: Invalid backend configuration argument
│ 
│ The backend configuration argument "assume_role.external_id" given on the command line is not expected for the selected backend type.

Steps to Reproduce

terraform -chdir=.iac init  \
      -backend=true \
      -backend-config="bucket=my-humble-bucket" \
      -backend-config="key=my-humble-key.tfstate" \
      -backend-config="region=af-south-1" \
      -backend-config="use_lockfile=true" \
      -backend-config="assume_role.role_arn=arn:aws:iam::123456789012:role/my-humble-role" \
      -backend-config="assume_role.external_id=my-humble-id" \
      -force-copy

Additional Context

I'm having the same issue when i true to use those exact same values on backend.config file, trying to pass with the -backend-config=PATH flag...

I noticed this issue after upgrading the Terraform version in our team's Continuous Integration tool from v1.8.5 to v1.11.4. Previously, we were able to provide the role_arn programmatically because it was a top-level attribute in the backend.s3 block. Now that it has to be nested under assume_role, Terraform throws an error when we try to pass it dynamically.

Edit:
I've found out this still works when using a .hcl file, passing a value like this one:

bucket = "my-humble-bucket"
key="test/my-humble-key.tfstate"
region="af-south-1"
use_lockfile="true"
assume_role= {
    role_arn = "arn:aws:iam::123456789012:role/my-humble-role"
    external_id="my-humble-id"
}

References

No response

Generative AI / LLM assisted development?

No response

[crw]

Thanks for this report!

[apparentlymart]

In case it's somehow useful, a long time ago I wrote github.com/apparentlymart/go-hcl-overlay with the goal of supporting stuff like this where a system needs to merge some command line options in with existing HCL body content.

Even if that library is not directly useful as an import, I remember there being some interesting design tradeoffs in making that work which might give some ideas on similar tradeoffs to make in solving this for Terraform. In particular, I experimented with a few different ways to deal with a situation where e.g. the CLI says assume_role.role_arn but yet there isn't already an assume_role block in the configuration for that to override into, and picked one that seemed to work relatively well for reasonable cases like this.

The example I included in the repository uses the very opinionated ExtractCLIOptions whose opinions don't match how -backend-config already works, but I think ParseCLIArgument is closer to how Terraform wants to treat -backend-config where the key=value is given as the value of an option rather than as the option itself.

Offered just in the hope that it's helpful!