google_compute_security_policy rules are always recreated

[RuBiCK]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

❯ terraform -v
Terraform v1.5.4
on darwin_arm64
[...]
+ provider registry.terraform.io/hashicorp/google v5.10.0
+ provider registry.terraform.io/hashicorp/google-beta v5.10.0
[...]

The same versions were also tested with amd64

Affected Resource(s)

• google_compute_security_policy

Using resource through module GoogleCloudPlatform/cloud-armor/google version 2.0.1

Expected Behavior

Terraform plan should be idempotent after terraform apply

Actual Behavior

Terraform always wants to recreate all rules in the security policy even if no changes are made and after applying and checking that all changes have been made.

Example of one rule:

  # module.<mymodule>.module.security_policy[0].google_compute_security_policy.policy will be updated in-place
  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/<project-id>/global/securityPolicies/<security-policy-id>"
        name        = "<security-policy-name>"
        # (5 unchanged attributes hidden)

      - rule {
          - action      = "throttle" -> null
          - description = "Throttle per api rule for <redacted>" -> null
          - preview     = false -> null
          - priority    = 80030 -> null

          - match {
              - expr {
                  - expression = <<-EOT
                        has(request.headers['Host'])
                                && (
                                  request.headers['Host'].contains('<redacted>')
                                ) && 
                                  request.path.matches('^/.*$')
                    EOT -> null
                }
            }
           [...]
      + rule {
          + action      = "throttle"
          + description = "Throttle per api rule for <redacted>"
          + preview     = false
          + priority    = 80030

          + match {
              + expr {
                  + expression = <<-EOT
                        has(request.headers['Host'])
                                && (
                                  request.headers['Host'].contains('<redacted>')
                                ) && 
                                  request.path.matches('^/.*$')
                    EOT
                }
            }

          + rate_limit_options {
              + conform_action = "allow"
              + enforce_on_key = "XFF_IP"
              + exceed_action  = "deny(429)"

              + rate_limit_threshold {
                  + count        = 500
                  + interval_sec = 60
                }
            }
        }

References

#9084
The same issue as described and related issues

b/318850546

[mt185252]

Facing similar issue with preconfigured_waf_config block being present in new rules being added via the GCP console.

Seems like its happening only with newly created rules. I have a security policy which was created via the GCP console & has a total of 22 rules(again created via the GCP console), one of these rules was added relatively recently when compared to remaining 21.

I am working on terraform import to construct the state of this security policy and I see that after import for this one particular rule, terraform always tries to delete the preconfigured_waf_config block & as a result re-creates that rule. All the other 21 are just fine.

If I run terraform apply and then immediately run terraform plan I see that it again tries to delete the preconfigured_waf_config block.

To confirm my suspicion that this is been happening only with relatively new rules, I performed the below on the mentioned order:

1. Created a new security policy and added 3 rules to it via the GCP console.
2. Imported the state using terraform import.
3. Ran terraform plan (this showed that the preconfigured_waf_config block in the existing rule has to be deleted & hence the rule will be deleted and recreated.)
4. Ran terraform apply(It was successful)
5. Immediately ran terraform plan again & had same observations as in step 3.

$ terraform plan -out test -var-file ./vars/dev.tfvars
module.cloud-armor["test"].google_compute_security_policy.policy: Refreshing state... [id=projects/<project_id>/global/securityPolicies/test]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.cloud-armor["test"].google_compute_security_policy.policy will be updated in-place
  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/<project_id>/global/securityPolicies/test"
        name        = "test"
        # (5 unchanged attributes hidden)

      - adaptive_protection_config {
          - layer_7_ddos_defense_config {
              - enable = false -> null
            }
        }

      - rule {
          - action      = "allow" -> null
          - description = "test1" -> null
          - preview     = false -> null
          - priority    = 10 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "1.1.1.1",
                      - "2.2.2.2",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      - rule {
          - action      = "allow" -> null
          - description = "test2" -> null
          - preview     = false -> null
          - priority    = 20 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "3.3.3.3",
                      - "4.4.4.4",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      - rule {
          - action      = "deny" -> null
          - description = "Default rule, higher priority overrides it" -> null
          - preview     = false -> null
          - priority    = 2147483647 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "*",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      - rule {
          - action      = "deny(502)" -> null
          - description = "test3" -> null
          - preview     = false -> null
          - priority    = 30 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "5.5.5.5",
                      - "6.6.6.6",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      + rule {
          + action      = "allow"
          + description = "test1"
          + preview     = false
          + priority    = 10

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "1.1.1.1",
                      + "2.2.2.2",
                    ]
                }
            }
        }
      + rule {
          + action      = "allow"
          + description = "test2"
          + preview     = false
          + priority    = 20

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "3.3.3.3",
                      + "4.4.4.4",
                    ]
                }
            }
        }
      + rule {
          + action      = "deny"
          + description = "Default rule, higher priority overrides it"
          + preview     = (known after apply)
          + priority    = 2147483647

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "*",
                    ]
                }
            }
        }
      + rule {
          + action      = "deny(502)"
          + description = "test3"
          + preview     = false
          + priority    = 30

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "5.5.5.5",
                      + "6.6.6.6",
                    ]
                }
            }
        }

        # (1 unchanged block hidden)
    }

Tried a couple of more times and repeatedly saw this issue occurring.

When I create the policy & rules completely using terraform then I dont see this happening(Using module version v1.1.0).

[scramblydevops]

+1

[aaronclong]

There might be some slight overlap here #17288

[mersive-raypitmon]

To prevent my rules from being recreated each time, I had to specify these values in each rule that contained a match.expr (I just have two rules):

    preview  = false
    preconfigured_waf_config {}

I also have a default rule with a match.versioned_expr that didn't need these set, as terraform doesn't think that rule needs to be replaced.)

Here are my versions:

$ terraform -version
Terraform v1.5.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v5.14.0
+ provider registry.terraform.io/hashicorp/google-beta v5.14.0
+ provider registry.terraform.io/hashicorp/random v3.6.0

(I am using the google-beta provider for my google_compute_security_policy.)

[rojomisin]

This bug makes managing cloud armor rules in terraform unusable, ie dangerous in production

[sarnepalli]

To prevent my rules from being recreated each time, I had to specify these values in each rule that contained a match.expr (I just have two rules):

    preview  = false
    preconfigured_waf_config {}

This didn't make any difference for me, I still see the rules being recreated.

[bradam12]

I had a single rule that was constantly changing with preconfigured_waf_config. I changed the priority of that rule in the UI manually, then reapplied the TF. No more reoccurring changes after that.

Edit: that didn't last long. Came back within a couple days.

[snek-git]

Facing the same problem.

  # module.security_policy.google_compute_security_policy.policy will be updated in-place
  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/divdot-production/global/securityPolicies/frontend-host-policy"
        name        = "frontend-host-policy"
        # (4 unchanged attributes hidden)

      - rule {
          - action      = "allow" -> null
          - description = "Allow Requests only from firebase frontend" -> null
          - preview     = false -> null
          - priority    = 1000 -> null

          - match {
              - expr {
                  - expression = <<-EOT
                        request.headers['host'].lower().contains('example.com')
                    EOT -> null
                }
            }
        }
      + rule {
          + action      = "allow"
          + description = "Allow Requests only from firebase frontend"
          + preview     = false
          + priority    = 1000

          + match {
              + expr {
                  + expression = <<-EOT
                        request.headers['host'].lower().contains('example.com')
                    EOT
                }
            }
        }

        # (3 unchanged blocks hidden)
    }

These rules were initially created using terraform and rerunning terraform apply recreates them every time.

[haimke111]

Hi guys,
I'm also facing the same issue, Any news regarding this?
For me it seems that the cause is in the preconfigured_waf_config for the rules and in layer_7_ddos_defense_config

  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/agp-exchange-prod-0u/global/securityPolicies/grafana-office"
        name        = "grafana-office"
        # (5 unchanged attributes hidden)

      - adaptive_protection_config {
          - layer_7_ddos_defense_config {
              - enable          = false -> null
                # (1 unchanged attribute hidden)
            }
        }

      - rule {
          - action      = "allow" -> null
          - description = "Allow only office CIDR" -> null
          - preview     = false -> null
          - priority    = 0 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "x.x.x.x/29",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      - rule {
          - action      = "deny(403)" -> null
          - description = "Default rule, higher priority overrides it" -> null
          - preview     = false -> null
          - priority    = 2147483647 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "*",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      + rule {
          + action      = "allow"
          + description = "Allow only office CIDR"
          + preview     = false
          + priority    = 0

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "x.x.x.x/29",
                    ]
                }
            }
        }
      + rule {
          + action      = "deny(403)"
          + description = "Default rule, higher priority overrides it"
          + preview     = (known after apply)
          + priority    = 2147483647

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "*",
                    ]
                }
            }
        }

        # (1 unchanged block hidden)
    }

[imrannayer]

This issue is blocking GoogleCloudPlatform/terraform-google-cloud-armor#94

[GangGreenTemperTatum]

also experience the exact same issue ➕ 😢

    terraform_version: 1.8.5
..

- Installed hashicorp/google v5.38.0 (signed by HashiCorp)
- Installing hashicorp/google-beta v5.38.0...
- Installed hashicorp/google-beta v5.38.0 (signed by HashiCorp)

[aebrahim]

I have the same issue, I believe this is because the API to fetch the security policy returns the rules in an unsorted order, so I believe this method should fix it

[Kaschman]

Suggested workaround while this is broken:

Add a lifecycle ignore changes block to google_compute_security_policy:
lifecycle { ignore_changes = [rule] }

Define your rules using the google_compute_security_policy_rule resource.

Your rules will be defined by a different resource, and you will ignore the policy's desire to delete them

[tonyay163]

I would use google_compute_security_policy_rule but it seems to be missing redirect_options for some reason

[spinningarrow]

FYI we're using google_compute_security_policy_rule but ran into this problem anyway. Terraform detected a 'change' in the google_compute_security_policy and deleted all the rules that were created using google_compute_security_policy_rules and then did not recreate them.
In effect, all our traffic resulted in 403s because all the rules were inexplicably removed.

[maxi-cit]

Hello there, I guess last PR fixes this issue, if that is so I guess we can close this.

[imrannayer]

@maxi-cit It seems like it closed the issue with pre-configured_waf but issue with max. expr_options. recaptcha_options is still there. There is another issue open for both of the issues.
#18596

It is resulting in recreation of the rules.

      - rule {
          - action      = "deny(403)" -> null
          - description = "Node JS attack" -> null
          - preview     = true -> null
          - priority    = 10 -> null

          - match {
              - expr {
                  - expression = "evaluatePreconfiguredWaf('nodejs-v33-stable', {'sensitivity': 4})" -> null
                }
              - expr_options {
                  - recaptcha_options {
                      - action_token_site_keys  = [] -> null
                      - session_token_site_keys = [] -> null
                    }
                }
            }
        }

[matheusaleixo-cit]

Hello everyone, I'm currently working on a fix for the issue with the recaptcha_options, I just need some time to manually test the changes.

One way I found to reproduce this permadiff state is to create the security policy and its rules via terraform, then manually set these empty recaptcha_options.action_token_site_keys and recaptcha_options.session_token_site_keys values using the patch endpoint (patchRule endpoint seems to ignore these fields when setting empty values).

[jrtaylorJH]

Seeing this issue on our end too!

[imrannayer]

Issue was resolved in [6.14].

compute: fixed permadiff on the recaptcha_options field for google_compute_security_policy resource #20617(https://github.com/hashicorp/terraform-provider-google/pull/20617)

Can you plz retry after upgrading provider?

[kameshraj]

I still see preconfigured_waf being recreated in 6.18

[imrannayer]

@kameshraj can u plz add your notes in
#18596

[koljaa]

+1

[kameshraj]

@kameshraj can u plz add your notes in #18596

@imrannayer I have updated my notes already here
#18596 (comment)