Skip to content

Add storage_control_folder_intelligence_config resource. #13394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 179 additions & 0 deletions mmv1/products/storagecontrol/FolderIntelligenceConfig.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
# Copyright 2025 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
# API resource name
name: 'FolderIntelligenceConfig'
kind: 'storagecontrol#intelligenceconfig'
# Resource description for the provider documentation.
description: |
The Folder Storage Intelligence resource represents GCS Storage Intelligence operating on individual GCP Folder. Storage Intelligence is a singleton resource and individual instance exists on each GCP Folder.

Storage Intelligence is for Storage Admins to manage GCP storage assets at scale for performance, cost, security & compliance.

docs:
warning: |
Storage Intelligence Config is a singleton resource which cannot be created or deleted. A single instance of Storage Intelligence Config exist for each GCP Folder. Terraform does not create or destroy this resource.
Terraform resource creation for this resource is simply an update operation on existing resource with specified properties. Terraform deletion won't have any effect on this resource rather it will only remove it from the state file.

# URL for the resource's standard Get method. https://google.aip.dev/131
self_link: 'folders/{{name}}/locations/global/intelligenceConfig'

custom_code:
custom_create: templates/terraform/custom_create/storage_control_folder_intelligence.go.tmpl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is custom create needed? pre_create seems sufficient to just alter the url. There is the question of, if we need this to be all on v2 if this is the right way to go about things. Because we will be creating on v2 and reading on v1 with the current implmentation. Open to call to discuss.

Copy link
Member Author

@kautikdk kautikdk Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ScottSuarez,
URL modification code was written a quite some time ago before we decided to create a new service storagecontrol for GCS JSON v2. I forgot to remove it, My bad! Regarding custom_create, The reason is the update_mask and PATCH method in create operation. Storage Intelligence can not be created so we mapped create operation to PATCH method but the concern is related to permanent(optional+computed) and optional fields. edition_config is a permanent field and back-filled by server side so user can change but it is not the field that we can clear but the filter block is optional and not back filled by server side. Now just to simulate the create operation, we can put full update_mask of every fields which are optional but no need to send empty value for the fields which are computed or has server side value.
Why we want put update_mask for optional fields?
The main reason is we are clearly stating that the create operation is actually update operation under the hood so the first create operation diff should represent whatever will be the end state and should not generate diff after apply. That necessitates requirement of update_mask for optional fields and if we don't do it then some diff can be generated after apply operation. That will be just similar to import operation that we already support.

I have removed that URL modification line. Open to discuss any concerns here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also empty value for edition_config will result in server side error so we can not put update_mask for that field until it is specified by the user in create operation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to @ScottSuarez's point. This should be possible to do with pre_create custom code rather than replacing the whole create method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened hashicorp/terraform-provider-google#22033 to track making this easier.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused what the concern is with O+C fields; in general, sending an empty value for them (triggering server-side default behavior) should be correct. Although it would be possible to persist the values that already exist on the server side that might be more confusing than beneficial.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh yes, checked how exactly pre-create injects code and realized it adds code snippet just before making an API call. Changed custom_create to pre_create and kept previous update_mask logic as it is while we are still discussing this behavior.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused what the concern is with O+C fields; in general, sending an empty value for them (triggering server-side default behavior) should be correct. Although it would be possible to persist the values that already exist on the server side that might be more confusing than beneficial.

So just to give an example in the Storage Intelligence resource, edition_config is a property of the resource and all the resources are having this field backfilled by the server. Users can configure this field, has server side value initially, and can not be removed so it should be Optional+Computed. Now since we can not remove it, we can not send empty value and update_mask together as it results in a InvalidValue error so we have to put update_mask only when user specifies it otherwise we will keep server side value in the create operation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So to make sure I understand right, edition_config behaves as follows:

  • The initial configuration is "empty"
  • The first time that a user calls this API, they can choose to supply a value; if they don't, the API will choose a value on the server side
  • Either way, once the field has a value, it can never be changed.

Is that correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Umm, Not quite like that.

API behavior is,

  • The initial value of edition_config field is INHERIT.
  • User can change the field value but can never remove edition_config property from the resource.

Terraform mapping,

  • Marked as O+C. That will allow users to keep server side value if not specified in the config and they can change it if they want.

Regarding the empty value reference, I believe my wordings made some confusion. There is an API side validation which checks if the update_mask is provided for edition_config. if it is provided and field is not present in the API request body then it will throw an InvalidValue error.
In the create operation, we will put update_mask of edition_config only when it is specified by user in the resource config. Just like whatever is being done here for every field: https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/templates/terraform/update_mask.go.tmpl


# The HTTP verb used to update a resource. Allowed values: :POST, :PUT, :PATCH. Default: :PUT.
update_verb: 'PATCH'
# If true, the resource sets an `updateMask` query parameter listing modified
# fields when updating the resource. If false, it does not.
update_mask: true

exclude_delete: true

import_format:
- 'folders/{{name}}/locations/global/intelligenceConfig'

# If true, code for handling long-running operations is generated along with
# the resource. If false, that code is not generated.
autogen_async: false

properties:
# Fields go here
- name: 'name'
type: String
required: true
immutable: true
url_param_only: true
description: |
Identifier of the GCP Folder. For GCP Folder, this field can be folder number.
- name: 'editionConfig'
type: String
required: false
default_from_api: true
description: |
Edition configuration of the Storage Intelligence resource. Valid values are INHERIT, TRIAL, DISABLED and STANDARD.
- name: 'updateTime'
type: String
output: true
description: |
The time at which the Storage Intelligence Config resource is last updated.
- name: 'filter'
type: NestedObject
diff_suppress_func: 'intelligenceFilterDiffSuppress'
description: |
Filter over location and bucket using include or exclude semantics. Resources that match the include or exclude filter are exclusively included or excluded from the Storage Intelligence plan.
properties:
- name: excludedCloudStorageBuckets
type: NestedObject
required: false
description: |
Buckets to exclude from the Storage Intelligence plan.
conflicts:
- 'filter.0.included_cloud_storage_buckets'
at_least_one_of:
- 'filter.0.included_cloud_storage_buckets'
- 'filter.0.excluded_cloud_storage_buckets'
- 'filter.0.included_cloud_storage_locations'
- 'filter.0.excluded_cloud_storage_locations'
diff_suppress_func: 'intelligenceFilterExcludedCloudStorageBucketsDiffSuppress'
properties:
- name: bucketIdRegexes
required: true
type: Array
send_empty_value: true
item_type:
type: String
description: |
List of bucket id regexes to exclude in the storage intelligence plan.
- name: includedCloudStorageBuckets
type: NestedObject
required: false
description: |
Buckets to include in the Storage Intelligence plan.
conflicts:
- 'filter.0.excluded_cloud_storage_buckets'
at_least_one_of:
- 'filter.0.included_cloud_storage_buckets'
- 'filter.0.excluded_cloud_storage_buckets'
- 'filter.0.included_cloud_storage_locations'
- 'filter.0.excluded_cloud_storage_locations'
diff_suppress_func: 'intelligenceFilterincludedCloudStorageBucketsDiffSuppress'
properties:
- name: bucketIdRegexes
required: true
send_empty_value: true
type: Array
item_type:
type: String
description: |
List of bucket id regexes to exclude in the storage intelligence plan.
- name: excludedCloudStorageLocations
type: NestedObject
required: false
description: |
Locations to exclude from the Storage Intelligence plan.
conflicts:
- 'filter.0.included_cloud_storage_locations'
at_least_one_of:
- 'filter.0.included_cloud_storage_buckets'
- 'filter.0.excluded_cloud_storage_buckets'
- 'filter.0.included_cloud_storage_locations'
- 'filter.0.excluded_cloud_storage_locations'
diff_suppress_func: 'intelligenceFilterExcludedCloudStorageLocationsDiffSuppress'
properties:
- name: locations
type: Array
required: true
send_empty_value: true
description: |
List of locations.
item_type:
type: String
- name: includedCloudStorageLocations
type: NestedObject
required: false
description: |
Locations to include in the Storage Intelligence plan.
conflicts:
- 'filter.0.excluded_cloud_storage_locations'
at_least_one_of:
- 'filter.0.included_cloud_storage_buckets'
- 'filter.0.excluded_cloud_storage_buckets'
- 'filter.0.included_cloud_storage_locations'
- 'filter.0.excluded_cloud_storage_locations'
diff_suppress_func: 'intelligenceFilterincludedCloudStorageLocationsDiffSuppress'
properties:
- name: locations
type: Array
required: true
send_empty_value: true
description: |
List of locations.
item_type:
type: String
- name: 'effectiveIntelligenceConfig'
output: true
description: |
The Intelligence config that is effective for the resource.
type: NestedObject
properties:
- name: intelligenceConfig
type: String
output: true
description: |
The Intelligence config resource that is applied for the target resource.
- name: effectiveEdition
type: String
output: true
description: |
The `StorageIntelligence` edition that is applicable for the resource.
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ properties:
required: false
default_from_api: true
description: |
Edition configuration of the Storage Intelligence resource. Valid values are INHERIT, DISABLED and STANDARD.
Edition configuration of the Storage Intelligence resource. Valid values are INHERIT, TRIAL, DISABLED and STANDARD.
- name: 'updateTime'
type: String
output: true
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@

userAgent, err := tpgresource.GenerateUserAgentString(d, config.UserAgent)
if err != nil {
return err
}

obj := make(map[string]interface{})
editionConfigProp, err := expandStorageControlFolderIntelligenceConfigEditionConfig(d.Get("edition_config"), d, config)
if err != nil {
return err
} else if v, ok := d.GetOkExists("edition_config"); !tpgresource.IsEmptyValue(reflect.ValueOf(editionConfigProp)) && (ok || !reflect.DeepEqual(v, editionConfigProp)) {
obj["editionConfig"] = editionConfigProp
}
filterProp, err := expandStorageControlFolderIntelligenceConfigFilter(d.Get("filter"), d, config)
if err != nil {
return err
} else if v, ok := d.GetOkExists("filter"); !tpgresource.IsEmptyValue(reflect.ValueOf(filterProp)) && (ok || !reflect.DeepEqual(v, filterProp)) {
obj["filter"] = filterProp
}

url, err := tpgresource.ReplaceVars(d, config, "{{"{{"}}StorageControlBasePath{{"}}"}}folders/{{"{{"}}name{{"}}"}}/locations/global/intelligenceConfig")
if err != nil {
return err
}

log.Printf("[DEBUG] Patching Intelligence config: %#v", obj)
billingProject := ""

// err == nil indicates that the billing_project value was found
if bp, err := tpgresource.GetBillingProject(d, config); err == nil {
billingProject = bp
}

headers := make(http.Header)
updateMask := []string{"filter"}

if d.HasChange("edition_config") {
updateMask = append(updateMask, "editionConfig")
}
// updateMask is a URL parameter but not present in the schema, so ReplaceVars
// won't set it
url, err = transport_tpg.AddQueryParams(url, map[string]string{"updateMask": strings.Join(updateMask, ",")})
if err != nil {
return err
}

url = strings.ReplaceAll(url, "storage/v1", "v2")

res, err := transport_tpg.SendRequest(transport_tpg.SendRequestOptions{
Config: config,
Method: "PATCH",
Project: billingProject,
RawURL: url,
UserAgent: userAgent,
Body: obj,
Timeout: d.Timeout(schema.TimeoutCreate),
Headers: headers,
})
if err != nil {
return fmt.Errorf("Error patching Intelligence config: %s", err)
}

// Store the ID now
id, err := tpgresource.ReplaceVars(d, config, "folders/{{"{{"}}name{{"}}"}}/locations/global/intelligenceConfig")
if err != nil {
return fmt.Errorf("Error constructing id: %s", err)
}
d.SetId(id)

log.Printf("[DEBUG] Finished patching Intelligence config %q: %#v", d.Id(), res)

return resourceStorageControlFolderIntelligenceConfigRead(d, meta)
Original file line number Diff line number Diff line change
Expand Up @@ -243,9 +243,10 @@ var handwrittenDatasources = map[string]*schema.Resource{
"google_storage_bucket_object": storage.DataSourceGoogleStorageBucketObject(),
"google_storage_bucket_objects": storage.DataSourceGoogleStorageBucketObjects(),
"google_storage_bucket_object_content": storage.DataSourceGoogleStorageBucketObjectContent(),
"google_storage_control_folder_intelligence_config": storagecontrol.DataSourceGoogleStorageControlFolderIntelligenceConfig(),
"google_storage_object_signed_url": storage.DataSourceGoogleSignedUrl(),
"google_storage_project_service_account": storage.DataSourceGoogleStorageProjectServiceAccount(),
"google_storage_control_project_intelligence_config": storagecontrol.DataSourceGoogleStorageControlProjectIntelligenceConfig(),
"google_storage_control_project_intelligence_config": storagecontrol.DataSourceGoogleStorageControlProjectIntelligenceConfig(),
"google_storage_transfer_project_service_account": storagetransfer.DataSourceGoogleStorageTransferProjectServiceAccount(),
"google_tags_tag_key": tags.DataSourceGoogleTagsTagKey(),
"google_tags_tag_keys": tags.DataSourceGoogleTagsTagKeys(),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package storagecontrol

import (
"fmt"

"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
"github.com/hashicorp/terraform-provider-google/google/tpgresource"
transport_tpg "github.com/hashicorp/terraform-provider-google/google/transport"
)

func DataSourceGoogleStorageControlFolderIntelligenceConfig() *schema.Resource {

dsSchema := tpgresource.DatasourceSchemaFromResourceSchema(ResourceStorageControlFolderIntelligenceConfig().Schema)
tpgresource.AddRequiredFieldsToSchema(dsSchema, "name")

return &schema.Resource{
Read: dataSourceGoogleStorageControlFolderIntelligenceConfigRead,
Schema: dsSchema,
}
}

func dataSourceGoogleStorageControlFolderIntelligenceConfigRead(d *schema.ResourceData, meta interface{}) error {
config := meta.(*transport_tpg.Config)

id, err := tpgresource.ReplaceVars(d, config, "folders/{{name}}/locations/global/intelligenceConfig")
if err != nil {
return fmt.Errorf("Error constructing id: %s", err)
}
d.SetId(id)
err = resourceStorageControlFolderIntelligenceConfigRead(d, meta)
if err != nil {
return err
}

if d.Id() == "" {
return fmt.Errorf("%s not found", id)
}

return nil
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package storagecontrol_test

import (
"testing"

"github.com/hashicorp/terraform-plugin-testing/helper/resource"
"github.com/hashicorp/terraform-provider-google/google/acctest"
"github.com/hashicorp/terraform-provider-google/google/envvar"
)

func TestAccDataSourceGoogleStorageControlFolderIntelligenceConfig_basic(t *testing.T) {
t.Parallel()

context := map[string]interface{}{
"random_suffix": acctest.RandString(t, 10),
"org_id": envvar.GetTestOrgFromEnv(t),
}

acctest.VcrTest(t, resource.TestCase{
PreCheck: func() { acctest.AccTestPreCheck(t) },
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories(t),
ExternalProviders: map[string]resource.ExternalProvider{
"time": {},
},
Steps: []resource.TestStep{
{
Config: testAccDataSourceGoogleStorageControlFolderIntelligenceConfig_basic(context),
Check: resource.ComposeTestCheckFunc(
acctest.CheckDataSourceStateMatchesResourceState("data.google_storage_control_folder_intelligence_config.folder_storage_intelligence", "google_storage_control_folder_intelligence_config.folder_storage_intelligence"),
),
},
},
})
}

func testAccDataSourceGoogleStorageControlFolderIntelligenceConfig_basic(context map[string]interface{}) string {
return acctest.Nprintf(`
resource "google_folder" "folder" {
parent = "organizations/%{org_id}"
display_name = "tf-test-folder-name%{random_suffix}"
deletion_protection=false
}

resource "time_sleep" "wait_120_seconds" {
depends_on = [google_folder.folder]
create_duration = "120s"
}

resource "google_storage_control_folder_intelligence_config" "folder_storage_intelligence" {
name = google_folder.folder.folder_id
edition_config = "STANDARD"
depends_on = [time_sleep.wait_120_seconds]
}

data "google_storage_control_folder_intelligence_config" "folder_storage_intelligence" {
name = google_storage_control_folder_intelligence_config.folder_storage_intelligence.name
}
`, context)
}
Loading
Loading