-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Add storage_control_folder_intelligence_config resource. #13394
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ScottSuarez
merged 8 commits into
GoogleCloudPlatform:main
from
kautikdk:folder_storage_intelligence
Mar 25, 2025
Merged
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
63605e2
Adds storage_control_folder_intelligence_config resource.
kautikdk d523aff
Add empty value tests and fixes formatting issues.
kautikdk fc898fe
Add empty value tests and fixes formatting issues.
kautikdk e0b1b64
Removes duplicate function declarations.
kautikdk cb82c62
Removes URL modifier
kautikdk 6da009e
Adds example documentation.
kautikdk 4804737
Merge branch 'main' into folder_storage_intelligence
kautikdk 41f0f36
Remove custom_create and add pre_create.
kautikdk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
179 changes: 179 additions & 0 deletions
179
mmv1/products/storagecontrol/FolderIntelligenceConfig.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,179 @@ | ||
# Copyright 2025 Google Inc. | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
--- | ||
# API resource name | ||
name: 'FolderIntelligenceConfig' | ||
kind: 'storagecontrol#intelligenceconfig' | ||
# Resource description for the provider documentation. | ||
description: | | ||
The Folder Storage Intelligence resource represents GCS Storage Intelligence operating on individual GCP Folder. Storage Intelligence is a singleton resource and individual instance exists on each GCP Folder. | ||
|
||
Storage Intelligence is for Storage Admins to manage GCP storage assets at scale for performance, cost, security & compliance. | ||
|
||
docs: | ||
warning: | | ||
Storage Intelligence Config is a singleton resource which cannot be created or deleted. A single instance of Storage Intelligence Config exist for each GCP Folder. Terraform does not create or destroy this resource. | ||
Terraform resource creation for this resource is simply an update operation on existing resource with specified properties. Terraform deletion won't have any effect on this resource rather it will only remove it from the state file. | ||
|
||
# URL for the resource's standard Get method. https://google.aip.dev/131 | ||
self_link: 'folders/{{name}}/locations/global/intelligenceConfig' | ||
|
||
custom_code: | ||
custom_create: templates/terraform/custom_create/storage_control_folder_intelligence.go.tmpl | ||
|
||
# The HTTP verb used to update a resource. Allowed values: :POST, :PUT, :PATCH. Default: :PUT. | ||
update_verb: 'PATCH' | ||
# If true, the resource sets an `updateMask` query parameter listing modified | ||
# fields when updating the resource. If false, it does not. | ||
update_mask: true | ||
|
||
exclude_delete: true | ||
|
||
import_format: | ||
- 'folders/{{name}}/locations/global/intelligenceConfig' | ||
|
||
# If true, code for handling long-running operations is generated along with | ||
# the resource. If false, that code is not generated. | ||
autogen_async: false | ||
ScottSuarez marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
properties: | ||
# Fields go here | ||
- name: 'name' | ||
type: String | ||
required: true | ||
immutable: true | ||
url_param_only: true | ||
description: | | ||
Identifier of the GCP Folder. For GCP Folder, this field can be folder number. | ||
- name: 'editionConfig' | ||
type: String | ||
required: false | ||
default_from_api: true | ||
description: | | ||
Edition configuration of the Storage Intelligence resource. Valid values are INHERIT, TRIAL, DISABLED and STANDARD. | ||
- name: 'updateTime' | ||
type: String | ||
output: true | ||
description: | | ||
The time at which the Storage Intelligence Config resource is last updated. | ||
- name: 'filter' | ||
type: NestedObject | ||
diff_suppress_func: 'intelligenceFilterDiffSuppress' | ||
description: | | ||
Filter over location and bucket using include or exclude semantics. Resources that match the include or exclude filter are exclusively included or excluded from the Storage Intelligence plan. | ||
properties: | ||
- name: excludedCloudStorageBuckets | ||
type: NestedObject | ||
required: false | ||
description: | | ||
Buckets to exclude from the Storage Intelligence plan. | ||
conflicts: | ||
- 'filter.0.included_cloud_storage_buckets' | ||
at_least_one_of: | ||
- 'filter.0.included_cloud_storage_buckets' | ||
- 'filter.0.excluded_cloud_storage_buckets' | ||
- 'filter.0.included_cloud_storage_locations' | ||
- 'filter.0.excluded_cloud_storage_locations' | ||
diff_suppress_func: 'intelligenceFilterExcludedCloudStorageBucketsDiffSuppress' | ||
properties: | ||
- name: bucketIdRegexes | ||
required: true | ||
type: Array | ||
send_empty_value: true | ||
item_type: | ||
type: String | ||
description: | | ||
List of bucket id regexes to exclude in the storage intelligence plan. | ||
- name: includedCloudStorageBuckets | ||
type: NestedObject | ||
required: false | ||
description: | | ||
Buckets to include in the Storage Intelligence plan. | ||
conflicts: | ||
- 'filter.0.excluded_cloud_storage_buckets' | ||
at_least_one_of: | ||
- 'filter.0.included_cloud_storage_buckets' | ||
- 'filter.0.excluded_cloud_storage_buckets' | ||
- 'filter.0.included_cloud_storage_locations' | ||
- 'filter.0.excluded_cloud_storage_locations' | ||
diff_suppress_func: 'intelligenceFilterincludedCloudStorageBucketsDiffSuppress' | ||
properties: | ||
- name: bucketIdRegexes | ||
required: true | ||
send_empty_value: true | ||
type: Array | ||
item_type: | ||
type: String | ||
description: | | ||
List of bucket id regexes to exclude in the storage intelligence plan. | ||
- name: excludedCloudStorageLocations | ||
type: NestedObject | ||
required: false | ||
description: | | ||
Locations to exclude from the Storage Intelligence plan. | ||
conflicts: | ||
- 'filter.0.included_cloud_storage_locations' | ||
at_least_one_of: | ||
- 'filter.0.included_cloud_storage_buckets' | ||
- 'filter.0.excluded_cloud_storage_buckets' | ||
- 'filter.0.included_cloud_storage_locations' | ||
- 'filter.0.excluded_cloud_storage_locations' | ||
diff_suppress_func: 'intelligenceFilterExcludedCloudStorageLocationsDiffSuppress' | ||
properties: | ||
- name: locations | ||
type: Array | ||
required: true | ||
send_empty_value: true | ||
description: | | ||
List of locations. | ||
item_type: | ||
type: String | ||
- name: includedCloudStorageLocations | ||
type: NestedObject | ||
required: false | ||
description: | | ||
Locations to include in the Storage Intelligence plan. | ||
conflicts: | ||
- 'filter.0.excluded_cloud_storage_locations' | ||
at_least_one_of: | ||
- 'filter.0.included_cloud_storage_buckets' | ||
- 'filter.0.excluded_cloud_storage_buckets' | ||
- 'filter.0.included_cloud_storage_locations' | ||
- 'filter.0.excluded_cloud_storage_locations' | ||
diff_suppress_func: 'intelligenceFilterincludedCloudStorageLocationsDiffSuppress' | ||
properties: | ||
- name: locations | ||
type: Array | ||
required: true | ||
send_empty_value: true | ||
description: | | ||
List of locations. | ||
item_type: | ||
type: String | ||
- name: 'effectiveIntelligenceConfig' | ||
output: true | ||
description: | | ||
The Intelligence config that is effective for the resource. | ||
type: NestedObject | ||
properties: | ||
- name: intelligenceConfig | ||
type: String | ||
output: true | ||
description: | | ||
The Intelligence config resource that is applied for the target resource. | ||
- name: effectiveEdition | ||
type: String | ||
output: true | ||
description: | | ||
The `StorageIntelligence` edition that is applicable for the resource. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
72 changes: 72 additions & 0 deletions
72
mmv1/templates/terraform/custom_create/storage_control_folder_intelligence.go.tmpl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
|
||
userAgent, err := tpgresource.GenerateUserAgentString(d, config.UserAgent) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
obj := make(map[string]interface{}) | ||
editionConfigProp, err := expandStorageControlFolderIntelligenceConfigEditionConfig(d.Get("edition_config"), d, config) | ||
if err != nil { | ||
return err | ||
} else if v, ok := d.GetOkExists("edition_config"); !tpgresource.IsEmptyValue(reflect.ValueOf(editionConfigProp)) && (ok || !reflect.DeepEqual(v, editionConfigProp)) { | ||
obj["editionConfig"] = editionConfigProp | ||
} | ||
filterProp, err := expandStorageControlFolderIntelligenceConfigFilter(d.Get("filter"), d, config) | ||
if err != nil { | ||
return err | ||
} else if v, ok := d.GetOkExists("filter"); !tpgresource.IsEmptyValue(reflect.ValueOf(filterProp)) && (ok || !reflect.DeepEqual(v, filterProp)) { | ||
obj["filter"] = filterProp | ||
} | ||
|
||
url, err := tpgresource.ReplaceVars(d, config, "{{"{{"}}StorageControlBasePath{{"}}"}}folders/{{"{{"}}name{{"}}"}}/locations/global/intelligenceConfig") | ||
if err != nil { | ||
return err | ||
} | ||
|
||
log.Printf("[DEBUG] Patching Intelligence config: %#v", obj) | ||
billingProject := "" | ||
|
||
// err == nil indicates that the billing_project value was found | ||
if bp, err := tpgresource.GetBillingProject(d, config); err == nil { | ||
billingProject = bp | ||
} | ||
|
||
headers := make(http.Header) | ||
updateMask := []string{"filter"} | ||
|
||
if d.HasChange("edition_config") { | ||
updateMask = append(updateMask, "editionConfig") | ||
} | ||
// updateMask is a URL parameter but not present in the schema, so ReplaceVars | ||
// won't set it | ||
url, err = transport_tpg.AddQueryParams(url, map[string]string{"updateMask": strings.Join(updateMask, ",")}) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
url = strings.ReplaceAll(url, "storage/v1", "v2") | ||
|
||
res, err := transport_tpg.SendRequest(transport_tpg.SendRequestOptions{ | ||
Config: config, | ||
Method: "PATCH", | ||
Project: billingProject, | ||
RawURL: url, | ||
UserAgent: userAgent, | ||
Body: obj, | ||
Timeout: d.Timeout(schema.TimeoutCreate), | ||
Headers: headers, | ||
}) | ||
if err != nil { | ||
return fmt.Errorf("Error patching Intelligence config: %s", err) | ||
} | ||
|
||
// Store the ID now | ||
id, err := tpgresource.ReplaceVars(d, config, "folders/{{"{{"}}name{{"}}"}}/locations/global/intelligenceConfig") | ||
if err != nil { | ||
return fmt.Errorf("Error constructing id: %s", err) | ||
} | ||
d.SetId(id) | ||
|
||
log.Printf("[DEBUG] Finished patching Intelligence config %q: %#v", d.Id(), res) | ||
|
||
return resourceStorageControlFolderIntelligenceConfigRead(d, meta) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
40 changes: 40 additions & 0 deletions
40
...rraform/services/storagecontrol/data_source_storage_control_folder_intelligence_config.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
package storagecontrol | ||
|
||
import ( | ||
"fmt" | ||
|
||
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema" | ||
"github.com/hashicorp/terraform-provider-google/google/tpgresource" | ||
transport_tpg "github.com/hashicorp/terraform-provider-google/google/transport" | ||
) | ||
|
||
func DataSourceGoogleStorageControlFolderIntelligenceConfig() *schema.Resource { | ||
|
||
dsSchema := tpgresource.DatasourceSchemaFromResourceSchema(ResourceStorageControlFolderIntelligenceConfig().Schema) | ||
tpgresource.AddRequiredFieldsToSchema(dsSchema, "name") | ||
|
||
return &schema.Resource{ | ||
Read: dataSourceGoogleStorageControlFolderIntelligenceConfigRead, | ||
Schema: dsSchema, | ||
} | ||
} | ||
|
||
func dataSourceGoogleStorageControlFolderIntelligenceConfigRead(d *schema.ResourceData, meta interface{}) error { | ||
config := meta.(*transport_tpg.Config) | ||
|
||
id, err := tpgresource.ReplaceVars(d, config, "folders/{{name}}/locations/global/intelligenceConfig") | ||
if err != nil { | ||
return fmt.Errorf("Error constructing id: %s", err) | ||
} | ||
d.SetId(id) | ||
err = resourceStorageControlFolderIntelligenceConfigRead(d, meta) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
if d.Id() == "" { | ||
return fmt.Errorf("%s not found", id) | ||
} | ||
|
||
return nil | ||
} |
61 changes: 61 additions & 0 deletions
61
...rm/services/storagecontrol/data_source_storage_control_folder_intelligence_config_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
// Copyright (c) HashiCorp, Inc. | ||
// SPDX-License-Identifier: MPL-2.0 | ||
package storagecontrol_test | ||
|
||
import ( | ||
"testing" | ||
|
||
"github.com/hashicorp/terraform-plugin-testing/helper/resource" | ||
"github.com/hashicorp/terraform-provider-google/google/acctest" | ||
"github.com/hashicorp/terraform-provider-google/google/envvar" | ||
) | ||
|
||
func TestAccDataSourceGoogleStorageControlFolderIntelligenceConfig_basic(t *testing.T) { | ||
t.Parallel() | ||
|
||
context := map[string]interface{}{ | ||
"random_suffix": acctest.RandString(t, 10), | ||
"org_id": envvar.GetTestOrgFromEnv(t), | ||
} | ||
|
||
acctest.VcrTest(t, resource.TestCase{ | ||
PreCheck: func() { acctest.AccTestPreCheck(t) }, | ||
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories(t), | ||
ExternalProviders: map[string]resource.ExternalProvider{ | ||
"time": {}, | ||
}, | ||
Steps: []resource.TestStep{ | ||
{ | ||
Config: testAccDataSourceGoogleStorageControlFolderIntelligenceConfig_basic(context), | ||
Check: resource.ComposeTestCheckFunc( | ||
acctest.CheckDataSourceStateMatchesResourceState("data.google_storage_control_folder_intelligence_config.folder_storage_intelligence", "google_storage_control_folder_intelligence_config.folder_storage_intelligence"), | ||
), | ||
}, | ||
}, | ||
}) | ||
} | ||
|
||
func testAccDataSourceGoogleStorageControlFolderIntelligenceConfig_basic(context map[string]interface{}) string { | ||
return acctest.Nprintf(` | ||
resource "google_folder" "folder" { | ||
parent = "organizations/%{org_id}" | ||
display_name = "tf-test-folder-name%{random_suffix}" | ||
deletion_protection=false | ||
} | ||
|
||
resource "time_sleep" "wait_120_seconds" { | ||
depends_on = [google_folder.folder] | ||
create_duration = "120s" | ||
} | ||
|
||
resource "google_storage_control_folder_intelligence_config" "folder_storage_intelligence" { | ||
name = google_folder.folder.folder_id | ||
edition_config = "STANDARD" | ||
depends_on = [time_sleep.wait_120_seconds] | ||
} | ||
|
||
data "google_storage_control_folder_intelligence_config" "folder_storage_intelligence" { | ||
name = google_storage_control_folder_intelligence_config.folder_storage_intelligence.name | ||
} | ||
`, context) | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is custom create needed? pre_create seems sufficient to just alter the url. There is the question of, if we need this to be all on v2 if this is the right way to go about things. Because we will be creating on v2 and reading on v1 with the current implmentation. Open to call to discuss.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @ScottSuarez,
URL modification code was written a quite some time ago before we decided to create a new service
storagecontrol
for GCS JSON v2. I forgot to remove it, My bad! Regarding custom_create, The reason is the update_mask and PATCH method in create operation. Storage Intelligence can not be created so we mapped create operation to PATCH method but the concern is related to permanent(optional+computed) and optional fields.edition_config
is a permanent field and back-filled by server side so user can change but it is not the field that we can clear but thefilter
block is optional and not back filled by server side. Now just to simulate the create operation, we can put full update_mask of every fields which are optional but no need to send empty value for the fields which are computed or has server side value.Why we want put
update_mask
for optional fields?The main reason is we are clearly stating that the create operation is actually update operation under the hood so the first create operation diff should represent whatever will be the end state and should not generate diff after apply. That necessitates requirement of update_mask for optional fields and if we don't do it then some diff can be generated after apply operation. That will be just similar to import operation that we already support.
I have removed that URL modification line. Open to discuss any concerns here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also empty value for
edition_config
will result in server side error so we can not put update_mask for that field until it is specified by the user in create operation.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 to @ScottSuarez's point. This should be possible to do with
pre_create
custom code rather than replacing the whole create method.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Opened hashicorp/terraform-provider-google#22033 to track making this easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a little confused what the concern is with O+C fields; in general, sending an empty value for them (triggering server-side default behavior) should be correct. Although it would be possible to persist the values that already exist on the server side that might be more confusing than beneficial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh yes, checked how exactly pre-create injects code and realized it adds code snippet just before making an API call. Changed custom_create to pre_create and kept previous update_mask logic as it is while we are still discussing this behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So just to give an example in the Storage Intelligence resource,
edition_config
is a property of the resource and all the resources are having this field backfilled by the server. Users can configure this field, has server side value initially, and can not be removed so it should be Optional+Computed. Now since we can not remove it, we can not send empty value and update_mask together as it results in a InvalidValue error so we have to put update_mask only when user specifies it otherwise we will keep server side value in the create operation.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So to make sure I understand right,
edition_config
behaves as follows:Is that correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Umm, Not quite like that.
API behavior is,
edition_config
field isINHERIT
.edition_config
property from the resource.Terraform mapping,
Regarding the empty value reference, I believe my wordings made some confusion. There is an API side validation which checks if the update_mask is provided for
edition_config
. if it is provided and field is not present in the API request body then it will throw an InvalidValue error.In the create operation, we will put update_mask of
edition_config
only when it is specified by user in the resource config. Just like whatever is being done here for every field: https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/templates/terraform/update_mask.go.tmpl