Add Dataproc Metastore Database Resource for IAM support

[Noremac201]

Fixes hashicorp/terraform-provider-google#21822

This resource is only meant for IAM usage, as it doesn't really "exist" in GCP. The "database" resource cannot be created via terraform, so the example configuration required a custom yaml.tmpl for testing.

Tested via running acctest with personal credentials.

metastore: Add `google_dataproc_metastore_database_iam_*` resources 

[github-actions]

Hello! I am a robot. Tests will require approval from a repository maintainer to run. Googlers: see go/terraform-auto-test-runs to set up automatic test runs.

@melinath, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[github-actions]

@melinath This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 793 insertions(+), 2 deletions(-))
google-beta provider: Diff ( 5 files changed, 793 insertions(+), 2 deletions(-))
terraform-google-conversion: Diff ( 3 files changed, 396 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_metastore_database_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_metastore_database_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 30
Passed tests: 25
Skipped tests: 2
Affected tests: 3

Click here to see the affected service packages

• dataprocmetastore

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDataprocMetastoreDatabaseIamBindingGenerated
• TestAccDataprocMetastoreDatabaseIamMemberGenerated
• TestAccDataprocMetastoreDatabaseIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

🔴 Tests failed during RECORDING mode:
TestAccDataprocMetastoreDatabaseIamBindingGenerated [Error message] [Debug log]
TestAccDataprocMetastoreDatabaseIamMemberGenerated [Error message] [Debug log]
TestAccDataprocMetastoreDatabaseIamPolicyGenerated [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[melinath]

It looks like the tests are failing because the database doesn't exist. You mentioned the database can't be provisioned with Terraform - why not? How is it expected to be provisioned?

[Noremac201]

I see, yeah I'm working on a fix, technically "default" db should always exist, but for the Table test, I need to add a fix anyways.

These are represented by Apache Hive Metastore databases created via Hive, Spark, etc.

The fix should be to create a deterministic DB via Dataproc Job resource, then update via Terraform.

[Noremac201]

Alright, it's kinda hacky, but for the most part, I had to copy what ./mmv1/products/iap/AppEngineService.yaml was doing to make the automated tests work.

I tested locally and it worked:

$ make testacc TEST=./google/services/dataprocmetastore/ TESTARGS='-run=TestAccDataprocMetastoreDataba
seIamPolicyGenerated$$'
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google/services/dataprocmetastore/ -v -run=TestAccDataprocMetastoreDatabaseIamPolicyGenerated$ -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccDataprocMetastoreDatabaseIamPolicyGenerated
=== PAUSE TestAccDataprocMetastoreDatabaseIamPolicyGenerated
=== CONT  TestAccDataprocMetastoreDatabaseIamPolicyGenerated
--- PASS: TestAccDataprocMetastoreDatabaseIamPolicyGenerated (1415.89s)
PASS
ok  	github.com/hashicorp/terraform-provider-google/google/services/dataprocmetastore	1416.224s

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
google-beta provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
terraform-google-conversion: Diff ( 3 files changed, 396 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_metastore_database_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_metastore_database_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 30
Passed tests: 25
Skipped tests: 2
Affected tests: 3

Click here to see the affected service packages

• dataprocmetastore

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDataprocMetastoreDatabaseIamBindingGenerated
• TestAccDataprocMetastoreDatabaseIamMemberGenerated
• TestAccDataprocMetastoreDatabaseIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccDataprocMetastoreDatabaseIamMemberGenerated [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🔴 Tests failed during RECORDING mode:
TestAccDataprocMetastoreDatabaseIamBindingGenerated [Error message] [Debug log]
TestAccDataprocMetastoreDatabaseIamPolicyGenerated [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[Noremac201]

Updated to have use the suffix for the dataproc_cluster

This should stop the 409s

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
google-beta provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
terraform-google-conversion: Diff ( 3 files changed, 396 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_metastore_database_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_metastore_database_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 30
Passed tests: 25
Skipped tests: 2
Affected tests: 3

Click here to see the affected service packages

• dataprocmetastore

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDataprocMetastoreDatabaseIamBindingGenerated
• TestAccDataprocMetastoreDatabaseIamMemberGenerated
• TestAccDataprocMetastoreDatabaseIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

🔴 Tests failed during RECORDING mode:
TestAccDataprocMetastoreDatabaseIamBindingGenerated [Error message] [Debug log]
TestAccDataprocMetastoreDatabaseIamMemberGenerated [Error message] [Debug log]
TestAccDataprocMetastoreDatabaseIamPolicyGenerated [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[Noremac201]

Fixed incorrect service_id to name.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
google-beta provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
terraform-google-conversion: Diff ( 3 files changed, 396 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_metastore_database_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_metastore_database_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 30
Passed tests: 25
Skipped tests: 2
Affected tests: 3

Click here to see the affected service packages

• dataprocmetastore

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDataprocMetastoreDatabaseIamBindingGenerated
• TestAccDataprocMetastoreDatabaseIamMemberGenerated
• TestAccDataprocMetastoreDatabaseIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccDataprocMetastoreDatabaseIamBindingGenerated [Debug log]
TestAccDataprocMetastoreDatabaseIamMemberGenerated [Debug log]
TestAccDataprocMetastoreDatabaseIamPolicyGenerated [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🟢 All tests passed!

View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
google-beta provider: Diff ( 5 files changed, 990 insertions(+), 2 deletions(-))
terraform-google-conversion: Diff ( 3 files changed, 396 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_metastore_database_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_metastore_database_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_metastore_database_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 30
Passed tests: 25
Skipped tests: 2
Affected tests: 3

Click here to see the affected service packages

• dataprocmetastore

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDataprocMetastoreDatabaseIamBindingGenerated
• TestAccDataprocMetastoreDatabaseIamMemberGenerated
• TestAccDataprocMetastoreDatabaseIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccDataprocMetastoreDatabaseIamBindingGenerated [Debug log]
TestAccDataprocMetastoreDatabaseIamMemberGenerated [Debug log]
TestAccDataprocMetastoreDatabaseIamPolicyGenerated [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🟢 All tests passed!

View the build log or the debug log for each test

[melinath]

LGTM