Skip to content

Add BigQuery BigLake Managed Tables as a Datastream destination #13189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

m-habib
Copy link
Member

@m-habib m-habib commented Feb 26, 2025

Introduce support for BigQuery BigLake Managed Tables as a destination for Datastream streams.

Release Note Template for Downstream PRs (will be copied)

datastream: added `blmt_config` field to `bigquery_destination_config` resource to enable support for BigLake Managed Tables streams.

@modular-magician modular-magician added the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Feb 26, 2025
@m-habib m-habib marked this pull request as ready for review March 23, 2025 08:28
@github-actions github-actions bot requested a review from c2thorn March 23, 2025 08:29
Copy link

Hello! I am a robot. Tests will require approval from a repository maintainer to run. Googlers: see go/terraform-auto-test-runs to set up automatic test runs.

@c2thorn, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@m-habib
Copy link
Member Author

m-habib commented Mar 23, 2025

Hi @c2thorn,

These changes have been tested locally. I can provide the test results privately upon request.

@m-habib m-habib changed the title Personal/mhabib/add blmt to datastream streams Add BigQuery BigLake Managed Tables as Datastream destination Mar 25, 2025
@m-habib m-habib changed the title Add BigQuery BigLake Managed Tables as Datastream destination Add BigQuery BigLake Managed Tables as a Datastream destination Mar 25, 2025
@github-actions github-actions bot requested a review from c2thorn March 27, 2025 20:30
@modular-magician modular-magician added service/datastream and removed awaiting-approval Pull requests that need reviewer's approval to run presubmit tests labels Mar 27, 2025
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 483 insertions(+))
google-beta provider: Diff ( 4 files changed, 483 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 74 insertions(+))
Open in Cloud Shell: Diff ( 4 files changed, 228 insertions(+))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 14
Passed tests: 5
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages
  • datastream
#### Non-exercised tests

🔴 Tests were added that are skipped in VCR:

  • TestAccDatastreamStream_datastreamStreamBigqueryBlmtExample
    🟢 All tests passed!

View the build log

@m-habib m-habib force-pushed the personal/mhabib/addBlmtToDatastreamStreams branch from 847bf5b to d43e9bc Compare March 27, 2025 20:46
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 481 insertions(+))
google-beta provider: Diff ( 4 files changed, 481 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 74 insertions(+))
Open in Cloud Shell: Diff ( 4 files changed, 227 insertions(+))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 14
Passed tests: 5
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages
  • datastream
#### Non-exercised tests

🔴 Tests were added that are skipped in VCR:

  • TestAccDatastreamStream_datastreamStreamBigqueryBlmtExample
    🟢 All tests passed!

View the build log

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 315 insertions(+))
google-beta provider: Diff ( 3 files changed, 315 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 74 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_datastream_stream (7 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_datastream_stream" "primary" {
  destination_config {
    bigquery_destination_config {
      blmt_config {
        bucket          = # value needed
        connection_name = # value needed
        file_format     = # value needed
        root_path       = # value needed
        table_format    = # value needed
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 13
Passed tests: 5
Skipped tests: 8
Affected tests: 0

Click here to see the affected service packages
  • datastream

🟢 All tests passed!

View the build log

Copy link
Member

@c2thorn c2thorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

received test logs offline, LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants