Add support for redshift testing in GitHub CI via tox #204
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
emmyoop
changed the title
dbeatty10
changed the title
dbeatty10
reviewed
Nov 20, 2024
dosubot
bot
added
the
size:M
dbeatty10
approved these changes
Nov 22, 2024
insuhpak
added a commit
to cdp-ucsc/cdp-ucsc-dbt-codegen
that referenced
this pull request
Apr 23, 2025
…t-codegen (#9) * Update Changelog, temporarily remove Case Sensitivity testing (dbt-labs#174) * Update changelog * Fix typo in CHANGELOG * Attempt to fix integration test case sensitivity for redshift * Always lower on Redshift * Fix indentation * Fix whitespace * Fix whitespace pt 2 * Update case sensitive seeds into folder * Use + for quote_columns config * Lower schema on redshift * Use target.type * Do some nonsense to make this work for Redshift * Move seeds config to properties.yml * Bypass redshift completely * Temporarily bypass Redshift in CI completely * Turn Redshift CI back on * Delete case sensitivity test * Delete case sensitive seed * Delete properties.yml for case sensitive seeds * Update testing structure for dbt Labs testing support - postgres (dbt-labs#181) * add tox * add postgres/redshift/bq * add more wh, update var names * fix profiles key * move supported adapters * Add CI workflow * update to allow circleci to keep working * fix BQ var name * remove config from profile * move to just support postgres * fix vars * use premade workflow * add newline * use merged version * add comments about future adapters * use tag * update readme * Add lines * reowkr profile so it can be reused * add sample env files and fix comment * Update .circleci/config.yml Co-authored-by: Doug Beatty <[email protected]> * let circleCI set the schema * fix readme instructions * undo non-postgres changes --------- Co-authored-by: Doug Beatty <[email protected]> * Restore CI test for case-sensitive identifiers when generating sources (dbt-labs#192) * Restore case sensitivity test * Try to make integration test adapter-agnostic * Try to make integration test CI-agnostic and adapter-agnostic * Use adapter-agnostic data types * Temporarily hard-code case-insensitive seed name for Snowflake * Temporarily hard-code case-insensitive seed name for Snowflake * Changlogs for 0.12.0, 0.12.1, and 0.13.0-b1 (dbt-labs#196) * add snowflake (dbt-labs#198) * add snowflake * fix profiles * check what’s installed * pass in snowflake password * and more env to gitignore * Fix quoted identifiers in the `generate_base_model` macro for BigQuery (dbt-labs#199) * Use `adapter.quote` to create a case-sensitive quoted identifier for column names * Force a failure for all adapters to help troubleshoot * Revert "Force a failure for all adapters to help troubleshoot" This reverts commit d707832. * Use `adapter.quote` to create a case-sensitive quoted identifier for column names in `generate_base_model` macro * Try removing Redshift-specific logic (dbt-labs#208) * Use the `cimg/postgres` Docker image created by CircleCI with continu… (dbt-labs#214) * Use the `cimg/postgres` Docker image created by CircleCI with continuous integration builds in mind * Add the root Postgres user to the environment * Independent workflow job for dbt-postgres (dbt-labs#215) * Independent workflow job for dbt-postgres * Remove activation of virtual environment * Try without `python -m` * Independent workflow job for dbt-redshift * Independent workflow job for dbt-snowflake * Independent workflow job for dbt-snowflake * Independent workflow job for dbt-bigquery * Independent workflow job for dbt-bigquery * Independent workflow job for dbt-bigquery * Independent workflow job for dbt-bigquery * Independent workflow job for dbt-bigquery * Setup environment variables for dbt-bigquery * Simplify environment variables for BigQuery in CircleCI (dbt-labs#216) * Simplify environment variables for BigQuery in CircleCI * Fix YAML parsing error * Fix reference to environment variable * Fix reference to environment variable * Stop installing prereleases from PyPI in favor of stable releases only (dbt-labs#220) * Upgrade to Python 3.11 in CircleCI (dbt-labs#222) * Use dynamic schema names rather than hardcoded ones (dbt-labs#224) * Disable two CI tests * Use a dynamic schema name based off the target schema rather than a hardcoded one * Restore one of the CI tests * Try updating the expected output * Update expected model given upstream changes * Restore the other CI test * Update expected model given upstream changes * add support for redshift testing (dbt-labs#204) Co-authored-by: Doug Beatty <[email protected]> * Add support for bigquery testing in GitHub CI via tox (dbt-labs#203) * add support for bigquery testing * add missing var in tox file * Temporarily only run CI tests for BigQuery * Prefix the schema for the data source in CI with the name of the target schema * Store artifacts for logs and target directories for BigQuery * Set up environment variable for BigQuery credentials (keyfile for service account JSON) * Set the custom schema in the source definition * Use the target schema * Try to align actual vs. expected when the schema name is variable * Remove extraneous storage of artifacts * Temporarily disable two failing CI tests * Revert "Temporarily disable two failing CI tests" This reverts commit d70d776. --------- Co-authored-by: Doug Beatty <[email protected]> * Update changelog for 0.13.0 release (dbt-labs#227) * Revert "Restore CI test for case-sensitive identifiers when generating sources (dbt-labs#192)" (dbt-labs#230) This reverts commit 7acc8c0. * Update changelog for 0.13.1 release (dbt-labs#232) * Upgrade from Postgres 9 to 17 (dbt-labs#234) * Upgrade from Postgres 9 to 17 * The postgres image in CircleCI needs a major and minor version specified * Update ci.yml (dbt-labs#235) * Bigquery repeated data type (dbt-labs#236) * Handle BigQuery repeated fields data_types * include nested repated structs * override repeated struct data_type with array * Add trailing newline * update changelog * Update CHANGELOG.md --------- Co-authored-by: Doug Beatty <[email protected]> * Remove "I have added an entry to CHANGELOG.md" from the PR template (dbt-labs#239) * Contributors shouldn't edit the `CHANGELOG.md` directly anymore (dbt-labs#240) * Remove the PR checklist items related to the type of change (dbt-labs#243) * Align the PR description with dbt-core, dbt-adapters, etc. (dbt-labs#244) * Align the PR description with dbt-core, dbt-adapters, etc. * Add hyperlink to the open issues * Align the pull request template with `dbt-utils` (dbt-labs#246) * Update CODEOWNERS file (dbt-labs#248) * Update CODEOWNERS file with global codeowner * Update .github/CODEOWNERS --------- Co-authored-by: Doug Beatty <[email protected]> * adding `generate_unit_test_template` macro (dbt-labs#251) * adding generate_unit_test_template_macro * added a relation exists check * adding arg to controle inline/multiline columns * Removing duplicated `get_resource_from_unique_id` macro * Add newline to end of file * Add a simple incremental model for testing * CI tests * Use dispatch, etc. * Usage documentation for `generate_unit_test_template` macro --------- Co-authored-by: Bruno Souza de Lima <[email protected]> Co-authored-by: Doug Beatty <[email protected]> --------- Co-authored-by: winnie <[email protected]> Co-authored-by: Emily Rockman <[email protected]> Co-authored-by: Doug Beatty <[email protected]> Co-authored-by: Doug Beatty <[email protected]> Co-authored-by: Magnús Þór Benediktsson <[email protected]> Co-authored-by: security-dbtlabs <[email protected]> Co-authored-by: Bruno S. de Lima <[email protected]> Co-authored-by: Bruno Souza de Lima <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #201
Description & motivation
Adds tox support for testing against redshift.
See dbt-labs/dbt-utils#919 and dbt-labs/dbt-utils#947 for example in
dbt-utils.More detail
Was originally hitting a schema error. After #224 was merged, no longer hitting that schema error.
Checklist