Skip to content

Handle ephemeral models better #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 29, 2020
Merged

Handle ephemeral models better #243

merged 3 commits into from
Jun 29, 2020

Conversation

clrcrl
Copy link
Contributor

@clrcrl clrcrl commented Jun 29, 2020

Description & motivation

We get questions about this semi-regularly, so we should just handle the error better.

Checklist

  • I have verified that these changes work locally
  • N/A: I have updated the README.md (if applicable)
  • N/A I have added tests & descriptions to my models (and macros if applicable)

I did not add any test for this macro, since I can't figure out how to test for a failure!

I also worry that I'm going to break someone's project somehow...

@clrcrl clrcrl requested a review from jtcohen6 June 29, 2020 13:55
@clrcrl
Copy link
Contributor Author

clrcrl commented Jun 29, 2020

Here's how I tested it locally:

$ cat integration_tests/models/my_ephemeral.sql
{{
    config(
        materialized='ephemeral'
    )
}}

select 1 as column
$ cat integration_tests/models/test_ephemeral.sql
-- test any non-ephemeral model
{% do dbt_utils._is_ephemeral(ref('test_date_spine'), 'test') %}

{% do dbt_utils._is_ephemeral(ref('my_ephemeral'), 'test') %}
$ cd integration_tests/

$ dbt run -m test_ephemeral
Running with dbt=0.17.1-rc1

10:14:35 | Concurrency: 1 threads (target='redshift')
10:14:35 |
10:14:36 | 1 of 1 START view model dbt_claire.test_ephemeral.................... [RUN]
10:14:36 | 1 of 1 ERROR creating view model dbt_claire.test_ephemeral........... [ERROR in 0.02s]
10:14:36 |
10:14:36 | Finished running 1 view model in 2.11s.

Completed with 1 error and 0 warnings:

Compilation Error in model test_ephemeral (models/test_ephemeral.sql)

  The `test` macro cannot be used with ephemeral models, as it relies on the information schema.

  `__dbt__CTE__my_ephemeral` is an ephemeral model. Consider making is a view or table instead.


  > in macro _is_ephemeral (macros/cross_db_utils/_is_ephemeral.sql)
  > called by model test_ephemeral (models/test_ephemeral.sql)
  > called by model test_ephemeral (models/test_ephemeral.sql)

Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

I can't find a good way to get back the model name instead of the CTE name — I could strip out the __dbt__CTE__, but am worried it might lead to other weirdness

@clrcrl clrcrl force-pushed the error-ephemeral-models branch from 273d8d6 to ba1faf8 Compare June 29, 2020 14:17
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Happy to see this change.

I could strip out the dbt__CTE, but am worried it might lead to other weirdness

Short of accessing graph.nodes, I think this is the only way to go—and I don't hate it, __dbt__CTE__ should be a "reserved" prefix.

Unrelated: in messing around with the above, TIL

Operations can not ref() ephemeral nodes, but my_first_dbt_model is ephemeral

@clrcrl
Copy link
Contributor Author

clrcrl commented Jun 29, 2020

Yah, related issue here

@clrcrl clrcrl merged commit 1f27b03 into master Jun 29, 2020
@joellabes joellabes deleted the error-ephemeral-models branch October 21, 2021 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants