Skip to content

feat: add force_batch to iceberg_merge and incremental_insert #482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

Danya-Fpnk
Copy link
Contributor

Description

Resolves: #481

Add force_batch to iceberg_merge and incremental_insert

Models used to test

{{ config(
        materialized='incremental',
        incremental_strategy='append',
        partitioned_by=['date_column'],
        force_batch=true
    )
}}

select
    random() as rnd,
    cast(date_column as date) as date_column
from (
    values (
        sequence(from_iso8601_date('2023-01-01'), from_iso8601_date('2023-07-31'), interval '1' day)
    )
) as t1(date_array)
cross join unnest(date_array) as t2(date_column)
{{ config(
        table_type='iceberg',
        materialized='incremental',
        incremental_strategy='merge',
        unique_key=['date_column'],
        partitioned_by=['date_column'],
        force_batch=true
    )
}}

{% if is_incremental() %}
    select
        1 as rnd,
        cast(date_column as date) as date_column
    from (
        values (
            sequence(from_iso8601_date('2023-01-01'), from_iso8601_date('2023-07-31'), interval '1' day)
        )
    ) as t1(date_array)
    cross join unnest(date_array) as t2(date_column)
{% else %}
    select
        2 as rnd,
        cast(date_column as date) as date_column
    from (
        values (
            sequence(from_iso8601_date('2023-01-01'), from_iso8601_date('2023-07-31'), interval '1' day)
        )
    ) as t1(date_array)
    cross join unnest(date_array) as t2(date_column)
{% endif %}

Checklist

  • You followed contributing section
  • You kept your Pull Request small and focused on a single feature or bug fix.
  • You added unit testing when necessary
  • You added functional testing when necessary

@Danya-Fpnk Danya-Fpnk changed the title feat: add force_batch to merge incremental feat: add force_batch to iceberg_merge and incremental_insert Oct 31, 2023
svdimchenko
svdimchenko previously approved these changes Oct 31, 2023
Copy link
Contributor

@svdimchenko svdimchenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solid job done 💪

Copy link
Contributor

@nicor88 nicor88 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job 💯

@svdimchenko svdimchenko merged commit fe44b1c into dbt-labs:main Oct 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add force_batch to merge incremental
3 participants