Release latest

Release latest

Release 0.9.0.Final

Notable changes

NOTE: Please be aware that these modifications may affect backward compatibility. Thorough testing is strongly recommended before deployment.

This release includes several updates, including an upgrade to Apache Iceberg 1.8.1 and Debezium 3.1.1.Final, the addition of support for Iceberg decimals, and a change in how data de-duplication is handled in upsert mode.

• Upgraded Apache Iceberg to 1.8.1: This brings the latest features, performance improvements, and bug fixes from the Iceberg project.
• Upgraded Debezium to 3.1.1.Final: This incorporates the latest updates and improvements from the Debezium project.
• Added support for Iceberg decimals: Users can now correctly handle decimal data types when writing to Iceberg tables.
• Enabled usage as a Maven dependency: The project can now be easily integrated into other Maven projects using the flatten-maven-plugin.
• Switched to __source_ts_ns for upsert de-duplication: Data de-duplication in upsert mode now relies on the __source_ts_ns field. Be aware that this change may affect existing upsert configurations and data consistency if not accounted for.
• Improved Docker build process: The Docker build now correctly reflects the release version.
• Fixed deserializer usage: Corrected the deserializer used, resolving a potential issue.
• Published documentation to GitHub Pages: Project documentation is now available on GitHub Pages.

What's Changed

• Bump org.apache.iceberg:iceberg-bom from 1.7.1 to 1.8.1 by @dependabot in #529
• Bump io.debezium:debezium-build-parent from 3.1.0.Alpha2 to 3.1.0.Beta1 by @dependabot in #532
• Bump version.debezium from 3.1.0.Alpha2 to 3.1.0.Beta1 by @dependabot in #533
• Bump version.debezium from 3.1.0.Beta1 to 3.1.0.CR1 by @dependabot in #534
• Bump io.debezium:debezium-build-parent from 3.1.0.Beta1 to 3.1.0.CR1 by @dependabot in #535
• Add support for Iceberg decimals by @zilder in #537
• Utilize flatten-maven-plugin to allow debezium-server-iceberg to be used as a maven dependency by @matthewmcnew in #538
• Fix docker build should reflect release version by @ismailsimsek in #539
• Bump io.debezium:debezium-build-parent from 3.1.0.CR1 to 3.1.0.Final by @dependabot in #542
• Bump version.debezium from 3.1.0.CR1 to 3.1.0.Final by @dependabot in #541
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.5.2 to 3.5.3 by @dependabot in #536
• Bump org.testcontainers:testcontainers-bom from 1.20.5 to 1.20.6 by @dependabot in #530
• Bump io.smallrye:jandex-maven-plugin from 3.2.7 to 3.3.0 by @dependabot in #540
• Switch using __source_ts_ns for data de-duplicatinon, for upsert mode by @ismailsimsek in #546
• Revert "Bump io.smallrye:jandex-maven-plugin from 3.2.7 to 3.3.0" by @ismailsimsek in #547
• Publish coumentation to github pages by @ismailsimsek in #551
• Fix, Use correct deserializer by @ismailsimsek in #552
• Bump io.debezium:debezium-build-parent from 3.1.0.Final to 3.1.1.Final by @dependabot in #556
• Bump version.debezium from 3.1.0.Final to 3.1.1.Final by @dependabot in #555
• Bump org.testcontainers:testcontainers-bom from 1.20.6 to 1.21.0 by @dependabot in #554

New Contributors

• @zilder made their first contribution in #537
• @matthewmcnew made their first contribution in #538

Full Changelog: 0.9.0.Alpha...0.9.0.Final

Release 0.9.0.Alpha

Notable changes

This release introduces several enhancements and fixes across various areas.

• Enhanced Packaging and Licensing:
  • Maven packaging artifacts are now published to GitHub Packages, ensuring improved distribution and compliance.
  • Packaging has been enhanced with the inclusion of comprehensive third-party licenses.
• Dependency Upgrades and Removals:
  • Debezium has been upgraded to version 3.1.0.Alpha2.
  • Hadoop and PostgreSQL dependencies have been removed, streamlining the project.
• Simplified Configuration Management:
  • Configuration access and management have been centralized through a dedicated config class, utilizing Quarkus ConfigMapping.
• Improved Data Type Handling:
  • Enhanced data type handling now supports datetime, timestamp, and date formats. (When time.precision.mode is not dynamic) The connector accurately converts these data types to their Iceberg equivalents.
• Optimized Default Configuration for Iceberg Consumer:
  • Default configuration values have been optimized for the Iceberg consumer, incorporating recommended Debezium settings. These settings are now applied by default.
    • Default time.precision.mode changed to isostring.
    • Default decimal.handling.mode changed to double.
    • debezium.transforms is now set to unwrap with preconfigured unwrap settings. See default values for details. To disable it set debezium.transforms=,

NOTE: Please be aware that these modifications may affect backward compatibility. Thorough testing is strongly recommended before deployment.

What's Changed

• Add third party licenses to packaging by @ismailsimsek in #484
• Add action to Publishing Packages to GitHub Packages by @ismailsimsek in #486
• Fix github maven package deployment, Related to (#486) by @ismailsimsek in #487
• Update Create Release status badge by @ismailsimsek in #488
• Fix a typo in README.md by @cedricbatailler in #490
• Enable IcebergSchemaHistoryTest and add minor improvements to tests by @ismailsimsek in #492
• Set setDefaultTimeout and setDefaultPollInterval of Awaitility by @ismailsimsek in #493
• Disable releasing nexus-staging by @ismailsimsek in #495
• Simplify setting up spark settings, and add ResolvingFileIO to examples by @ismailsimsek in #496
• Remove hadoop dependencies and hadoop catalog by @ismailsimsek in #491
• Fix build warnins by @ismailsimsek in #497
• Improve naming and add code comments for IcebergSchemaInfo.java by @ismailsimsek in #503
• Move SchemaConverter to separate class by @ismailsimsek in #504
• Remove mongodb java driver from dependencies, use upstream driver by @ismailsimsek in #507
• Upgrade debezium to 3.1.0.Alpha2 by @ismailsimsek in #508
• Improve Handling Date fields and values by @ismailsimsek in #505
• Use config class, to centralize config access and management by @ismailsimsek in #510
• Improve the usage of test helper classes by @ismailsimsek in #511
• Use config values to handle data and schema conversion accordingly by @ismailsimsek in #509
• Improve handling Timestamp fields and values by @ismailsimsek in #513
• Simplify case when switch code by @ismailsimsek in #514
• Improve Time handling by @ismailsimsek in #515
• Create Config Class for BatchSizeWait by @ismailsimsek in #518
• Organize Config And Activate recommended default values for iceberg consumer by @ismailsimsek in #517
• Set default Decimal handling mode to double by @ismailsimsek in #519
• Document configuration and default values by @ismailsimsek in #520
• Move batch config to correct class by @ismailsimsek in #521
• Upgrade maven version to 3.9 for Dockerfile by @ismailsimsek in #523
• Remove postgresql test dependency by @ismailsimsek in #524
• Unify test injections by @ismailsimsek in #528
• Bump org.awaitility:awaitility from 4.2.2 to 4.3.0 by @dependabot in #525
• Bump org.testcontainers:testcontainers-bom from 1.20.4 to 1.20.5 by @dependabot in #526
• Bump io.smallrye:jandex-maven-plugin from 3.1.8 to 3.2.7 by @dependabot in #500

New Contributors

• @cedricbatailler made their first contribution in #490

Full Changelog: 0.8.2.Final...0.9.0.Alpha

Release 0.8.2.Final

Release 0.8.2.Final

Notable changes

• Debezium upgraded to 3.0.7.Final
• Quarkus and other dependency versions are aligned to upstream Debezium
• Various code improvements added around: python package, java testing and "schema change topic" handling

What's Changed

• Upgrade debezium to 3.x and Simplify dependencies by @ismailsimsek in #455
• Use iceberg-bom dependency and Fix jandex plugin version by @ismailsimsek in #456
• Cache maven packages by @ismailsimsek in #457
• Switch caching to actions/setup-java to cache maven by @ismailsimsek in #458
• Update dist pom, align with Debezium by @ismailsimsek in #461
• corrected directory path by @SiddiqueAhmad in #462
• Bump io.debezium:debezium-build-parent from 3.0.5.Final to 3.0.6.Final by @dependabot in #465
• Bump version.debezium from 3.0.5.Final to 3.0.6.Final by @dependabot in #464
• Update documentation and java version in Installation section by @ismailsimsek in #467
• Simplify Minio container test resource by @ismailsimsek in #468
• Add test using jdbc catalog by @ismailsimsek in #469
• Add test using rest catalog by @ismailsimsek in #470
• Add informative exception message for deduplication by @ismailsimsek in #471
• Improve python lib, add pyproject.toml by @ismailsimsek in #477
• Bump actions/stale from 9.0.0 to 9.1.0 by @dependabot in #478
• Bump io.minio:minio from 8.5.14 to 8.5.17 by @dependabot in #476
• Bump io.debezium:debezium-build-parent from 3.0.6.Final to 3.0.7.Final by @dependabot in #475
• Bump version.debezium from 3.0.6.Final to 3.0.7.Final by @dependabot in #474
• Improve schema change topic handling by @ismailsimsek in #480

New Contributors

• @SiddiqueAhmad made their first contribution in #462

Full Changelog: 0.8.1.Final...0.8.2.Final

Release 0.8.1.Final

Release 0.8.1.Final

What's Changed

• Bump version.debezium from 2.7.3.Final to 2.7.4.Final by @dependabot in #448
• Revert Quarkus to a Debezium compatible version by @panga in #447
• Align groovy dependency with Debezium. Update It to 4.0.17 and switch to new artifact org.apache.groovy by @ismailsimsek in #449
• Upgrade jackson from 2.14.2 to 2.16.2, align with Debezium by @ismailsimsek in #451
• Upgrade iceberg from 1.7.0 to 1.7.1 by @ismailsimsek in #452

New Contributors

• @panga made their first contribution in #447

Full Changelog: 0.8.0.Final...0.8.1.Final

Release 0.8.0.Final

Release 0.8.0.Final

Release 0.7.0.Final

• Debezium: 2.7.2.Final
• Apache Iceberg: 1.6.1
• Quarkus: 3.13.3

Breaking Changes

Equality deletes:

• Now Equality delete files are containing key fields only and they are written with record projection. Now delete files only includes key files, previously delete files were including full delete record.

Notable changes

• Various code improvement added. Now code base is more closely aligned with upstream iceberg-kafka-connect
• Now size of the files generated is defined write.target-file-size-bytes config, previously Long.MAX_VALUE was used
• Iceberg upgraded from 1.6.0 to 1.6.1
• Debezium upgraded from 2.7.0.Final to 2.7.2.Final
• Quarkus upgraded from 3.13.0 to 3.13.3

What's Changed

• Bump version.quarkus from 3.13.0 to 3.13.2 by @dependabot in #400
• Bump io.smallrye:jandex-maven-plugin from 3.2.1 to 3.2.2 by @dependabot in #399
• Bump version.debezium from 2.7.0.Final to 2.7.1.Final by @dependabot in #398
• Bump version.testcontainers from 1.20.0 to 1.20.1 by @dependabot in #396
• Add code improvements by @ismailsimsek in #409
• Improve code, Improve class and method namings by @ismailsimsek in #410
• Improve deduplication code by @ismailsimsek in #411
• Use write.target-file-size-bytes config, previously Long.MAX_VALUE was used by @ismailsimsek in #412
• Fix, Create GenericAppenderFactory respecting to identifier fields. And create it with table properties by @ismailsimsek in #413
• Explicitly set operationId as UUID when creating OutputFileFactory by @ismailsimsek in #414
• Use Key fields and Record Projection for deletes by @ismailsimsek in #408
• Rename variable equalityFieldIds to identifierFieldIds by @ismailsimsek in #415
• Bump org.awaitility:awaitility from 4.2.1 to 4.2.2 by @dependabot in #406
• Bump org.postgresql:postgresql from 42.7.3 to 42.7.4 by @dependabot in #407
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.1 to 3.5.0 by @dependabot in #405
• In case of exception, call writer.abort() to cleanup by @ismailsimsek in #417
• Upgrade debezium and Quarkus to latest minor version by @ismailsimsek in #418

Full Changelog: 0.6.0.Final...0.7.0.Final

Release 0.6.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.13.0

Notable changes

• new config added to control casing of iceberg table names. To change tables names uppercase or lowercase folloing configuration properties could be used, see documentation. debezium.sink.iceberg.destination-uppercase-table-names=true or debezium.sink.iceberg.destination-lowercase-table-names=true
• Quarkus upgraded from 3.12.x to 3.13.x, See Quarkus release notes here

What's Changed

• Add option to change case of destination/iceberg table names by @ismailsimsek in #389
• Bump version.quarkus from 3.12.3 to 3.13.0 by @dependabot in #392
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.5 to 3.3.1 by @dependabot in #395
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.0 to 3.3.1 by @dependabot in #394
• Bump io.smallrye:jandex-maven-plugin from 3.2.0 to 3.2.1 by @dependabot in #393

Full Changelog: 0.5.0.Final...0.6.0.Final

Release 0.5.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.12.3

What's Changed

• Bump io.minio:minio from 8.5.7 to 8.5.11 by @dependabot in #372
• Bump org.codehaus.groovy:groovy-bom from 3.0.21 to 3.0.22 by @dependabot in #374
• Bump com.squareup.okio:okio from 3.4.0 to 3.9.0 by @dependabot in #376
• Upgrade quarkus from 3.10.0 to 3.10.2 by @ismailsimsek in #382
• Bump version.testcontainers from 1.19.8 to 1.20.0 by @dependabot in #387
• Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 by @dependabot in #385

Full Changelog: 0.5.0.Beta...0.5.0.Final

Release 0.5.0.Beta

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.5.2
• Quarkus: 3.10.0

Breaking Changes

Debezium version upgraded from 2.5.x to 2.7.x:

• For Debezium 2.6 Breaking Changes please see release notes
• For Debezium 2.7 Breaking Changes please see release notes

IcebergSchemaHistory wrong table location corrected:

IcebergSchemaHistory class was using wrong table namespace {catalog-name}.{table-name} now this is corrected to use {table-namespace}.{table-name}
If you were using different catalog-name please follow following steps for migration

1. stop debezium server
2. copy the IcebergSchemaHistory table to new location: {table-namespace}.{table-name}
3. start debezium server

Configuration change:

Configuration option debezium.sink.iceberg.upsert-op-column, defaultValue = __op is removed.

Notable changes

• Quarkus upgraded from 3.6.x to 3.10.x, See Quarkus release notes here
• Java release target upgraded from 11 to 17
• Full support for debezium nested events added. Now its possible to run consumer without event flattening.
• Debezium event processing improved, unnecessary schema de-serialization is avoided.

What's Changed

• Upgrade java to 17 by @ismailsimsek in #318
• Upgrade debezium to 2.7.0.Alpha1 by @ismailsimsek in #317
• Bump version.quarkus from 3.6.9 to 3.10.0 by @dependabot in #313
• Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
• Minor, Improve naming of state tables used in tests by @ismailsimsek in #322
• Add mongodb flattening settings to example config by @ismailsimsek in #324
• Fix reading config values in debezium storage classes by @ismailsimsek in #327
• Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328
• Update codeql-analysis.yml to use java 17 by @ismailsimsek in #329
• Upgrade spark to 4.0.x version and enable Mysql tests by @ismailsimsek in #331
• Remove DynamicBatchSizeWait class by @ismailsimsek in #332
• Enforce maven version is greater than 3.8.6 by @ismailsimsek in #334
• Don't partition tables on append mode. by @ismailsimsek in #340
• Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 by @dependabot in #338
• Bump org.postgresql:postgresql from 42.7.2 to 42.7.3 by @dependabot in #337
• Bump version.testcontainers from 1.19.7 to 1.19.8 by @dependabot in #336
• Improve IcebergChangeEvent class, Optimize event and schema deserialization by @ismailsimsek in #343
• Fail deduplication if event key is null by @ismailsimsek in #344
• Improve naming, rename JsonSchema to ChangeEventSchema by @ismailsimsek in #345
• Improve code and schema conversion tests by @ismailsimsek in #346
• Add more tests on schema conversion by @ismailsimsek in #348
• Improve conversion of debezium schema to iceberg schema by @ismailsimsek in #347
• Revert "Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 (#338)" by @ismailsimsek in #349
• Improve setting identifier fields by @ismailsimsek in #350
• Allow nested fields to be set as identifier by @ismailsimsek in #351
• Lower minimum maven version enforcement to 3.6.3 by @ismailsimsek in #353
• Optimize test logs, decrease verbose logs by @ismailsimsek in #354
• Check if configuration contains event flattening by @ismailsimsek in #352
• Test Field Addition is working, automatic schema expansion by @ismailsimsek in #355
• Test consumer without event flattening, Add flag to control identifier-field creation by @ismailsimsek in #356
• Document new features and improve documentation by @ismailsimsek in #357
• Remove stage repo used for spark 4.0.0-preview1 dependency by @ismailsimsek in #358
• Update README.md by @ismailsimsek in #359
• Update dependabot to weekly interval by @ismailsimsek in #360
• Bump docker/build-push-action from 5 to 6 by @dependabot in #361
• Improve handling delete events and logging message by @ismailsimsek in #366
• Improve documentation, schema change behaviour by @ismailsimsek in #367
• Bump com.squareup.okhttp3:okhttp from 4.11.0 to 4.12.0 by @dependabot in #364
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.5 to 3.3.0 by @dependabot in #363
• Bump version.debezium from 2.7.0.Alpha1 to 2.7.0.Final by @dependabot in #371

Full Changelog: 0.4.0.Final...0.5.0.Beta