Skip to content

Disallow creation of invalid PartitionSpec #12870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
1 of 3 tasks
devinrsmith opened this issue Apr 22, 2025 · 3 comments · May be fixed by #12887
Open
1 of 3 tasks

Disallow creation of invalid PartitionSpec #12870

devinrsmith opened this issue Apr 22, 2025 · 3 comments · May be fixed by #12887
Labels
improvement PR that improves existing functionality

Comments

@devinrsmith
Copy link
Contributor

devinrsmith commented Apr 22, 2025

Bug

The specification says in relationship to partitioning:

The source columns, selected by ids, must be a primitive type and cannot be contained in a map or list, but may be nested in a struct.

The java implementation provides the necessary safety checks around the "primitive type" part of this statement, but does not provide safety checks for the "contained in a map or list" part.

final Schema schema =
    new Schema(
        NestedField.required(
            2, "MyList", Types.ListType.ofRequired(1, Types.IntegerType.get())));
// Does not currently fail, but ideally should
PartitionSpec.builderFor(schema).identity("MyList.element").build();

I've got a branch with changes to TestPartitionSpecValidation.java that further demonstrates the current behavior.

Query engine

None

Willingness to contribute

  • I can contribute this improvement/feature independently
  • I would be willing to contribute this improvement/feature with guidance from the Iceberg community
  • I cannot contribute this improvement/feature at this time
@devinrsmith devinrsmith added the improvement PR that improves existing functionality label Apr 22, 2025
@devinrsmith
Copy link
Contributor Author

I'm not sure if this would be considered a new feature, or a bug.

@Fokko
Copy link
Contributor

Fokko commented Apr 23, 2025

@devinrsmith I think this is a bug, and would be good to get fixed to disallow invalid specs :) Are you interested in providing a fix?

@devinrsmith
Copy link
Contributor Author

I should be able to provide a PR for this, thanks.

devinrsmith added a commit to devinrsmith/iceberg that referenced this issue Apr 24, 2025
devinrsmith added a commit to devinrsmith/iceberg that referenced this issue Apr 24, 2025
devinrsmith added a commit to devinrsmith/iceberg that referenced this issue Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement PR that improves existing functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants