Supports Zip format in import SQL

[lordofthejars]

Description

Sometimes the data file (import.sql) you want to import for testing (specially in AI applications where vectors are huge), is so big that you cannot push directly in Git (yes you can always use LFS), but if you zip the SQL file then the size is reduced by 2/3, so make it more manageable.

Implementation ideas

We could add support for import.sql and import.sql.zip and when a zip Quarkus uncompresses the file, or provide an event triggered before the import.sql logic so we can code the unzipping part. I know that this is not that easy as the file must be in classpath and so on.

Well, I just wanted to note a limitation I found today, that's all :)

[quarkus-bot]

/cc @gsmet (hibernate-orm)

[yrodiere]

Hey.

This does indeed seem useful.

FTR, this is the relevant code in Hibernate ORM: https://github.com/hibernate/hibernate-orm/blob/018b8eeda3627e114ec25bd48407ccb9c47564ce/hibernate-core/src/main/java/org/hibernate/tool/schema/internal/Helper.java#L59-L99

As you can see, it expects either strings pointing to the scripts (either as classpath resource names or URLs), or a Reader instance.

The Reader isn't an option IMO, because we'd be limited to a single script, and because resource closing / reopening might be an issue.

So I think the only way to make this work in Quarkus, without a fix in Hibernate ORM, would be to have some code at runtime init, before Hibernate ORM boots, to parse the property hibernate.hbm2ddl.import_files, detect ZIP files, extract them, and replace their entry in hibernate.hbm2ddl.import_files with the URL to the extracted file (file://...).
I think the best place to do that would be there:

quarkus/extensions/hibernate-orm/runtime/src/main/java/io/quarkus/hibernate/orm/runtime/FastBootHibernatePersistenceProvider.java

Lines 200 to 205 in 6ac02d1

	private RuntimeSettings buildRuntimeSettings(String persistenceUnitName, RecordedState recordedState,
	HibernateOrmRuntimeConfigPersistenceUnit persistenceUnitConfig) {
	final BuildTimeSettings buildTimeSettings = recordedState.getBuildTimeSettings();
	final IntegrationSettings integrationSettings = recordedState.getIntegrationSettings();
	Builder runtimeSettingsBuilder = new Builder(buildTimeSettings, integrationSettings);

Note you'd need to do the same for Hibernate Reactive, in io.quarkus.hibernate.reactive.runtime.FastBootHibernateReactivePersistenceProvider.

If anyone wants to have a look, this could be an interesting first contribution to Quarkus :)

EDIT: Alternatively, this can be contributed as a first-class feature in Hibernate ORM, e.g. by implementing the ZIP extraction in org.hibernate.tool.schema.internal.Helper. But then I think the cleanup of extracted ZIPs would be more challenging to implement (org.hibernate.tool.schema.internal.Helper doesn't have any lifecycle hooks it can use).

[alihmzyv]

Hi, there. @yrodiere could you please assign me to this? This will be my first ever open source contribution if possible :)
Thanks in advance

[yrodiere]

Hi,

Sure, done. Please keep us updated so I know you're working on this: we're getting lots of requests to be assigned that aren't followed by a PR, so I tend to unassign issues after a while.

I hope the description above is enough, but if you have specific issues, feel free to ask here.

Cheers.

[alihmzyv]

Okay, thanks

[kto-viktor]

@alihmzyv maybe that will helps, i have this very very draft which working for one import file and non-reactive case only: kto-viktor@3be1b7d

[alihmzyv]

@kto-viktor thanks

[alihmzyv]

I started today working on this, @yrodiere just so you know

[alihmzyv]

Hi there. I am having difficulty building the hibernate-orm extension. I mean that build went as success but none of the classes outside io.quarkus.hibernate.orm is resolved. Do I have to build the whole project? It takes too long as I see. Could someone help me with this? I have used this command to build the extension:
./mvnw -f extensions/hibernate-orm clean install

Used this as well, but did not help:
MAVEN_OPTS="-Xmx4g" ./mvnw -Dquickly