Create Zarr in parallel by appending to an array which size and indices are not known: valid use case? #2140
Unanswered
NicolasM-Forsk
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hello
I'd like to create a compact zarr containing an item list which is not determined by advance.
The zarr contains an (item_id, y, x) shape, chunked by item_id (ex 8), resulting in 8 raster items being stored in 1 file of the item_data array.
I want to append raster item 8 by 8 from different celery workers with compression enabled, and I see that item_id/.zarray and item_id/0 are updated each time.
I wonder if I can lock only these files, and if this is a valid use case for Zarr V2 or Zarr V3.
Beta Was this translation helpful? Give feedback.
All reactions