MODIS time series extraction hitting rate limit?

[ethanshafron]

Hi folks!

I am working on a project that involves getting moving window sums of MODIS GPP for around 1500 small bounding boxes across the globe. So far I have been getting this data by going through the STAC endpoint for a given bounding box, and iterating through each item returned by an item search. This is quite slow and difficult to scale up without hitting rate limits extremely quickly. I have access to a computing cluster as well, and have been trying to parallelize this process, but it seems like the bottleneck for that is still rate limits. Currently I'm going through the STAC endpoint. I have recently been looking more into using GeoParquet assets to bulk-extract data, but haven't found any concrete examples online of what that might look like. Any suggestions or guidance for how to scale this up would be greatly appreciated!

So far I've really enjoyed working with the platform, and truly appreciate all the open-source contributions you all have made!

[TomAugspurger]

Thanks for opening up this discussion. Do you happen to have a stacktrace available that shows the rate limiting?

As for getting around that, I think you have a couple options:

1. Reduce the number of requests to the STAC API by merging bounding boxes into a larger MultiPolygon. https://github.com/microsoft/PlanetaryComputerExamples/blob/main/tutorials/mosaiks.ipynb and https://nbviewer.org/gist/TomAugspurger/ceadc4b2f8b7e4263ff172ee1ea76dbb go into this more. See Add request splitting feature stac-utils/pystac-client#77 for background.
2. Use the geoparquet snapshots. https://planetarycomputer.microsoft.com/docs/quickstarts/stac-geoparquet/ has an example.

Let me know if you have any follow-up questions.