ERA5 timeseries chunking #502
windy-mark-2
started this conversation in
Data request
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I am using the Microsoft Planetary Computer ERA5 collection to retrieve a very small spatial domain, such as a single grid point or a 3×3 grid, over several decades of hourly data. The current Zarr layout appears optimized for broader spatial access within individual monthly assets. For point-based time-series workloads, the client must open hundreds of monthly assets and read chunks containing substantially more spatial data than requested. As a result, retrieving one location over a long period can require many requests and take considerably longer than the small output size would suggest.
ECMWF now provides an alternative ERA5 data source with a layout chunked by grid node, making long time-series retrieval for one or a few locations much faster. Would Microsoft consider publishing an additional Planetary Computer representation optimized for this access pattern, with spatial chunks corresponding to individual grid nodes, or similarly small spatial units, and longer chunks along the time dimension?
This could complement rather than replace the existing layout. The current organization may remain preferable for regional or map-oriented analysis, while a point-optimized representation would support weather stations, renewable-energy sites, asset locations, and other applications that repeatedly retrieve small domains over long periods. It would also reduce unnecessary data transfer and the number of monthly asset and metadata requests required for these workloads.
See link:
https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels-timeseries?tab=analysis_ready_data
All reactions