CONUS404 Temporal Aggregation#
Create daily averages from hourly data, write to a zarr dataset
import fsspec
import xarray as xr
import hvplot.xarray
import intake
import os
import warnings
from dask.distributed import LocalCluster, Client
warnings.filterwarnings('ignore')
Open dataset from Intake Catalog#
Select
on-premdataset from /caldera if running on prem (Denali/Tallgrass)Select
cloud/osnobject store data if running elsewhere
# open the hytest data intake catalog
hytest_cat = intake.open_catalog("https://raw.githubusercontent.com/hytest-org/hytest/main/dataset_catalog/hytest_intake_catalog.yml")
list(hytest_cat)
['conus404-catalog',
'benchmarks-catalog',
'conus404-drb-eval-tutorial-catalog',
'nhm-v1.0-daymet-catalog',
'nhm-v1.1-c404-bc-catalog',
'nhm-v1.1-gridmet-catalog',
'trends-and-drivers-catalog',
'nhm-prms-v1.1-gridmet-format-testing-catalog',
'nwis-streamflow-usgs-gages-onprem',
'nwis-streamflow-usgs-gages-osn',
'nwm21-streamflow-usgs-gages-onprem',
'nwm21-streamflow-usgs-gages-osn',
'nwm21-streamflow-cloud',
'geofabric_v1_1-zip-osn',
'geofabric_v1_1_POIs_v1_1-osn',
'geofabric_v1_1_TBtoGFv1_POIs-osn',
'geofabric_v1_1_nhru_v1_1-osn',
'geofabric_v1_1_nhru_v1_1_simp-osn',
'geofabric_v1_1_nsegment_v1_1-osn',
'gages2_nndar-osn',
'wbd-zip-osn',
'huc12-geoparquet-osn',
'huc12-gpkg-osn',
'nwm21-scores',
'lcmap-cloud',
'rechunking-tutorial-osn',
'pointsample-tutorial-sites-osn',
'pointsample-tutorial-output-osn']
# open the conus404 sub-catalog
cat = hytest_cat['conus404-catalog']
list(cat)
['conus404-hourly-onprem-hw',
'conus404-hourly-osn',
'conus404-daily-diagnostic-onprem-hw',
'conus404-daily-diagnostic-osn',
'conus404-daily-onprem-hw',
'conus404-daily-osn',
'conus404-monthly-onprem-hw',
'conus404-monthly-osn',
'conus404-hourly-ba-onprem-hw',
'conus404-hourly-ba-osn',
'conus404-daily-ba-onprem-hw',
'conus404-daily-ba-osn',
'conus404-pgw-hourly-onprem-hw',
'conus404-pgw-hourly-osn',
'conus404-pgw-daily-diagnostic-onprem-hw',
'conus404-pgw-daily-diagnostic-osn',
'conus404-pgw-daily-onprem-hw',
'conus404-pgw-daily-osn']
## Select the dataset you want to read into your notebook and preview its metadata
dataset = 'conus404-hourly-osn'
cat[dataset]
<intake_xarray.xzarr.ZarrSource at 0x7f45cb00eff0>
2) Set Up AWS Credentials (Optional)#
This notebook reads data from the OSN pod by default, which is object store data on a high speed internet connection that is free to access from any environment. If you change this notebook to use one of the CONUS404 datasets stored on S3 (options ending in -cloud), you will be pulling data from a requester-pays S3 bucket. This means you have to set up your AWS credentials, else we won’t be able to load the data. Please note that reading the -cloud data from S3 may incur charges if you are reading data outside of the us-west-2 region or running the notebook outside of the cloud altogether. If you would like to access one of the -cloud options, uncomment and run the following code snippet to set up your AWS credentials. You can find more info about this AWS helper function here.
# uncomment the lines below to read in your AWS credentials if you want to access data from a requester-pays bucket (-cloud)
# os.environ['AWS_PROFILE'] = 'default'
# %run ../../environment_set_up/Help_AWS_Credentials.ipynb
Parallelize with Dask#
Some of the steps we will take are aware of parallel clustered compute environments
using dask. We’re going to start a cluster now so that future steps can take advantage
of this ability.
This is an optional step, but speed ups data loading significantly, especially when accessing data from the cloud.
We have documentation on how to start a Dask Cluster in different computing environments here.
%run ../../environment_set_up/Start_Dask_Cluster_Nebari.ipynb
## If this notebook is not being run on Nebari/ESIP, replace the above
## path name with a helper appropriate to your compute environment. Examples:
# %run ../../environment_set_up/Start_Dask_Cluster_Denali.ipynb
# %run ../../environment_set_up/Start_Dask_Cluster_Tallgrass.ipynb
# %run ../../environment_set_up/Start_Dask_Cluster_Desktop.ipynb
# %run ../../environment_set_up/Start_Dask_Cluster_PangeoCHS.ipynb
The 'cluster' object can be used to adjust cluster behavior. i.e. 'cluster.adapt(minimum=10)'
The 'client' object can be used to directly interact with the cluster. i.e. 'client.submit(func)'
The link to view the client dashboard is:
> https://nebari.chs.usgs.gov/gateway/clusters/nebari.e900e9de289b4385af5f2dd1fa64db17/status
Explore the dataset#
ds = cat[dataset].to_dask()
ds
<xarray.Dataset> Size: 222TB
Dimensions: (time: 376945, y: 1015, x: 1367, bottom_top_stag: 51,
bottom_top: 50, soil_layers_stag: 4, x_stag: 1368,
y_stag: 1016, snow_layers_stag: 3, snso_layers_stag: 7)
Coordinates:
lat (y, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lat_u (y, x_stag) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lat_v (y_stag, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lon (y, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lon_u (y, x_stag) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lon_v (y_stag, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
* time (time) datetime64[ns] 3MB 1979-10-01 ... 2022-10-01
* x (x) float64 11kB -2.732e+06 -2.728e+06 ... 2.732e+06
* y (y) float64 8kB -2.028e+06 -2.024e+06 ... 2.028e+06
Dimensions without coordinates: bottom_top_stag, bottom_top, soil_layers_stag,
x_stag, y_stag, snow_layers_stag,
snso_layers_stag
Data variables: (12/153)
ACDEWC (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
ACDRIPR (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
ACDRIPS (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
ACECAN (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
ACEDIR (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
ACETLSM (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
... ...
ZNU (bottom_top) float32 200B dask.array<chunksize=(50,), meta=np.ndarray>
ZNW (bottom_top_stag) float32 204B dask.array<chunksize=(51,), meta=np.ndarray>
ZS (soil_layers_stag) float32 16B dask.array<chunksize=(4,), meta=np.ndarray>
ZSNSO (time, snso_layers_stag, y, x) float32 15TB dask.array<chunksize=(144, 7, 175, 175), meta=np.ndarray>
ZWT (time, y, x) float32 2TB dask.array<chunksize=(144, 175, 175), meta=np.ndarray>
crs int64 8B ...
Attributes: (12/148)
AER_ANGEXP_OPT: 1
AER_ANGEXP_VAL: 1.2999999523162842
AER_AOD550_OPT: 1
AER_AOD550_VAL: 0.11999999731779099
AER_ASY_OPT: 1
AER_ASY_VAL: 0.8999999761581421
... ...
WEST-EAST_PATCH_START_STAG: 1
WEST-EAST_PATCH_START_UNSTAG: 1
W_DAMPING: 1
YSU_TOPDOWN_PBLMIX: 0
history: Tue Mar 29 16:35:22 2022: ncrcat -A -vW ...
history_of_appended_files: Tue Mar 29 16:35:22 2022: Appended file ...ds.T2
<xarray.DataArray 'T2' (time: 376945, y: 1015, x: 1367)> Size: 2TB
dask.array<open_dataset-T2, shape=(376945, 1015, 1367), dtype=float32, chunksize=(144, 175, 175), chunktype=numpy.ndarray>
Coordinates:
lat (y, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lon (y, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
* time (time) datetime64[ns] 3MB 1979-10-01 ... 2022-10-01
* x (x) float64 11kB -2.732e+06 -2.728e+06 ... 2.728e+06 2.732e+06
* y (y) float64 8kB -2.028e+06 -2.024e+06 ... 2.024e+06 2.028e+06
Attributes:
description: TEMP at 2 M
grid_mapping: crs
long_name: Temperature at 2 meters
units: KDaily averages#
Time averages of any type are easy to do with xarray. Here we do 24 hour averages, and set the time offset to 12 hours, so that the time values are in the middle of the averaging period.
Digital Earth Africa has a great Working with Time in Xarray tutorial.
In the example below we just do a few days with a few variables as a quick demo.
%%time
ds_subset = ds[['T2','U10']].sel(time=slice('2017-01-02','2017-01-13'))
CPU times: user 17.4 ms, sys: 166 μs, total: 17.5 ms
Wall time: 17 ms
ds_subset_daily = ds_subset.resample(time="24H", offset="12h", label='right').mean()
ds_subset_daily
<xarray.Dataset> Size: 155MB
Dimensions: (time: 13, y: 1015, x: 1367)
Coordinates:
lat (y, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
lon (y, x) float32 6MB dask.array<chunksize=(175, 175), meta=np.ndarray>
* x (x) float64 11kB -2.732e+06 -2.728e+06 ... 2.728e+06 2.732e+06
* y (y) float64 8kB -2.028e+06 -2.024e+06 ... 2.024e+06 2.028e+06
* time (time) datetime64[ns] 104B 2017-01-02T12:00:00 ... 2017-01-14T12...
Data variables:
T2 (time, y, x) float32 72MB dask.array<chunksize=(13, 175, 175), meta=np.ndarray>
U10 (time, y, x) float32 72MB dask.array<chunksize=(13, 175, 175), meta=np.ndarray>
Attributes: (12/148)
AER_ANGEXP_OPT: 1
AER_ANGEXP_VAL: 1.2999999523162842
AER_AOD550_OPT: 1
AER_AOD550_VAL: 0.11999999731779099
AER_ASY_OPT: 1
AER_ASY_VAL: 0.8999999761581421
... ...
WEST-EAST_PATCH_START_STAG: 1
WEST-EAST_PATCH_START_UNSTAG: 1
W_DAMPING: 1
YSU_TOPDOWN_PBLMIX: 0
history: Tue Mar 29 16:35:22 2022: ncrcat -A -vW ...
history_of_appended_files: Tue Mar 29 16:35:22 2022: Appended file ...ds_subset_daily.hvplot.quadmesh(x='lon', y='lat', rasterize=True,
geo=True, tiles='OSM', alpha=0.7, cmap='turbo')
WARNING:param.dynamic_operation: Callable raised "FutureCancelledError()".
Invoked as dynamic_operation(numpy.datetime64('2017-01-02T12:00:00.000000000'), height=400, scale=1.0, width=400, x_range=None, y_range=None)
WARNING:param.dynamic_mul: Callable raised "FutureCancelledError()".
Invoked as dynamic_mul(numpy.datetime64('2017-01-02T12:00:00.000000000'))
WARNING:param.dynamic_operation: Callable raised "FutureCancelledError()".
Invoked as dynamic_operation(numpy.datetime64('2017-01-02T12:00:00.000000000'))
---------------------------------------------------------------------------
FutureCancelledError Traceback (most recent call last)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/IPython/core/formatters.py:1036, in MimeBundleFormatter.__call__(self, obj, include, exclude)
1033 method = get_real_method(obj, self.print_method)
1035 if method is not None:
-> 1036 return method(include=include, exclude=exclude)
1037 return None
1038 else:
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/dimension.py:1277, in Dimensioned._repr_mimebundle_(self, include, exclude)
1270 def _repr_mimebundle_(self, include=None, exclude=None):
1271 """
1272 Resolves the class hierarchy for the class rendering the
1273 object using any display hooks registered on Store.display
1274 hooks. The output of all registered display_hooks is then
1275 combined and returned.
1276 """
-> 1277 return Store.render(self)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/options.py:1423, in Store.render(cls, obj)
1421 data, metadata = {}, {}
1422 for hook in hooks:
-> 1423 ret = hook(obj)
1424 if ret is None:
1425 continue
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/ipython/display_hooks.py:287, in pprint_display(obj)
285 if not ip.display_formatter.formatters['text/plain'].pprint:
286 return None
--> 287 return display(obj, raw_output=True)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/ipython/display_hooks.py:261, in display(obj, raw_output, **kwargs)
259 elif isinstance(obj, (HoloMap, DynamicMap)):
260 with option_state(obj):
--> 261 output = map_display(obj)
262 elif isinstance(obj, Plot):
263 output = render(obj)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/ipython/display_hooks.py:149, in display_hook.<locals>.wrapped(element)
147 try:
148 max_frames = OutputSettings.options['max_frames']
--> 149 mimebundle = fn(element, max_frames=max_frames)
150 if mimebundle is None:
151 return {}, {}
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/ipython/display_hooks.py:209, in map_display(vmap, max_frames)
206 max_frame_warning(max_frames)
207 return None
--> 209 return render(vmap)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/ipython/display_hooks.py:76, in render(obj, **kwargs)
73 if renderer.fig == 'pdf':
74 renderer = renderer.instance(fig='png')
---> 76 return renderer.components(obj, **kwargs)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/plotting/renderer.py:396, in Renderer.components(self, obj, fmt, comm, **kwargs)
394 embed = (not (dynamic or streams or self.widget_mode == 'live') or config.embed)
395 if embed or config.comms == 'default':
--> 396 return self._render_panel(plot, embed, comm)
397 return self._render_ipywidget(plot)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/plotting/renderer.py:403, in Renderer._render_panel(self, plot, embed, comm)
401 doc = Document()
402 with config.set(embed=embed):
--> 403 model = plot.layout._render_model(doc, comm)
404 if embed:
405 return render_model(model, comm)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/viewable.py:768, in Viewable._render_model(self, doc, comm)
766 if comm is None:
767 comm = state._comm_manager.get_server_comm()
--> 768 model = self.get_root(doc, comm)
770 if self._design and self._design.theme.bokeh_theme:
771 doc.theme = self._design.theme.bokeh_theme
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/layout/base.py:329, in Panel.get_root(self, doc, comm, preprocess)
325 def get_root(
326 self, doc: Document | None = None, comm: Comm | None = None,
327 preprocess: bool = True
328 ) -> Model:
--> 329 root = super().get_root(doc, comm, preprocess)
330 # ALERT: Find a better way to handle this
331 if hasattr(root, 'styles') and 'overflow-x' in root.styles:
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/viewable.py:698, in Renderable.get_root(self, doc, comm, preprocess)
696 wrapper = self._design._wrapper(self)
697 if wrapper is self:
--> 698 root = self._get_model(doc, comm=comm)
699 if preprocess:
700 self._preprocess(root)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/layout/base.py:313, in Panel._get_model(self, doc, root, parent, comm)
311 root = root or model
312 self._models[root.ref['id']] = (model, parent)
--> 313 objects, _ = self._get_objects(model, [], doc, root, comm)
314 props = self._get_properties(doc)
315 props[self._property_mapping['objects']] = objects
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/layout/base.py:295, in Panel._get_objects(self, model, old_objects, doc, root, comm)
293 else:
294 try:
--> 295 child = pane._get_model(doc, root, model, comm)
296 except RerenderError as e:
297 if e.layout is not None and e.layout is not self:
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/pane/holoviews.py:437, in HoloViews._get_model(self, doc, root, parent, comm)
435 plot = self.object
436 else:
--> 437 plot = self._render(doc, comm, root)
439 plot.pane = self
440 backend = plot.renderer.backend
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/panel/pane/holoviews.py:531, in HoloViews._render(self, doc, comm, root)
528 if comm:
529 kwargs['comm'] = comm
--> 531 return renderer.get_plot(self.object, **kwargs)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/plotting/bokeh/renderer.py:68, in BokehRenderer.get_plot(self_or_cls, obj, doc, renderer, **kwargs)
61 @bothmethod
62 def get_plot(self_or_cls, obj, doc=None, renderer=None, **kwargs):
63 """
64 Given a HoloViews Viewable return a corresponding plot instance.
65 Allows supplying a document attach the plot to, useful when
66 combining the bokeh model with another plot.
67 """
---> 68 plot = super().get_plot(obj, doc, renderer, **kwargs)
69 if plot.document is None:
70 plot.document = Document() if self_or_cls.notebook_context else curdoc()
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/plotting/renderer.py:216, in Renderer.get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs)
213 raise SkipRendering(msg.format(dims=dims))
215 # Initialize DynamicMaps with first data item
--> 216 initialize_dynamic(obj)
218 if not renderer:
219 renderer = self_or_cls
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/plotting/util.py:277, in initialize_dynamic(obj)
275 continue
276 if not len(dmap):
--> 277 dmap[dmap._initial_key()]
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:1216, in DynamicMap.__getitem__(self, key)
1214 # Not a cross product and nothing cached so compute element.
1215 if cache is not None: return cache
-> 1216 val = self._execute_callback(*tuple_key)
1217 if data_slice:
1218 val = self._dataslice(val, data_slice)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:983, in DynamicMap._execute_callback(self, *args)
980 kwargs['_memoization_hash_'] = hash_items
982 with dynamicmap_memoization(self.callback, self.streams):
--> 983 retval = self.callback(*args, **kwargs)
984 return self._style(retval)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:581, in Callable.__call__(self, *args, **kwargs)
578 args, kwargs = (), dict(pos_kwargs, **kwargs)
580 try:
--> 581 ret = self.callable(*args, **kwargs)
582 except KeyError:
583 # KeyError is caught separately because it is used to signal
584 # invalid keys on DynamicMap and should not warn
585 raise
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/util/__init__.py:1052, in Dynamic._dynamic_operation.<locals>.dynamic_operation(*key, **kwargs)
1051 def dynamic_operation(*key, **kwargs):
-> 1052 key, obj = resolve(key, kwargs)
1053 return apply(obj, *key, **kwargs)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/util/__init__.py:1041, in Dynamic._dynamic_operation.<locals>.resolve(key, kwargs)
1039 elif isinstance(map_obj, DynamicMap) and map_obj._posarg_keys and not key:
1040 key = tuple(kwargs[k] for k in map_obj._posarg_keys)
-> 1041 return key, map_obj[key]
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:1216, in DynamicMap.__getitem__(self, key)
1214 # Not a cross product and nothing cached so compute element.
1215 if cache is not None: return cache
-> 1216 val = self._execute_callback(*tuple_key)
1217 if data_slice:
1218 val = self._dataslice(val, data_slice)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:983, in DynamicMap._execute_callback(self, *args)
980 kwargs['_memoization_hash_'] = hash_items
982 with dynamicmap_memoization(self.callback, self.streams):
--> 983 retval = self.callback(*args, **kwargs)
984 return self._style(retval)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:581, in Callable.__call__(self, *args, **kwargs)
578 args, kwargs = (), dict(pos_kwargs, **kwargs)
580 try:
--> 581 ret = self.callable(*args, **kwargs)
582 except KeyError:
583 # KeyError is caught separately because it is used to signal
584 # invalid keys on DynamicMap and should not warn
585 raise
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/overlay.py:34, in Overlayable.__mul__.<locals>.dynamic_mul(*args, **kwargs)
33 def dynamic_mul(*args, **kwargs):
---> 34 element = other[args]
35 return self * element
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:1216, in DynamicMap.__getitem__(self, key)
1214 # Not a cross product and nothing cached so compute element.
1215 if cache is not None: return cache
-> 1216 val = self._execute_callback(*tuple_key)
1217 if data_slice:
1218 val = self._dataslice(val, data_slice)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:983, in DynamicMap._execute_callback(self, *args)
980 kwargs['_memoization_hash_'] = hash_items
982 with dynamicmap_memoization(self.callback, self.streams):
--> 983 retval = self.callback(*args, **kwargs)
984 return self._style(retval)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/spaces.py:581, in Callable.__call__(self, *args, **kwargs)
578 args, kwargs = (), dict(pos_kwargs, **kwargs)
580 try:
--> 581 ret = self.callable(*args, **kwargs)
582 except KeyError:
583 # KeyError is caught separately because it is used to signal
584 # invalid keys on DynamicMap and should not warn
585 raise
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/util/__init__.py:1053, in Dynamic._dynamic_operation.<locals>.dynamic_operation(*key, **kwargs)
1051 def dynamic_operation(*key, **kwargs):
1052 key, obj = resolve(key, kwargs)
-> 1053 return apply(obj, *key, **kwargs)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/util/__init__.py:1045, in Dynamic._dynamic_operation.<locals>.apply(element, *key, **kwargs)
1043 def apply(element, *key, **kwargs):
1044 kwargs = dict(util.resolve_dependent_kwargs(self.p.kwargs), **kwargs)
-> 1045 processed = self._process(element, key, kwargs)
1046 if (self.p.link_dataset and isinstance(element, Dataset) and
1047 isinstance(processed, Dataset) and processed._dataset is None):
1048 processed._dataset = element.dataset
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/util/__init__.py:1027, in Dynamic._process(self, element, key, kwargs)
1025 elif isinstance(self.p.operation, Operation):
1026 kwargs = {k: v for k, v in kwargs.items() if k in self.p.operation.param}
-> 1027 return self.p.operation.process_element(element, key, **kwargs)
1028 else:
1029 return self.p.operation(element, **kwargs)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/operation.py:192, in Operation.process_element(self, element, key, **params)
189 else:
190 self.p = param.ParamOverrides(self, params,
191 allow_extra_keywords=self._allow_extra_keywords)
--> 192 return self._apply(element, key)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/operation.py:141, in Operation._apply(self, element, key)
139 if not in_method:
140 element._in_method = True
--> 141 ret = self._process(element, key)
142 if hasattr(element, '_in_method') and not in_method:
143 element._in_method = in_method
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/operation/datashader.py:1546, in rasterize._process(self, element, key)
1543 op = transform.instance(**{k:v for k,v in extended_kws.items()
1544 if k in transform.param})
1545 op._precomputed = self._precomputed
-> 1546 element = element.map(op, predicate)
1547 self._precomputed = op._precomputed
1549 unused_params = list(all_supplied_kws - all_allowed_kws)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/data/__init__.py:195, in PipelineMeta.pipelined.<locals>.pipelined_fn(*args, **kwargs)
192 inst._in_method = True
194 try:
--> 195 result = method_fn(*args, **kwargs)
196 if PipelineMeta.disable:
197 return result
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/data/__init__.py:1212, in Dataset.map(self, *args, **kwargs)
1210 @wraps(LabelledData.map)
1211 def map(self, *args, **kwargs):
-> 1212 return super().map(*args, **kwargs)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/dimension.py:697, in LabelledData.map(self, map_fn, specs, clone)
695 return deep_mapped
696 else:
--> 697 return map_fn(self) if applies else self
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/operation.py:212, in Operation.__call__(self, element, **kwargs)
208 return element.clone([(k, self._apply(el, key=k))
209 for k, el in element.items()])
210 elif ((self._per_element and isinstance(element, Element)) or
211 (not self._per_element and isinstance(element, ViewableElement))):
--> 212 return self._apply(element)
213 elif 'streams' not in kwargs:
214 kwargs['streams'] = self.p.streams
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/core/operation.py:141, in Operation._apply(self, element, key)
139 if not in_method:
140 element._in_method = True
--> 141 ret = self._process(element, key)
142 if hasattr(element, '_in_method') and not in_method:
143 element._in_method = in_method
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/holoviews/operation/datashader.py:1144, in quadmesh_rasterize._process(self, element, key)
1140 cvs = ds.Canvas(plot_width=width, plot_height=height,
1141 x_range=x_range, y_range=y_range)
1143 vdim = getattr(agg_fn, 'column', element.vdims[0].name)
-> 1144 agg = cvs.quadmesh(data[vdim], x.name, y.name, agg_fn)
1145 xdim, ydim = list(agg.dims)[:2][::-1]
1146 if xtype == "datetime":
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/datashader/core.py:885, in Canvas.quadmesh(self, source, x, y, agg)
883 elif xarr.ndim == 2:
884 glyph = QuadMeshCurvilinear(x, y, name)
--> 885 return bypixel(source, self, glyph, agg)
886 else:
887 raise ValueError("""\
888 x- and y-coordinate arrays must have 1 or 2 dimensions.
889 Received arrays with dimensions: {dims}""".format(
890 dims=list(xarr.dims)))
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/datashader/core.py:1351, in bypixel(source, canvas, glyph, agg, antialias)
1349 with warnings.catch_warnings():
1350 warnings.filterwarnings('ignore', r'All-NaN (slice|axis) encountered')
-> 1351 return bypixel.pipeline(source, schema, canvas, glyph, agg, antialias=antialias)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/datashader/utils.py:118, in Dispatcher.__call__(self, head, *rest, **kwargs)
116 typ = type(head)
117 if typ in lk:
--> 118 return lk[typ](head, *rest, **kwargs)
119 for cls in getmro(typ)[1:]:
120 if cls in lk:
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/datashader/data_libraries/xarray.py:32, in xarray_pipeline(xr_ds, schema, canvas, glyph, summary, antialias)
30 else:
31 from datashader.data_libraries.dask_xarray import dask_xarray_pipeline
---> 32 return dask_xarray_pipeline(
33 glyph, xr_ds, schema, canvas, summary, antialias=antialias, cuda=cuda)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/datashader/data_libraries/dask_xarray.py:29, in dask_xarray_pipeline(glyph, xr_ds, schema, canvas, summary, antialias, cuda)
25 graph = xr_ds.__dask_graph__()
27 dsk.update(optimize(graph, keys))
---> 29 return scheduler(dsk, name)
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/distributed/client.py:3493, in Client.get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
3491 should_rejoin = False
3492 try:
-> 3493 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
3494 finally:
3495 for f in futures.values():
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/distributed/client.py:2566, in Client.gather(self, futures, errors, direct, asynchronous)
2563 local_worker = None
2565 with shorten_traceback():
-> 2566 return self.sync(
2567 self._gather,
2568 futures,
2569 errors=errors,
2570 direct=direct,
2571 local_worker=local_worker,
2572 asynchronous=asynchronous,
2573 )
File /home/conda/hytest/785d3670-1747942806-104-pangeo/lib/python3.12/site-packages/distributed/client.py:2427, in Client._gather(self, futures, errors, direct, local_worker)
2425 exception = st.exception
2426 traceback = st.traceback
-> 2427 raise exception.with_traceback(traceback)
2428 if errors == "skip":
2429 bad_keys.add(key)
FutureCancelledError: 9e6cab1f89d254b7a440aa1093848c0d cancelled for reason: scheduler-connection-lost.
Client lost the connection to the scheduler. Please check your connection and re-run your work.
:DynamicMap [time]
Write daily values as a Zarr dataset (to onprem or cloud)#
You will need to to turn the following cell from raw to code and update the filepaths in order to save out your data.
Shutdown cluster#
client.close(); cluster.shutdown()