Introduction to Chunking

In this first series of notebooks, we will go over basic introductory topics associated with chunking. As you will soon learn, “chunking” is an an essential part of the data preparation workflow, particularly for large datasets. The key concepts you should understand have after this series include:

• HyTEST Project
• Environment Set-Up
  • Compute Environments
    • General QuickStart
    • Cloud: Nebari Quick-Start
    • HPC Server: Open OnDemand Quick-Start
    • HPC Server: Jupyter Forward Quick-Start
    • HPC Server: Start Script Quick-Start
  • AWS Credentials
  • Starting a Dask Cluster
    • Nebari Dask Cluster
    • Denali HPC Dask Cluster
    • Hovenweep/Tallgrass HPC Dask Cluster
    • Local Desktop Dask Cluster
• Getting Started
  • Essential Reading
    • Foundations
  • Data Sources
    • Data/APIs
    • Data/Cloud Storage
  • Parallelism with Dask
• Dataset Access
  • Data Catalogs
    • Data Storage Locations
    • Water Mission Area STAC Catalog
    • HyTEST Data Catalog (Intake)
    • HyTEST Intake Sub-Catalogs
  • CONUS404 Products Data Access
    • CONUS404 Zarr Changelog
    • Explore CONUS404 Dataset
• Zarr Creation
  • Data Chunking Tutorial Overview
  • Introduction to Chunking
    • Why (re)Chunk Data?
    • How to Examine a Stored Dataset’s Chunk Shape
    • Basics of Chunk Shape and Size
    • Writing Chunked Files
    • Rechunking
  • Advanced Topics in Chunking
    • Rechunking Larger Datasets with Dask
    • Generating a Virtual Zarr Store
    • Adding Coordinate Reference Systems (CRS) to Zarr Datasets
    • Choosing an Optimal Chunk Size and Shape
  • Chunking References
    • Glossary
    • Additional Resources
• Dataset Processing
  • CONUS404 Temporal Aggregation
  • Spatial Aggregation
    • gdptools CONUS404 Spatial Aggregation over DRB-extent HUC12s
    • gdptools CONUS404 Spatial Aggregation over CONUS-extent GFv1.1
    • gdptools CONUS404 Spatial Aggregation over CONUS-extent HUC12s
    • Pangeo CONUS404 Spatial Aggregation over DRB-extent HUC12s
    • gdptools-Pangeo Method Comparison for CONUS404 Spatial Aggregation
  • CONUS404 Site Data Selection
  • CONUS404 Regridding (Curvilinear => Rectilinear)
• Model Evaluation
  • Evaluation Metrics
    • D-Score Suite (v1) Benchmark
    • D-Score Suite (v1) Benchmark -- Usage Examples
    • Standard Suite (v1) Metrics
    • Standard Suite (v1) Metrics -- Usage Examples
  • Streamflow Model Evaluation
    • Streamflow Eval :: Data Preparation
    • Streamflow Eval :: StdSuite Analysis
    • Steamflow Eval :: DScore Analysis
    • Streamflow Eval :: Visualization
• Back Matter
  • Contributing
  • License

Buckle up... we will get up to speed fast.