A quarter-century of Landsat over 1,000,000 km² exceeds any single machine's memory. The naïve pipeline dies at the boundary — the engineering is in never loading it all at once.
The pipeline is built cloud-native rather than download-first: scenes stay as Cloud-Optimized GeoTIFFs on S3 and are discovered through a STAC catalog, so a query returns metadata and byte ranges instead of terabytes. Nothing downstream ever opens a raw scene file — every stage after S3 talks to a lazy Xarray datacube, never to paths.
That makes the two hard parts tractable. Dask chunking keeps memory bounded while foundation-model inference is distributed across workers, and the resulting embeddings are serialised to GeoParquet — a columnar store that a downstream causal model can query by geometry and time without rerunning the encoder.
FTP makes you know the filename in advance. STAC lets you query by geometry and date — the catalog does the indexing, not a folder-naming convention.
Hand-rolled GDAL loops force you to track tiling and reprojection yourself. Xarray gives labeled, lazy N-D arrays — Dask handles the chunking underneath.
A hand-picked index like NDVI throws away everything it wasn't designed to see. A foundation-model embedding keeps the full spectral-temporal signal for later causal work.
# 1. SpatioTemporal Discovery catalog = pystac_client.Client.open("https://landsatlook.usgs.gov/stac-browser") items = catalog.search(bbox=[...], datetime="2000/2025", collections=["landsat-c2-l2"]).item_collection() # 2. Lazy "data cube" construction # No data is read until .compute() is called cube = stackstac.stack(items, epsg=4326, resolution=30, chunksize=2048) # 3. Distributed inference (foundation model) embeddings = cube.groupby("time").map(lambda x: fm_encoder.predict(x)) # 4. Serialize to columnar feature store embeddings.to_geoparquet("planetary_archive.parquet", compression="snappy")
Together, STAC discovery, lazy cube construction, and GeoParquet serialization mean nothing is read or computed until it is actually needed — so a re-run only touches the chunks that changed. The same task graph runs identically on a laptop or on TACC, which makes the archive reproducible rather than something I have to babysit.