Remote Sensing Labs
Browse all docs

Cloud-Native Geospatial

Cloud-native geospatial is less a technology than a change of default: instead of moving data to wherever the analysis runs, you leave the data in object storage and bring the computation to it, reading only the pieces each task needs. That inversion is what lets a system work with archives far larger than any single machine could download, and it reorganizes every layer above it. The sections below cover the engineering principles behind that pattern — the access model, the tradeoffs between moving data and moving compute, the layout choices that decide performance, the boundaries between storage, catalogs, engines, and applications, and the reliability concerns that come with reading over a network.

The access model

The foundation is boring and important: object storage plus formats and metadata designed for partial reads. An object store holds files as addressable blobs and, crucially, serves byte-range requests — a client can ask for bytes m through n of an object rather than the whole thing. Cloud-native formats are the ones built to exploit that. A cloud-optimized GeoTIFF tiles a raster internally and writes its directory up front, so a reader fetches a header and then only the tiles covering its area of interest. The analogous idea appears for other data types: a small index or metadata block — at the start of the file, at the end (Parquet keeps its metadata in a footer), or in separate small objects (as Zarr does) — plus chunked payload a reader can navigate before downloading. The family includes Zarr for multidimensional arrays, GeoParquet and FlatGeobuf for vector features, COPC for lidar point clouds, and PMTiles for map tiles. On top of the files sits a catalog such as STAC, which turns “petabytes of scenes in a bucket” into something queryable by space, time, and properties. The combination is the whole trick: the catalog tells you which objects you want, and the format lets you read just the parts of those objects you need. The same two-part access model serves both sides of the raster versus vector divide.

Moving compute to data, or data to the application

The central design decision in any cloud-native geospatial system is where the computation happens relative to the bytes, and it is genuinely a tradeoff rather than a rule. Moving compute to the data — running processing in the same cloud region as the storage — keeps transfers short and cheap, which is what makes large batch jobs and server-side tiling affordable; the cost is that you need compute near the data and must express the work in a way that can run there. Moving data to the application — pulling bytes down to a laptop, a browser, or an on-premises system — is simpler and sometimes unavoidable, but it pays the network cost every time and does not scale to archive-sized questions. The reason cloud-native formats matter so much is that they soften this choice: because a reader can range-read just the window it needs, “moving data to the application” can mean a few megabytes instead of a whole scene, which is what makes an interactive map or a notebook over a remote archive feel local. The practical guidance is to match the pattern to the question — interactive viewing and small study-area analysis can stream windows to the client, while full-collection processing belongs next to the storage — and to avoid the worst case, which is repeatedly downloading whole files to compute over a small part of each.

Layout decides performance

Performance in a cloud-native system is set less by CPU than by how the data is laid out for reading, because most of the wall-clock time is network round trips. Tiling and chunking determine how much of a file you must fetch to answer a spatial query: well-chosen chunks mean the bytes for a region are contiguous and reachable in a few reads, while a poor layout turns one logical read into many scattered ones. Overviews — pre-computed downsampled copies — let a zoomed-out view read coarse data instead of every native pixel, so display cost tracks the resolution actually shown rather than the resolution stored. Indexing decides how fast the catalog can turn a spatial-and-temporal question into a list of objects without scanning everything. And metadata layout matters because a reader that can learn an object’s structure — its tiling, its coordinate system and geotransform, its bands and nodata — from a small header can plan a precise range read instead of guessing. The recurring principle is that the number and size of reads a common query requires is the real performance metric; a system is fast when its typical access is a few well-placed range reads, and slow when it is many or when it falls back to whole-file transfers.

System boundaries

A cloud-native geospatial stack is easiest to reason about as four layers with clean responsibilities. Storage holds the bytes and serves range requests; it does not know what the data means. Catalogs describe and locate objects and answer discovery queries; they hold metadata, not pixels. Compute engines read objects and transform them — masking, reprojection, compositing, analysis — ideally close to the storage. Applications — interactive maps, notebooks, APIs, batch jobs — sit on top and consume the results. Keeping these boundaries clean is what keeps the system maintainable: the storage format can change without touching the catalog schema, the catalog can be re-indexed without rewriting the data, and an application can switch between streaming a window for a map and dispatching a server-side job for a large computation without either layer leaking into the other. Open data platforms such as the Microsoft Planetary Computer are opinionated assemblies of some of these layers — object-stored COGs, a STAC catalog over them, and access services on top, designed for compute that the user runs in the same region — which is why understanding the layers transfers even when the provider does not.

Reliability and reproducibility

Reading over a network introduces failure modes that a local pipeline never had, and a serious system plans for them rather than hoping. Transient errors are normal at scale — a range read can time out or a request can be throttled — so clients retry with backoff and treat occasional failure as expected rather than exceptional. Caching cuts both latency and cost by keeping frequently read tiles and metadata close to the consumer, though every cache raises the question of staleness when the underlying data is reprocessed. Signed, expiring access means a workflow that pauses between selecting objects and reading them has to account for credentials that lapse. Underlying all of this is reproducibility: because a re-run query can return different items, a cloud-native workflow records the exact items and parameters it used — the provenance that a catalog like STAC preserves — so a run can be repeated and audited. That is what lets a downstream product such as NDVI monitoring be trusted: not that the network never fails, but that the system is built to fail gracefully and to reproduce its inputs exactly. Judged in practice, a dataset is cloud-native when it can be discovered through a catalog and read in place by range requests; a system built on such data becomes efficient and reproducible when it also processes near the storage and records identifiers for every run.

Sources