| Title: | Load 'Overture' Datasets as 'dbplyr' and 'sf'-Ready Data Frames |
|---|---|
| Description: | An integrated R interface to the 'Overture' API (<https://docs.overturemaps.org/>). Allows R users to return 'Overture' data as 'dbplyr' data frames or materialized 'sf' spatial data frames. |
| Authors: | Arthur Gailes [aut, cre, cph] (ORCID: <https://orcid.org/0009-0006-8176-8653>) |
| Maintainer: | Arthur Gailes <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.3.2 |
| Built: | 2026-10-16 17:43:53 UTC |
| Source: | https://github.com/arthurgailes/overturer |
Adds the overture_call class to a tbl_sql object. open_curtain()
does this for you; call it directly on a lazy table of Overture data you
built yourself, so that collect() returns sf and record_overture()
knows the type and theme of the data.
as_overture(x, type, theme = get_theme_from_type(type), release = NULL)as_overture(x, type, theme = get_theme_from_type(type), release = NULL)
x |
A tbl_sql object representing an Overture Maps dataset. |
type |
A string specifying the type of overture dataset to read.
Setting to "*" or |
theme |
Inferred from type by default. Must be set if type is "*" or
|
release |
The Overture release the data came from, such as
|
A tbl_sql object with the additional class overture_call and an
overture_playbill attribute: a list with type, theme and release.
# The open_curtain() function already uses as_overture() internally, # but you can also use it directly: conn <- stage_conn() division <- open_curtain("division", tablename = "test") class(division) # views division2 <- tbl(conn, "test") division2 <- as_overture(division2, "division") strike_stage(conn)# The open_curtain() function already uses as_overture() internally, # but you can also use it directly: conn <- stage_conn() division <- open_curtain("division", tablename = "test") class(division) # views division2 <- tbl(conn, "test") division2 <- as_overture(division2, "division") strike_stage(conn)
overtureR caches what it reads from Overture's STAC catalog (the list of
types in a release, and the bounding box of every Parquet file) in memory
and on disk under tools::R_user_dir("overtureR", "cache"). Releases never
change, so the cache never goes stale, but you can clear it here. Set
options(overturer_cache = FALSE) to keep the cache in memory only, or
options(overturer_cache_dir = ) to move it.
clear_overture_cache()clear_overture_cache()
The cache directory, invisibly.
## Not run: clear_overture_cache() ## End(Not run)## Not run: clear_overture_cache() ## End(Not run)
Collects a lazy dbplyr view and materializes it as an
in-memory sf table. collect_sf is a deprecated alias.
## S3 method for class 'overture_call' collect(x, ..., geom_col = "geometry", crs = 4326) collect_sf(...)## S3 method for class 'overture_call' collect(x, ..., geom_col = "geometry", crs = 4326) collect_sf(...)
x |
A lazy data frame backed by a database query. |
... |
Further arguments passed to |
geom_col |
The name of the geometry column. Will auto-detect names matching 'geom'. |
crs |
The coordinate reference system to use for the geometries, specified by its EPSG code. The default is 4326 (WGS 84). |
The geometry column is read back as well-known binary and converted with
sf::st_as_sfc(). If the column is already binary (for example, after a
mutate(geometry = ST_AsWKB(geometry))), it is used as is. If it is
neither DuckDB GEOMETRY nor binary, the result is returned as a plain
data frame.
An 'sf' object with the dataset converted to spatial features.
bbox <- c(xmin = -120.5, ymin = 35.5, xmax = -120.0, ymax = 36.0) lazy_tbl <- open_curtain("building", bbox) collect(lazy_tbl)bbox <- c(xmin = -120.5, ymin = 35.5, xmax = -120.0, ymax = 36.0) lazy_tbl <- open_curtain("building", bbox) collect(lazy_tbl)
Check duckdb extension and config settings
config_extensions(conn)config_extensions(conn)
conn |
A connection to a duckdb database. |
open_curtain() needs a release date/version to build its S3 path (e.g.
"2026-08-19.0"). Overture cuts a new release roughly monthly, so hardcoding
one quickly goes stale. This queries Overture's STAC catalog
(https://stac.overturemaps.org/catalog.json) for its current latest
release, so callers don't have to track releases themselves or wait on a
package update. The result is cached for the session.
latest_overture_release(conn = NULL, refresh = FALSE)latest_overture_release(conn = NULL, refresh = FALSE)
conn |
A duckdb connection. Uses the cached session connection by default. |
refresh |
If |
If the catalog can't be reached, the newest release in the package's local
catalog cache is used with a warning (Overture removes releases after a few
months, so it may itself be gone). With no cache, the call fails with an
error; pass base_url to open_curtain() to work offline or from a local
copy.
A string identifying the latest release, e.g. "2026-08-19.0".
latest_overture_release()latest_overture_release()
Fetches overture data from AWS.
If a spatial filter is provided, it applies spatial filtering to only
include records within that area. The core code is copied from duckdbfs,
which deserves all credit for the implementation
open_curtain( type, spatial_filter = NULL, theme = get_theme_from_type(type), conn = NULL, as_sf = FALSE, mode = "view", tablename = NULL, read_opts = list(), predicate = "intersects", release = NULL, base_url = NULL, bbox = NULL )open_curtain( type, spatial_filter = NULL, theme = get_theme_from_type(type), conn = NULL, as_sf = FALSE, mode = "view", tablename = NULL, read_opts = list(), predicate = "intersects", release = NULL, base_url = NULL, bbox = NULL )
type |
A string specifying the type of overture dataset to read.
Setting to "*" or |
spatial_filter |
An object to spatially filter the result: a named
numeric vector or |
theme |
Inferred from type by default. Must be set if type is "*" or
|
conn |
A connection to a duckdb database. |
as_sf |
If TRUE, return an sf dataframe |
mode |
Either "view" (default) or "table". If "table", will download the dataset into memory. |
tablename |
The name of the table to create in the database. |
read_opts |
A named list of key-value pairs passed to
DuckDB's read_parquet.
|
predicate |
How a feature must relate to |
release |
An Overture release, such as |
base_url |
Read from a different mirror, such as a local directory from
|
bbox |
alias for |
When spatial_filter is set and base_url points at an Overture release
or at a directory written by record_overture(), open_curtain() reads
only the Parquet files whose bounding box touches the filter, using the
file list in Overture's STAC catalog or the local copy's manifest (see
overture_types() and clear_overture_cache()). This turns a cold query
over hundreds of files into one over a handful. Set
options(overturer_prune = FALSE) to always read the whole partition.
To pin every query in a session to one release, set
options(overturer_release = "2026-08-19.0").
An dbplyr lazy dataframe, or an sf dataframe if as_sf is TRUE
bbox <- c(xmin = -120.5, ymin = 35.5, xmax = -120.0, ymax = 36.0) open_curtain("building", bbox) # pin a release so the script returns the same rows next month open_curtain("building", bbox, release = "2026-08-19.0") # only buildings entirely inside the box open_curtain("building", bbox, predicate = "within")bbox <- c(xmin = -120.5, ymin = 35.5, xmax = -120.0, ymax = 36.0) open_curtain("building", bbox) # pin a release so the script returns the same rows next month open_curtain("building", bbox, release = "2026-08-19.0") # only buildings entirely inside the box open_curtain("building", bbox, predicate = "within")
Overture publishes a release about once a month and removes old ones after
a few months. This reads the releases its STAC catalog currently lists, so
you can pick one for open_curtain(release = ) or check that a pinned
release is still available.
overture_releases(conn = NULL)overture_releases(conn = NULL)
conn |
A duckdb connection. Uses the cached session connection by default. |
A character vector of releases, newest first, such as
c("2026-08-19.0", "2026-07-22.0").
overture_releases()overture_releases()
sf::st_bbox() and sf::st_crs() methods for lazy overture_call tables,
so you can read a query's extent and coordinate reference system straight
from DuckDB, without pulling the rows into R with collect().
## S3 method for class 'overture_call' st_crs(x, ...) ## S3 method for class 'overture_call' st_bbox(obj, ...)## S3 method for class 'overture_call' st_crs(x, ...) ## S3 method for class 'overture_call' st_bbox(obj, ...)
... |
Unused, for compatibility with the generics. |
obj, x
|
An |
st_bbox() runs one ST_Extent_Agg() query over the geometry column.
st_crs() reads the coordinate reference system from DuckDB's typed
GEOMETRY column (for example GEOMETRY('OGC:CRS84') on duckdb >= 1.5),
falling back to EPSG:4326, which is what Overture stores.
st_bbox() returns an sf::st_bbox() object; st_crs() returns an
sf::st_crs() object.
bbox <- c(xmin = -120.5, ymin = 35.5, xmax = -120.0, ymax = 36.0) buildings <- open_curtain("building", bbox) sf::st_crs(buildings) sf::st_bbox(buildings)bbox <- c(xmin = -120.5, ymin = 35.5, xmax = -120.0, ymax = 36.0) buildings <- open_curtain("building", bbox) sf::st_crs(buildings) sf::st_bbox(buildings)
Reads the type-to-theme table for a release from Overture's STAC catalog,
so new types (such as bathymetry) appear without a package update. The
answer is cached per release. If the catalog can't be reached, the
package's built-in table is returned with a warning.
overture_types(release = latest_overture_release(conn), conn = NULL)overture_types(release = latest_overture_release(conn), conn = NULL)
release |
An Overture release, such as |
conn |
A duckdb connection. Uses the cached session connection by default. |
A data frame with columns type and theme.
overture_types()overture_types()
Writes the rows of an overture_call to Parquet files under output_dir,
in Overture's own theme=<theme>/type=<type>/ layout, and returns a new
overture_call that reads from the copy. Each type= directory also gets
an _overture.json manifest recording the source release and the bounding
box of every file, so open_curtain() on the copy skips files by location
just as it does on S3. snapshot_overture() defaults output_dir to
tempdir() and overwrite to TRUE.
record_overture( curtain_call, output_dir, overwrite = FALSE, write_opts = NULL, partition_by = NULL, grid = NULL, spatial_filter = NULL, ... ) snapshot_overture( curtain_call, output_dir = tempdir(), overwrite = TRUE, write_opts = NULL, partition_by = NULL, grid = NULL, spatial_filter = NULL, ... )record_overture( curtain_call, output_dir, overwrite = FALSE, write_opts = NULL, partition_by = NULL, grid = NULL, spatial_filter = NULL, ... ) snapshot_overture( curtain_call, output_dir = tempdir(), overwrite = TRUE, write_opts = NULL, partition_by = NULL, grid = NULL, spatial_filter = NULL, ... )
curtain_call |
An |
output_dir |
The directory where the data will be saved. |
overwrite |
If |
write_opts |
A character vector of extra options for DuckDB's |
partition_by |
Names of columns to partition by, below |
grid |
Cell size in degrees. If set, the copy is further partitioned
into a grid of that size (columns |
spatial_filter, ...
|
Passed to |
An overture_call reading from the downloaded data. Use
dplyr::show_query() to see its query and dplyr::collect() to bring
the rows into R.
DuckDB documentation on partitioned writes
broadway <- c(xmin = -73.99, ymin = 40.755, xmax = -73.98, ymax = 40.762) buildings <- open_curtain("building", spatial_filter = broadway) local_buildings <- record_overture(buildings, tempdir(), overwrite = TRUE) # or in one call local_buildings <- record_overture( "building", tempdir(), overwrite = TRUE, spatial_filter = broadway )broadway <- c(xmin = -73.99, ymin = 40.755, xmax = -73.98, ymax = 40.762) buildings <- open_curtain("building", spatial_filter = broadway) local_buildings <- record_overture(buildings, tempdir(), overwrite = TRUE) # or in one call local_buildings <- record_overture( "building", tempdir(), overwrite = TRUE, spatial_filter = broadway )
A thin wrapper around duckdb::duckdb_register() that creates a virtual
table, then casts the geometry column to DuckDB's GEOMETRY type in the
returned dbplyr representation. Mostly useful for join and spatial
operations within DuckDB. No data is copied.
sf_as_dbplyr( conn, name, sf_obj, geom_only = isFALSE(inherits(sf_obj, "sf")), overwrite = FALSE, ... )sf_as_dbplyr( conn, name, sf_obj, geom_only = isFALSE(inherits(sf_obj, "sf")), overwrite = FALSE, ... )
conn |
A DuckDB connection, created by |
name |
The name for the virtual table that is registered or unregistered |
sf_obj |
sf object to be registered to duckdb |
geom_only |
if TRUE, only the geometry column is registered. Always TRUE for sfc or sfg objects |
overwrite |
Should an existing registration be overwritten? |
... |
additional arguments passed to duckdb_register |
Behind the scenes, this function creates an initial view (name_init) with
the geometry stored as well-known binary via sf::st_as_binary. It then
creates the view name which replaces the geometry column with DuckDB's
internal geometry type. DuckDB geometries carry no coordinate reference
system; the geometry is registered in whatever system sf_obj uses.
a dbplyr lazy table
library(sf) con <- stage_conn() sf_obj <- st_sf(a = 3, geometry = st_sfc(st_point(1:2))) sf_as_dbplyr(con, "test", sf_obj) DBI::dbDisconnect(con)library(sf) con <- stage_conn() sf_obj <- st_sf(a = 3, geometry = st_sfc(st_point(1:2))) sf_as_dbplyr(con, "test", sf_obj) DBI::dbDisconnect(con)
stage_conn is primarily intended for internal use by other
overtureR functions. However, it can be called directly by
the user whenever it is desirable to have direct access to the
connection object. The core code is copied from duckdbfs, which deserves
all credit for the implementation
stage_conn( dbdir = ":memory:", read_only = FALSE, bigint = "numeric", config = list(), ... ) strike_stage(conn = getOption("overturer_conn", NULL))stage_conn( dbdir = ":memory:", read_only = FALSE, bigint = "numeric", config = list(), ... ) strike_stage(conn = getOption("overturer_conn", NULL))
dbdir |
Location for database files. Should be a path to an existing
directory in the file system. With the default (or |
read_only |
Set to |
bigint |
How 64-bit integers should be returned. There are two options: |
config |
Named list with DuckDB configuration flags, see https://duckdb.org/docs/configuration/overview#configuration-reference for the possible options. These flags are only applied when the database object is instantiated. Subsequent connections will silently ignore these flags. |
... |
Further arguments passed to DBI::dbConnect |
conn |
A duckdb connection. Defaults to the cached session connection, if there is one. |
When first called (by a user or internal function),
this function both creates a duckdb connection and places
that connection into a cache (overturer_conn option).
On subsequent calls, this function returns the cached connection,
rather than recreating a fresh connection. The dbdir, read_only,
bigint, and config arguments only take effect when a connection is
created.
This frees the user from the responsibility of managing a connection object, because functions needing access to the connection can use this to create or access the existing connection. At the close of the global environment, this function's finalizer should gracefully shutdown the connection before removing the cache.
strike_stage closes the connection.
a duckdb::duckdb()connection object
con <- stage_conn() strike_stage(con)con <- stage_conn() strike_stage(con)