Register Dataset

post /registry/register

Authorizations

Authorization string required

Bearer token. Pass as Authorization: Bearer <token> in the request header.

Request

catalog string required

Producer identity — organisation or group registering this dataset.

domain string required

Owning service or router. Examples: wikimedia, storywrangler, babynames. Query GET /registry/domains for the current list.

dataset_id string required

Short identifier, unique within domain. e.g. 'ngrams', 'revisions'

version string

Dataset version. 'latest' (default) is the mutable development slot — safe to re-register freely; each re-registration overwrites the previous entry. Semver strings (e.g. '1.0.0') create immutable snapshots: re-registering the same version string returns 409 Conflict.

PATCH — bug fixes (same schema, corrected values)
MINOR — new data (new time range, new entities — backward compatible)
MAJOR — breaking schema changes (column rename, endpoint_schema change)

See https://semver.org/.

data_location any required

Where to find the data. Three forms are supported for parquet:

Single file: /data/babynames.parquet
Flat directory: /data/babynames/ (all .parquet files read)
File list: ["/data/f1.parquet", "/data/f2.parquet"] — use this when the pipeline manages multiple snapshot files (e.g. DuckLake) and you need to pin exactly the live files at submit time.

For parquet_hive, provide the root of the hive partition tree — the directory directly above the first col=val/ level (e.g. /data/ngrams/ where subdirectories are ngram_size=1/granularity=daily/…). Do not point to a subdirectory.

data_format string required

Storage format. One of parquet (single file or directory) or parquet_hive (directory-partitioned by entity/time). See format reference.

description string required

Human-readable description of the dataset

data_schema any

Column names → DuckDB type strings (e.g. {'ngram': 'VARCHAR', 'pv_count': 'BIGINT'}). When provided, this is the authoritative schema. When omitted, schema is auto-derived from the data files.

manifest object

Coverage index: pre-computed availability and partition_index. Never read at query time.

ownership object required

Ownership and succession metadata.

lineage object required

Lineage metadata for a dataset: sources (external raw URLs), derived_from (intra-registry upstream datasets), consumers (downstream stories or scripts), and repo (producing pipeline URL).

entity_mapping object

Schema declaration for entity ID resolution (entity_type + local_id_column).

entities any

Entity mapping rows to upsert. Can also be submitted via the batch entities endpoint.

endpoint_schema object

Output shape: endpoint type and column names for types and counts.

transform object

Query slice axes: time dimension and categorical filter columns.

schema_version string

Version of storywrangler-schemas used at registration time. Auto-populated — do not set manually. Records the software-data version coupling so consumers know which registration contract was in effect when this entry was created.

curl -X POST http://localhost:8000/registry/register \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{}'