# Storywrangler Documentation --- # Getting started Storywrangler is a text-analysis data platform from the Vermont Complex Systems Institute. It serves n-gram frequencies, time series, and rank-turbulence divergence (allotaxonometry) over small and large parquet datasets through a single FastAPI service backed by DuckDB. Register a parquet dataset once, and its analytical instruments — the allotaxonometer, top n-grams, per-term time series — work on it immediately. This guide takes you from install to a first registered dataset you can query. ## Core concepts - **Domain** — a top-level data family with its own router and endpoints. `GET /registry/domains` lists the valid ones. - **Dataset** — a registered parquet source inside a domain, identified as `{domain}/{dataset_id}` (e.g. `wikimedia/ngrams`). The registry stores its location, layout, slice axes, and introspected metadata. - **Registry** — the catalog. `GET /registry/` lists every dataset with its `level_order` (hive nesting), `filter_values` (valid values per dimension), and `endpoint_schema` (output shape). This is the ground truth for what is queryable — always check it before constructing queries. - **Instruments** — the analysis endpoints layered on datasets: top n-grams, per-term time series, and the [allotaxonometer](/tools/allotaxonometer) (rank-turbulence divergence between two systems). - **Entities** — some datasets are partitioned by an entity (a country, a town). Entities use namespaced identifiers such as `wikidata:Q30` (United States). `GET /registry/{domain}/{dataset_id}/adapter` maps local IDs to canonical entity IDs and human-readable names. Entity-less datasets (e.g. `reddit/ngrams`) are sliced by plain filter dimensions instead and compared across dates. ## Install the SDK Read endpoints (registry lookups, domain queries) are public, but registration requires a Bearer API key — see [authentication](/authentication). We recommend installing the SDK with [uv](https://docs.astral.sh/uv/) (or pip): ```bash uv init --python 3.12 # create environment uv sync # creates the ~/.venv uv add storywrangler ``` The API base URL comes from the `STORYWRANGLER_URL` environment variable (or pass `base_url=`); your key from `API_KEY`. Interactive OpenAPI docs live at the API's `/docs`, and a machine-readable spec at `/openapi.json`. This documentation site is also machine-readable: `/llms.txt` returns everything as plain markdown, and `/sections.json` lists every section with a per-section `/{slug}/llms.txt` export — designed for LLM agents working with the platform. ## Scaffold a dataset project The SDK ships a scaffolder that lays out a complete submission project — `extract/` → `transform/` → `load/` — with a `submit.py` wired to the current schema and the agent assets (MCP config + Claude skills) already in place: ```bash uvx storywrangler new babynames --format parquet # or, for hive-partitioned data: uvx storywrangler new ngrams --format parquet_hive ``` Fill in `.env` (`DATASET_ID`, `DOMAIN`, `DATA_PATH`, `API_KEY`), map your entities in `config/entities.yaml`, edit `load/submit.py`, then `make submit` to register. See [registering a dataset](/register) for the field-by-field walkthrough. ## Register your first dataset Registration is a single POST. Suppose your parquet has one row per name, year, and sex: ``` types,counts,year,sex John,4394,1925,M Robert,2559,1925,M Axell,1956,1925,M Donald,1565,1925,M Peter,1464,1925,M ... ``` You tell the API where the data lives, its output shape (`endpoint_schema`), and which axes it can be sliced on (`transform`): ```python from storywrangler import Storywrangler, DatasetCreate client = Storywrangler(api_key="") # or set API_KEY / STORYWRANGLER_URL # Verify the connection client.users.whoami() dataset = DatasetCreate( catalog="vcsi", domain="babynames", dataset_id="ngrams", data_location="/mydata/babynames.parquet", data_format="parquet", description="Babynames frequencies by year and sex in the US.", endpoint_schema={"type": "types-counts"}, transform={"time_dimension": "year", "filter_dimensions": ["sex"]}, ownership={"owner_group": "vcsi", "contact": "vcsi@uvm.edu"}, lineage={"repo": "https://github.com/Vermont-Complex-Systems/babynames"}, ) client.registry.register(dataset) ``` `endpoint_schema={"type": "types-counts"}` is the shape the allotaxonometer expects; `transform` declares the sliceable axes (here `year` as the time dimension and `sex` as a filter). The server introspects the parquet at registration time to derive valid filter values and availability — you don't compute them client-side. ## Query it Once a dataset is registered, use the dataset-scoped client. It discovers filters and validates them against the registry before sending a request, so mistakes surface as clear errors instead of empty results: ```python wiki = client.dataset("wikimedia", "ngrams") wiki.filters # {'ngram_size': {'default': 1, 'valid': [1, 2]}, 'granularity': {...}} wiki.availability # date ranges per entity, from manifest.availability result = wiki.allotax( entity="wikidata:Q30", entity2="wikidata:Q145", dates="2026-05-01", dates2="2026-05-01", ngram_size=1, granularity="daily", ) ``` For a fast, lightweight comparison between two dates on a single entity, use `rtd` — it returns the wordshift only (no diamond plot or balance): ```python result = client.instrument.rtd( domain="babynames", dataset="ngrams", entity="wikidata:Q30", dates="1925", dates2="2025", sex="M", ) print(result["wordshift"][:5]) ``` ```python [{'type': 'Jackson', 'rank1': 676.0, 'rank2': 74.0, 'divergence': 0.00013999022173321902}, {'type': 'Duvall', 'rank1': 10309.5, 'rank2': 428.0, 'divergence': 0.0001209265034150297}, {'type': 'Bunny', 'rank1': 564.0, 'rank2': 5765.0, 'divergence': -9.604679669786964e-05}, {'type': 'Weaver', 'rank1': 8522.5, 'rank2': 736.0, 'divergence': 9.389480911305102e-05}, {'type': 'Bowl', 'rank1': 254.0, 'rank2': 1169.0, 'divergence': -8.660948752039387e-05}] ``` `GET /version` reports the API, schemas, DuckDB, allotax, and wordshift versions in effect. Storywrangler [versions](/versioning) the interaction between an instrument and a pipeline, so any result stays reproducible for papers and pipelines. ## Where to go next - [Querying datasets](/querying) — the discovery-first query workflow, instrument endpoints, and performance guidance. - [Registering a dataset](/register) — how to publish a new parquet dataset to the platform, whether through an existing endpoint type or a bespoke endpoint. - [Registering big data](/register-big-data) — the hive-partitioning convention and hash buckets for large datasets. - [Why Storywrangler?](/manifesto) — the motivation behind the platform. --- # The challenge of building digital commons for academia Academia is struggling to build the digital infrastructure that would benefit everyone collectively, yet is too costly for any individual group to undertake alone. This is another instance of the tragedy of the commons. ## Background At the Vermont Complex Systems Institute, we study collective attention — how populations allocate their awareness across ideas, events, and narratives over time. The raw material for this work is language: n-grams, phrases, and topics extracted from the platforms through which modern societies distribute their attention. Twitter, Reddit, Bluesky, Wikipedia, news archives, Google Books — in a reductive but powerful sense, all of these can be modeled as *ecological time series*: words propagating through amplification mechanisms, competing for collective focus. To study these systems at scale requires both large datasets and sophisticated analytical instruments. The [allotaxonometer](/tools/allotaxonometer), for instance, can compare two word-frequency distributions and identify which terms drove a divergence — but it needs a consistent data contract to do so across sources. Building that contract once per dataset, manually, does not scale. ## The tragedy of the commons in academic data The bulk of computational work in academia is done by graduate students who come and go. A student builds a pipeline to collect and process social media data, writes a paper, and leaves. The pipeline lives on a personal laptop or a research VM no one else can access. The dataset is never formally archived. The next student starts from scratch. This is not negligence — it is a structural problem. No individual student has the incentive to invest in infrastructure that outlasts them. No individual lab can sustain the engineering effort needed to build proper data governance. The community as a whole would benefit enormously from shared, maintained datasets and tools, but the cost of building that commons falls disproportionately on whoever acts first. Existing platforms help at the margins. [Zenodo](https://zenodo.org) provides DOIs for datasets and code. [GitHub](https://github.com) hosts pipelines. [Harvard Dataverse](https://dataverse.harvard.edu) archives large files. But none of these create the feedback loop that makes sharing feel worthwhile: knowing that your data is being *used*, seeing who built on it, having a clear path from archived data to live analysis. ## The current landscape Industry has solved parts of this problem. Projects like [Unity Catalog](https://www.unitycatalog.io/) and [Apache Polaris](https://polaris.apache.org/) provide data catalogs that register datasets centrally, enforce schemas, and make data discoverable across teams. But these tools are designed for organizations with dedicated data engineering teams, not for research groups where the "data engineer" is a second-year PhD student with three other papers to write. What academia needs is something narrower and more opinionated: a catalog designed around a specific research community, with instruments already built in, and governance rules that match academic reality — where ownership is personal, turnover is high, and the motivation to share is social rather than commercial. ## The Storywrangler platform Storywrangler is our attempt at that infrastructure. It is a decentralized data catalog that stores pointers to datasets — not the data itself — and enforces a schema contract at registration time. The key design decision is that **registering a dataset and making it analysis-ready are the same act**: if your dataset conforms to a known endpoint schema, the allotaxonometer (and future instruments) will work on it immediately, without any additional integration work. This is the opposite of the typical academic workflow, where a researcher adapts a tool to their data format after the fact. Here, the instrument defines the contract; the submitter meets it once; everyone benefits automatically. ### Why would you register your dataset? The answer depends on where you are in your research lifecycle: - **You want your analysis to be reproducible.** The registry records the data contract, the instrument version, and the query parameters of every result. A colleague can reproduce your allotaxonometer figure by pointing at the same registry entry. - **You want your data to outlast you.** When you leave, ownership transfers to your group or the institute. The dataset doesn't disappear with your GitHub account. - **You want to know who is using your data.** Downstream groups register their dependency in the registry. Their work appears in your impact record — research credit propagates without anyone coordinating directly. - **You want to share a dataset that isn't fully open.** Fine-grained access control means you can expose aggregate query results to the public, a filtered subset to collaborators, and full access to your own group — without having to choose between all-or-nothing. ## Simplicity as a design principle The platform is deliberately narrow. It does not try to be a compute environment, a notebook host, or a publication system. It does one thing: connect datasets to instruments through a shared schema contract, and record the governance metadata that makes that connection trustworthy over time. Registration is a single API call. The SDK reduces it to a few lines of Python. The platform validates schema compatibility immediately — if your data doesn't have the columns the instrument expects, you find out at registration time, not when a collaborator tries to reproduce your results six months later. ```python from storywrangler import Storywrangler, DatasetCreate client = Storywrangler(api_key="...") client.registry.register(DatasetCreate( domain="my-domain", dataset_id="my-dataset", data_location="/data/my-dataset.parquet", data_format="parquet", endpoint_schema={"type": "types-counts"}, transform={"time_dimension": "date", "filter_dimensions": ["language"]}, )) ``` ## Summary Storywrangler is a bet that the right response to the academic data commons problem is not a better archive, but a better interface between data and instruments. If the cost of sharing is low enough — one registration call, a schema you were going to write anyway — and the benefit is immediate (your data works with existing tools, your impact is tracked, your datasets survive your departure), then the commons sustains itself. We are building this at VCSI because we need it ourselves. The wikigrams pipeline, the babynames data, the Open Academic Analytics project — all of these live in the registry because the alternative is re-implementing the same integration code every time a new student joins the lab. We are opening it to the broader community because the problem is not specific to us, and the value of the catalog grows with every dataset registered. If you study collective attention, language, or sociotechnical systems — and you have data that can be modeled as an ecological time series — [register it](/register). The instruments are already waiting. --- # Authentication The Storywrangler API uses **API key authentication**. Accounts are created by an admin — you cannot self-register. Once your account exists, you exchange your credentials for an API key and use it as a Bearer token on all authenticated requests. ## Get your API key Once your account exists, POST your credentials to `/auth/login`. The response includes your permanent `api_key`. ```bash curl -X POST https://api.storywrangler.uvm.edu/auth/login \ -H 'Content-Type: application/json' \ -d '{"username": "alice", "password": "secret"}' ``` Response: ```json { "id": 1, "username": "alice", "email": "alice@uvm.edu", "role": "user", "api_key": "sk_abc123...", "is_active": true } ``` Save your `api_key` — this is the token you'll use for every subsequent request. ## Use your API key Pass the key as a Bearer token in the `Authorization` header: ```bash curl https://api.storywrangler.uvm.edu/auth/me \ -H 'Authorization: Bearer sk_abc123...' ``` With the SDK: ```python from storywrangler import Storywrangler client = Storywrangler(api_key="sk_abc123...") client.registry.register(dataset) ``` ## Roles | Role | What it can do | | --- | --- | | `user` | Register and update datasets. Query all public endpoints. | | `admin` | Everything a `user` can do, plus create accounts, manage roles, and access `/admin/*` endpoints. | ## Admin bootstrap A single admin account is seeded automatically on first server startup using environment variables: ```bash ADMIN_USERNAME=admin ADMIN_PASSWORD=changeme ADMIN_EMAIL=admin@example.com ``` The generated API key is printed to the server log on first boot. Use it to provision the first real accounts. ## Verify your token ```bash curl https://api.storywrangler.uvm.edu/auth/me \ -H 'Authorization: Bearer ' ``` Returns your user profile, or `401 Unauthorized` if the key is invalid or your account has been deactivated. --- # Agents & MCP Storywrangler is built to be driven by LLM agents, not just humans. Everything on this site is also available as plain text and as live tools, so an agent — Claude Code, or any [Model Context Protocol](https://modelcontextprotocol.io) client — can read the docs, discover what is registered, and validate a submission without scraping HTML or guessing. There are three ways in: the `llms.txt` exports (the docs as plain text), the MCP server (live tools over the registry and docs), and the Claude skills (durable workflow craft). ## Machine-readable docs Every page here has a plain-text twin, generated from the same source: - `/llms.txt` — the entire documentation as one markdown file. - `/sections.json` — every section with its discovery keywords. - `/{slug}/llms.txt` — a single guide (e.g. [`/querying/llms.txt`](/querying/llms.txt)). - `/api-reference/{tag}/llms.txt` — the endpoint reference for one tag, rendered live from the API's OpenAPI spec. Point an agent at [`/llms.txt`](/llms.txt) and it has the whole platform in context. ## MCP server `storywrangler-mcp` exposes the registry and the docs as MCP tools — the same tools over two transports. **Local (stdio)** — recommended for Claude Code. Drop this into your `.mcp.json`: ```json { "mcpServers": { "storywrangler": { "command": "uvx", "args": ["storywrangler-mcp"] } } } ``` `uvx` fetches and runs it — no install step. It points at the public deployment by default; override with `STORYWRANGLER_URL` and `STORYWRANGLER_DOCS_URL`, and set `STORYWRANGLER_INSECURE=1` while the uvm.edu TLS certificate mismatch persists. **Remote (streamable HTTP)** — the same server is mounted on the API at `/mcp` (stateless), for hosted agents that connect over HTTP instead of spawning a process. ### Tools | Tool | What it does | | --- | --- | | `list-sections` | List the documentation sections (title + discovery keywords). | | `get-documentation` | Fetch one section as markdown. | | `list-datasets` | List registered datasets from the live registry. | | `get-dataset` | One dataset's `level_order`, `filter_values`, and `availability` — the ground truth for building a valid query. | | `validate-submission` | Dry-run a `DatasetCreate` locally against the real schema and the registration guards, before you POST. | Because the registry tools return introspected metadata, an agent builds queries from what actually exists — no guessed entity IDs, granularities, or date ranges. ## Skills Two Claude skills carry the *when and why* — the discovery-first analyst craft ([querying](/querying)) and the submission craft ([registering](/register)) — while the exact field and endpoint reference stays in the docs and MCP tools, so nothing drifts. Here they are in full; copy either into your agent, or let `storywrangler new` scaffold both: ## Wire it into your project `storywrangler new` scaffolds the agent setup into a fresh dataset project — it writes `.mcp.json` and `.claude/skills/` automatically, so a Claude Code session in that project has the MCP tools and skills out of the box. Prefer a plugin? Install from the in-repo marketplace: ``` /plugin marketplace add Vermont-Complex-Systems/storywrangler ``` --- # Querying Datasets The Storywrangler platform is a set of API endpoints that expose the metadata of datasets contributed by the community. This means reading the registry is cheap, as it is only metadata. The required metadata is specified according to the Storywrangler [Specifications](/specification), which provide a formal standard by which we declare what metadata is necessary and the whereabouts of the data. The workflow is discovery-first: read the registry to learn what a dataset can answer, then build the query from what you found. Queries built from the registry work the first time; queries built from assumptions come back empty. ## Run it live The fastest way to learn the query workflow is to run it. These notebooks are executable copies of everything on this page: - [Getting started](https://colab.research.google.com/drive/1j6nxG0iMFaGXBoK9iaxc-NQwckcQgz9F?usp=sharing): registry exploration, dataset deep dive, first time series. - [Multiplatform](https://colab.research.google.com/drive/1I-PPb4kLXbVWApx-jn77pZOU0NVfIweV?usp=sharing): the same term across Twitter, Reddit, and Wikimedia, merged into one analysis. - [Instruments](https://colab.research.google.com/drive/1cZq56-RbilZgaog39Zx-_qShDL_iML_u?usp=sharing): allotaxonometer and wordshift on registered datasets. They also live in the repo under [`notebooks/`](https://github.com/Vermont-Complex-Systems/storywrangler/tree/main/notebooks). ## Step 1 - Discover datasets ``` GET /registry/ → all datasets (latest version each) ``` The per-dataset response contains everything needed to build a valid query. The registry can be glanced over from the [API references](https://storywrangler.uvm.edu/api-reference) page. For each domain, we list available datasets with query parameters and their description. With the SDK, the same catalog is a dataframe away: ```python from storywrangler import Storywrangler client = Storywrangler() client.registry.list().df() ``` Datasets are identified by three-level naming: `catalog/domain/dataset_id`, e.g. `vcsi/wikimedia/ngrams`. ## Step 2 - Dive into a dataset ``` GET /registry/{domain}/{dataset} → one dataset's metadata ``` The dataset-scoped client wraps this metadata and answers the three questions you need before querying: ```python wiki = client.dataset("wikimedia", "ngrams") wiki.filters # what axes can I slice on, with valid values and defaults wiki.availability # what time range exists, per entity and granularity wiki.adapter.df() # how do I name the entity: local names ↔ namespaced IDs ``` Filter names are per-dataset, not platform-wide: `wikimedia/ngrams` slices on `ngram_size`, `reddit/ngrams` on `n`, because that is what each pipeline registered. The registry is the ground truth, so check `filters` rather than guessing. For entity datasets, `adapter` maps human-readable names to namespaced identifiers such as `wikidata:Q30` (United States). Query endpoints accept either form. ## Step 3 - Query Datasets registered with the `types-counts` endpoint schema get the generic query endpoints out of the box, no bespoke code needed: ```python # what was trending in a window wiki.top_ngrams( entity="wikidata:Q30", dates="2025-01-01", dates2="2025-01-10", limit=10, ) # one term through time wiki.term_series( "Trump", entity="United States", dates="2024-09-01,2026-06-01", ).df() ``` Every response object has `.df()` to turn it into a pandas dataframe. The full parameter reference for each endpoint, including per-dataset extras such as `include="articles"` on Wikimedia, lives in the [API reference](https://storywrangler.uvm.edu/api-reference). ## Instruments The same discovery-first workflow drives the analysis endpoints: the [allotaxonometer](/tools/allotaxonometer) (rank-turbulence divergence between two systems), its lightweight `rtd` variant, and [wordshift](/tools/wordshift). They take the same entity, date, and filter arguments as the query endpoints. The [instruments notebook](https://colab.research.google.com/drive/1cZq56-RbilZgaog39Zx-_qShDL_iML_u?usp=sharing) runs all of them. ## When results come back empty or slow - **Empty result, no error.** The query was valid SQL over data that is not there. Check that the dates fall inside `availability` for that entity, and that the entity name matches the `adapter`. - **422 on a filter.** The value is not in the registered `filter_values` for that dataset. Read the error message: it lists the valid values. - **Slow query.** Constrain the date range. Full-history scans over large corpora can take minutes; `availability` tells you the tightest range worth asking for. Agents get the same discovery workflow through the [MCP server and llms.txt exports](/llms), so anything on this page works from Claude or any MCP client as well. --- # Building a dataset pipeline The registration call is deliberately small; the pipeline that produces the data is where the real work lives. This guide describes the storywrangler way of building that pipeline — standard data-engineering patterns (medallion tiers, columnar storage, declarative orchestration) adapted to academic reality, where the "data engineering team" is one grad student, compute is a laptop plus a SLURM allocation, storage is institutional NFS, and the pipeline must outlive its author. `storywrangler new` scaffolds this structure. The [Wikimedia](/case-studies/wikimedia) and [scisciDB](/case-studies/scisciDB) case studies are worked examples of everything below. ## The shape: extract → transform → load ``` my-dataset/ extract/ acquire raw data (scrape, download, API pulls) transform/ raw → submission-shaped parquet load/ submit.py — the registration payload, nothing else config/ entities.yaml — entity mappings tests/ entity coverage + contract checks Makefile extract / transform / validate / submit / test ``` The triad is literally ETL — with one platform-specific twist: the **L loads the catalog, not a warehouse**. Registration is a metadata upsert (a pointer plus the contract); the parquet files stay where the transform wrote them. That is data sovereignty made visible in the folder layout. Each stage owns one thing. `extract` fetches and never reshapes; `transform` owns all reshaping; `load` owns only the contract (`build_payload()` + `register()`). If `submit.py` needs to mangle data, that logic belongs in `transform` — the load step should stay boring enough that re-registering is a non-event. ## Medallion tiers, academically Industry lakehouses organise data as bronze → silver → gold. The same discipline works on institutional storage without any lakehouse machinery: - **Bronze** — raw inputs exactly as fetched, immutable. Wikipedia enterprise dumps, SSA zip files, API responses. Keep them: re-running `transform` must never require re-scraping. - **Silver** — cleaned, submission-shaped parquet. **This is what you register.** One table-like layout, stable column names, hive-partitioned when large. - **Gold** — derived artifacts (aggregations, embeddings, model outputs). These register later as their own datasets with `lineage.derived_from` pointing at the silver dataset — not as mutations of it. The payoff of the discipline is the same as in industry: every tier is reproducible from the one below it, and consumers (the query layer, your colleagues, your successor) only ever touch silver and gold. ## Parquet-first, out-of-core All transforms target parquet, and DuckDB is the workhorse. Columnar compression plus predicate pushdown means terabyte-scale n-gram counts are tractable on a laptop — the Wikimedia pipeline turns >100 GB/day of dumps into partitioned parquet without a cluster. There is no Spark, no warehouse: the same files your pipeline writes are the files the platform queries with `read_parquet()`. Data sovereignty comes free — you keep the files, the platform keeps a pointer. ## Partitioning: hive when large, flat when not - **Flat parquet** (single file or directory) is right up to a few GB. Don't partition small data. - **Partition** (`parquet_hive`) when the data is large *and* queried by slices. Partition keys should be exactly the axes callers filter on — entity, granularity, time — nothing else. Every level is named `col=val/`, and the tree root is what you register as `data_location`. - **Size files in the hundreds of MB.** Wikimedia lands at 300–400 MB per daily file: big enough that DuckDB isn't drowning in file opens, small enough to stream. Within each file, sort by your type column — rank lookups become range scans. - **Hash buckets** (`transform.hash_bucket` + `assign_bucket()` from the SDK) only when you need term-first lookups (one term across all dates). They are routing, not query axes — most datasets never need them. ## Design for submission from day one These choices cost nothing at the start and are expensive to retrofit after terabytes are written: 1. **Pick the endpoint type first** — `types-counts` (rank distributions, feeds the allotaxonometer) or `time-series` (tabular GROUP BY). It dictates your column shape. 2. **Name the columns** — `types`/`counts` by default, or plan to declare `type_column`/`count_column` overrides. 3. **One time column** (`time_dimension`), consistent across granularities; **one entity column** whose values you can map in `config/entities.yaml`. 4. **Validate early**: `uvx storywrangler-mcp validate-submission` on a sample payload/layout, before the full run — not after. The field-by-field contract is [registering a dataset](/register). ## Orchestration: make first, snakemake for clusters `make` is the default — five targets (`extract`, `transform`, `validate`, `submit`, `test`) anyone can read. Reach for `snakemake` when runs move to SLURM and you want sentinels, logs, and resumability. Either way the last two steps are the same, because registration is an upsert: ``` make validate # dry-run the payload through the validator (exits non-zero on errors) make submit # re-register; the `latest` slot updates freely on every run ``` `validate` gating `submit` in CI is the cheap insurance that a pipeline change didn't silently break the contract. ## Tests that earn their keep Two checks catch most real-world drift: - **Entity coverage** — every distinct value in the entity column has a mapping in `entities.yaml`. Upstream sources add countries, states, and venues without telling you. - **No null entities** — a null entity row is unqueryable and invisible. These are scaffolded in `tests/`; keep them wired to the real transform output, not fixtures. ## Built to outlive you The academic failure mode is not bad code — it's the pipeline that leaves with its author. The conventions that prevent it: - Machine-specifics live in `.env` (`DATA_PATH`, keys), never hardcoded. - Data lives on institutional storage (declare `ownership.storage_risk`), not a laptop. - `lineage.repo` points at the pipeline repository; `lineage.archival_doi` records the archived copy when a version is citable. - The registry records `schema_version` and derived availability at every registration — your successor can see exactly what contract was in effect. ## Where to go next - [Registering a dataset](/register) — the submission contract, field by field. - [Registering at scale](/register-big-data) — the `parquet_hive` and `hash_bucket` declarations for large datasets. - [Versioning](/versioning) — when a re-run is just a re-run and when it's a release. - [Wikimedia](/case-studies/wikimedia) and [scisciDB](/case-studies/scisciDB) — the patterns above applied to 100 GB/day of dumps and 200M paper records. --- # Registering a dataset Registering a dataset stores a pointer in Storywrangler's catalog: where the parquet lives, how it is laid out, who owns it, and where it came from. The data itself never moves — the platform queries it in place with DuckDB. There are two typical reasons to register: 1. **Share it more widely through the platform.** Your data has its own shape and no existing endpoint type fits, so the platform hosts it behind an endpoint written *for* it — which takes a PR to the [Storywrangler repo](https://github.com/Vermont-Complex-Systems/storywrangler/). Registration is then mostly a data-catalog entry, and that is already useful: discovery, health checks, ownership, lineage. 2. **Serve it through an existing endpoint type.** Your data matches a recurring shape — e.g. `types-counts`, a column of token values and a column of counts — and generic endpoints serve it immediately. This is how a dataset gets access to VCSI instruments such as the [allotaxonometer](/tools/allotaxonometer), which can then be served anywhere on the web. This page walks through both, assuming a relatively small dataset: a single parquet file or a flat directory of parquet files, up to roughly a gigabyte. Past a few gigabytes — or once queries slice into a much larger whole — hive partitioning keeps them fast: see [registering big data](/register-big-data). In both cases the dataset registers under an accepted domain (`GET /registry/domains` lists them). Domains cluster related datasets and improve discovery. Registration starts minimal and progressively adds capabilities along three axes (the performance axis is covered in [registering big data](/register-big-data)): ## Sharing data through the platform The platform can also help share data behind a bespoke endpoint. Adding it means opening a PR against the [Storywrangler repo](https://github.com/Vermont-Complex-Systems/storywrangler): a router function that looks the dataset up by its identity and hardcodes the query to the data's shape (see for instance the [`semantic-timeseries` endpoint](https://github.com/Vermont-Complex-Systems/storywrangler/blob/main/backend/app/routers/wikimedia.py), shown below). The registration is somewhat hardcoded too — `domain` and `dataset_id` become part of the contract, since the endpoint resolves the dataset by name. The PR can come after registration; the catalog entry is valid on its own. What registration buys here is the data catalog and its governance. ```json { "catalog": "compstorylab", "domain": "wikimedia", "dataset_id": "semantic-timeseries", "data_location": "/netfiles/compstorylab/semantic_timeseries.parquet", "data_format": "parquet", "description": "Daily labMT and ousiometric scores for each country's pageview-weighted corpus.", "ownership": {"owner_group": "Computational Story Lab", "contact": "compstorylab@uvm.edu"}, "lineage": { "repo": "https://github.com/Vermont-Complex-Systems/wikipedia-parsing", "derived_from": ["wikimedia/ngrams"] } } ``` Even a catalog-only entry gets the platform's guarantees: - The dataset appears in discovery (`GET /registry/domains`, the MCP `list-datasets` tool) and on [dataset health](/status). - `ownership` and `lineage` record who maintains it and what it was derived from — `derived_from` is what builds the dependency graph between datasets. ### Opting into the platform's axes Bespoke does not mean bare: the declarations from the first use case work here too, and bespoke endpoints use them. - **Entity system** — declaring `entity_mapping` (plus `entities` rows) standardizes the endpoint's parameter: callers pass `?entity=wikidata:Q30` or the raw local value, and the router resolves either one to the stored `country` value before querying. - **Time dimension** — declaring `transform.time_dimension` auto-derives `manifest.availability` at registration (min/max coverage per entity), so the UI and health checks know valid date ranges without touching the data. ```diff + "entity_mapping": {"local_id_column": "country", "entity_namespace": "wikidata"}, + "entities": [ + {"local_id": "United States", "entity_id": "wikidata:Q30", "entity_name": "United States"}, + // … one row per country + ], + "transform": {"time_dimension": "date"}, ``` In the router, entity resolution is one added line — swap the raw `country` parameter for `entity`, and resolve it to the stored value before querying (`resolve_entity` accepts canonical IDs and raw local values alike): ```python local_id = (await resolve_entity(db, "wikimedia", "semantic-timeseries", entity)).local_id ``` And the endpoint now answers to a standardized identifier: ```bash curl "https://storywrangler.uvm.edu/wikimedia/semantic-timeseries?entity=wikidata:Q30" ``` ## Serving through an existing endpoint type An instrument accepts any dataset that fulfils its requirements — the allotaxonometer requires a `types-counts` endpoint schema (see the instrument page). `types-counts` is the endpoint type for any rank-frequency distribution: a column of token values and a column of counts. At least one comparison axis is required — without one the API rejects the registration, since the allotaxonometer has no way to distinguish system 1 from system 2. `filter_dimensions` are categorical axes that serve as that comparison axis: the allotaxonometer compares `?town=Arlington vs ?town2=Addison`. At query time, omitting the parameter aggregates over all its values. Payloads on this page are plain JSON: pass one as-is to `client.registry.register(...)` in Python, to the `validate-submission` MCP tool for a local dry-run, or as the body of `POST /registry/register`. ```json { "catalog": "verso", "domain": "vt-zoning-atlas", "dataset_id": "ngrams", "data_location": "/data/vt-zoning/ngrams.parquet", "data_format": "parquet", "description": "Word frequencies from Vermont zoning bylaws by town.", "endpoint_schema": {"type": "types-counts"}, "transform": {"filter_dimensions": ["town"]}, "ownership": {"owner_group": "verso", "contact": "verso@uvm.edu"}, "lineage": { "repo": "https://github.com/Vermont-Complex-Systems/vt-zoning-atlas" } } ``` Once registered, each `filter_dimensions` entry becomes a bare query parameter on the allotaxonometer. Comparing Arlington vs Addison: ```bash curl "https://storywrangler.uvm.edu/storywrangler/allotax\ ?domain=vt-zoning-atlas&dataset=ngrams\ &town=Arlington&town2=Addison" ``` Without any entity mapping, we adopt the convention of simply incrementing provided filter dimensions when querying the API, e.g. `town` and `town2`. ### Providing entity mapping Drop `filter_dimensions` and add `entity_mapping` instead. The SDK validates all `entity_id` values locally before anything reaches the server. This also standardizes the API parameter: regardless of what the local column is called (`town`, `geo`, `country`…), callers always use `?entity=` and `?entity2=` — accepting either a canonical ID (`wikidata:Q675558`) or the raw local value (`Arlington`): ```diff - "transform": {"filter_dimensions": ["town"]}, + "entity_mapping": {"local_id_column": "town", "entity_namespace": "wikidata"}, + "entities": [ + {"local_id": "Arlington", "entity_id": "wikidata:Q675558", "entity_name": "Arlington, Vermont"}, + {"local_id": "Addison", "entity_id": "wikidata:Q353095", "entity_name": "Addison, Vermont"}, + // ... one row per town + ], ``` The corresponding curl command: ```bash curl "https://storywrangler.uvm.edu/storywrangler/allotax\ ?domain=vt-zoning-atlas&dataset=ngrams\ &entity=wikidata:Q675558&entity2=wikidata:Q353095" ``` By analogy to `filter_dimension`, the API now expects `entity` and `entity2` keys but values can either be the standardized or local identifiers. ### Adding a time axis `transform.time_dimension` opens a date-range axis for `BETWEEN` queries. The meaningful comparisons are same location across two time ranges (e.g. US 1990 vs US 2020), or same time range across two locations (e.g. US 2020 vs Quebec 2020). When `time_dimension` is set, the platform auto-populates `manifest.availability` at registration time — computing min/max date coverage per entity, so the UI knows valid ranges without querying the data. Start without entity mapping: `geo` stays in `filter_dimensions` and callers pass raw local IDs directly — `?geo=united_states` — with no namespace resolution. The manifest is keyed by the same local IDs. ```json { "catalog": "vcsi", "domain": "babynames", "dataset_id": "ngrams", "data_location": "/data/babynames/ngrams.parquet", "data_format": "parquet", "description": "Baby names by popularity, year, and location.", "endpoint_schema": {"type": "types-counts"}, "transform": { "filter_dimensions": ["year", "sex", "geo"] }, "ownership": {"owner_group": "vcsi", "contact": "compstorylab@uvm.edu"}, "lineage": {"repo": "https://github.com/Vermont-Complex-Systems/babynames"} } ``` And the corresponding curl command: ```bash curl "https://storywrangler.uvm.edu/storywrangler/allotax\ ?domain=babynames&dataset=ngrams\ &year=1990&year2=2020\ &geo=united_states&sex=F" ``` Moving `year` to `time_dimension` unlocks range queries and standardizes the API parameter: regardless of the underlying column name (`year`, `date`…), callers always use `?dates=` and `?dates2=`. Availability is auto-derived at registration — it tells the UI what date ranges are valid per entity without touching the data: ```diff "transform": { - "filter_dimensions": ["year", "sex", "geo"], + "filter_dimensions": ["sex", "geo"], + "time_dimension": "year", }, + // availability is auto-populated at registration: + // {"united_states": {"min": 1880, "max": 2022}, "quebec": {"min": 1980, "max": 2022}} ``` Adding `entity_mapping` promotes `geo` out of `filter_dimensions`. Availability keys auto-upgrade to canonical entity IDs: ```diff - "filter_dimensions": ["sex", "geo"], + "filter_dimensions": ["sex"], + "entity_mapping": {"local_id_column": "geo", "entity_namespace": "wikidata"}, + "entities": [ + {"local_id": "united_states", "entity_id": "wikidata:Q30", "entity_name": "United States"}, + {"local_id": "quebec", "entity_id": "wikidata:Q176", "entity_name": "Quebec"}, + ], ``` The corresponding curl command: ```bash curl "https://storywrangler.uvm.edu/storywrangler/allotax\ ?domain=babynames&dataset=ngrams\ &entity=wikidata:Q30\ &dates=1990&dates2=2020\ &sex=F" ``` ## Case studies The [scisciDB pipeline](/case-studies/scisciDB) walks the existing-endpoint-type path end to end (`time-series`, pre-aggregation, GROUP BY queries). The [Wikimedia pipeline](/case-studies/wikimedia) is a big data submission (`parquet_hive`) — see [registering big data](/register-big data) for those fields. --- # Registering big data Flat parquet is right up to a few gigabytes — don't partition small data. Past that, and once queries slice into the whole (one country, one granularity, one date range), switch `data_format` to `parquet_hive`: DuckDB then prunes partitions from directory names instead of opening files. This page covers the two at-scale declarations: hive-partitioned storage and hash-bucketed partitions. The field-by-field registration basics are in [registering a dataset](/register); the pipeline-side craft — choosing partition keys, sizing files — is in [building a pipeline](/pipelines). ## Hive-partitioned storage Set `data_format` to `parquet_hive` to enable [hive_partitioning](https://duckdb.org/docs/current/data/partitioning/hive_partitioning). All hive partition levels are auto-discovered from the directory structure — you only need to declare `time_dimension`. `data_location` points to the root of the hive tree: ```bash wikigrams/ ← data_location ngram_size=1/ granularity=daily/ country=United%20States/ date=2024-01-01/data.parquet ``` ```json { "catalog": "vcsi", "domain": "wikimedia", "dataset_id": "ngrams", "data_format": "parquet_hive", "data_location": "/netfiles/wikimedia_snapshots/wikigrams", "description": "Wikipedia n-gram frequencies by country and date.", "endpoint_schema": { "type": "types-counts", "type_column": "ngram", "count_column": "pv_count" }, "transform": {"time_dimension": "date"}, "ownership": {"owner_group": "vcsi", "contact": "compstorylab@uvm.edu"}, "lineage": {"repo": "https://gitlab.com/compstorylab/wikipedia-parsing"} } ``` `type_column` and `count_column` declare the data's non-default column names — declare around the data rather than renaming it. Note what is *not* declared: the `ngram_size`, `granularity`, and `country` levels are auto-discovered from the tree at registration, and callers reach the time dimension through the standardized `?dates=` parameter. The `entities` list is truncated here — the real one has ~100 rows, one per Wikipedia language edition. At query time, known values (entity, partition defaults) are pinned directly in the path, while the time level gets a wildcard and is filtered via `WHERE`. DuckDB only opens the matching files — no directory scanning: ```sql FROM read_parquet( 'ngram_size=1/granularity=daily/country=United%20States/date=*/*.parquet', hive_partitioning=true ) WHERE date BETWEEN '2024-10-01' AND '2024-10-31' ``` Auto-discovered partition levels become regular query params with server-injected defaults: ```bash curl "https://storywrangler.uvm.edu/storywrangler/allotax\ ?domain=wikimedia&dataset=ngrams\ &entity=wikidata:Q30&entity2=wikidata:Q145\ &dates=2024-10-01,2024-10-31&dates2=2024-10-01,2024-10-31\ &granularity=daily" ``` ### Gotchas - **Every directory level must be `col=val/`.** Non-hive names (`1grams/`, `daily/`) are not supported anywhere in the tree. Level discovery follows hive-named entries only — a level without one ends the walk, and whatever sits below it is invisible to the platform. Values with spaces or special characters are URL-encoded on disk (`country=United%20States`); DuckDB's partitioned writes do this automatically. - **`data_location` is the root, not a partition.** Point it at the directory directly above the first `col=val/` level. One level too deep and every derived level, default, and pinned path shifts. - **Nothing else lives under the root.** The query layer builds fixed-depth wildcard paths (one `/*` per level), so stray directories, scratch files, or loose parquet at the wrong depth break reads. The tree must also be uniform — same keys, same nesting order, same depth on every branch: discovery walks a single branch and assumes it represents the whole tree. - **Time values must sort chronologically as text.** The time level doesn't have to be a calendar date — plain integers (`year=1990`) work — but availability bounds and partition pruning compare directory values as strings, so use zero-padded ISO dates (`date=2024-01-01`, never `date=2024-1-1`). Callers' `?dates=` values are cast to the column's actual type before the `BETWEEN`. - **A 200 is not proof the tree was read.** Introspection failures (unreachable path, empty tree) are logged, not raised — registration succeeds and queries fail later. After registering, `GET /registry/{domain}/{dataset_id}` and confirm `level_order` and `manifest.availability` came back populated; empty means the walk failed. ## Hash-bucketed partitions The previous example is *date-first*: fast for loading all terms in a time window. For *term-first* lookups (e.g. a sparkline for a single word across all dates), `transform.hash_bucket` adds a content-sharded partition level — each term is hashed to a bucket, so the query layer reads exactly one file: ``` sparklines/ ← data_location country=United%20States/ ngram_bucket=0/data.parquet ← terms hashed to bucket 0 ngram_bucket=1/data.parquet ... ngram_bucket=15/data.parquet ← 16 buckets for the US ``` The hash *is* the index. The tempting alternative for term-first lookups is sorting terms into range files with an index sidecar mapping term ranges to filenames. That design costs an extra read per query to consult the index, and the sidecar sits inside the data tree where every glob picks it up as data. With a hash bucket the routing table is a function instead of a file: the pipeline computes `bucket = hash(term) % count` when writing, the query layer computes the same expression when reading, and they agree on where every term lives without storing or consulting any lookup structure. > **Why murmur3?** The hash must be fast, stable across every writer and reader, and available in every language a pipeline might use. `assign_bucket()` uses murmur3-32 with seed 0 and the sign bit cleared, matching DuckDB and DuckLake's built-in `murmur3_32()` — so buckets written by a DuckLake pipeline, a plain parquet writer, and the query layer all land identically. The implementation lives in one place (`storywrangler_schemas.hashing`, re-exported by the SDK as `storywrangler.hashing`); never reimplement it in a pipeline. You only declare the column name — the server auto-derives bucket counts by counting the bucket directories on disk, per partition combination: ```json { "catalog": "vcsi", "domain": "wikimedia", "dataset_id": "sparklines", "data_location": "/netfiles/compethicslab/wikimedia/sparklines", "data_format": "parquet_hive", "description": "Precomputed per-term sparkline time series (counts + rank) across all dates.", "entity_mapping": {"local_id_column": "country", "entity_namespace": "wikidata"}, "entities": [ {"local_id": "United States", "entity_id": "wikidata:Q30", "entity_name": "United States"}, {"local_id": "United Kingdom", "entity_id": "wikidata:Q145", "entity_name": "United Kingdom"} ], "transform": { "time_dimension": "date", "hash_bucket": "ngram_bucket" }, "lineage": { "repo": "https://github.com/Vermont-Complex-Systems/wikipedia-parsing", "derived_from": ["wikimedia/ngrams"] }, "ownership": {"owner_group": "vcsi", "contact": "compstorylab@uvm.edu"} } ``` `lineage.derived_from` records that the sparklines are computed from `wikimedia/ngrams` — the two datasets are linked in the dependency graph. In your transform step, use `assign_bucket()` from the SDK (same hash function as the query layer): ```python from storywrangler.hashing import assign_bucket # In your transform step — assign each row to a bucket bucket = assign_bucket(term="hello world", num_buckets=16) # → row goes into ngram_bucket={bucket}/data.parquet ``` At query time, the API hashes the term and reads only the matching bucket file: ```sql FROM read_parquet('sparklines/country=United%20States/ngram_bucket=7/data.parquet') WHERE ngram = 'hello world' ORDER BY date ``` Bucket counts can differ per combination — shard big partitions harder (United States at 32 while small countries keep 16; a large language at 64 while tiny ones need 1). Registration counts the bucket directories in every combination and stores a default plus overrides, keyed by the combination's level values in tree order: ```json {"column": "ngram_bucket", "default_count": 16, "overrides": {"1/United States": 32}} ``` This works with or without an entity level — a dataset partitioned only by `n=` and `lang=` gets keys like `"1/en"`: ``` ngrams_ts/ ← data_location n=1/ lang=en/ ngram_bucket=0/data.parquet ... ngram_bucket=63/data.parquet ← 64 buckets for English lang=gn/ ngram_bucket=0/data.parquet ← 1 bucket is enough for Guaraní n=2/ ... ``` ```json {"column": "ngram_bucket", "default_count": 1, "overrides": {"1/en": 64, "2/en": 64}} ``` ### Gotchas - **Never reimplement the hash.** A pipeline that buckets with a different function or seed produces a tree that registers cleanly and misroutes every term — indistinguishable from "no data" at query time. Always `assign_bucket()`. - **Re-shard and re-register are a pair.** The query layer resolves the modulus from the counts recorded at registration. If a rebuild changes any combination's bucket count and the dataset is not re-registered, terms silently route to the wrong bucket. - **Nothing but data in bucket directories.** Anything named `*.parquet` under a bucket level is read as data — index files, manifests, and markers must live outside the tree or use a different extension. - **Sort rows by term within each bucket.** The bucket bounds which file is read; the in-file sort lets parquet row groups prune within it, which is what keeps large buckets fast. ## Case studies The [Wikimedia pipeline](/case-studies/wikimedia) shows what a complete `submit.py` looks like for `parquet_hive`: raw Wikipedia dump → silver n-gram frequencies, partitioned by country, granularity, and date. --- # Design & architecture The Storywrangler API is a **data catalog with decentralized storage**, (very) loosely inspired by projects like [Unity Catalog](https://www.unitycatalog.io/) and [Apache Polaris](https://polaris.apache.org/). The registry stores metadata; the data stays on the submitter's own infrastructure. The API's dual goals are to improve the governance and discoverability of datasets that would otherwise remain siloed, and to wire them up with VCSI instruments so that their analysis is reproducible. It also scales to terabytes of data, bringing standardization to large pipelines that have historically been opaque and hard to compare. Registration requires learning about a few fields (see the [registering a dataset](/register) page), but not all at once. Arguably, even the simplest registration can goes while unlocking concrete benefits: access to analytical instruments developed at the institute, versioned data+instrument pairings for reproducibility, lineage tracking that exposes who maintains each dataset, and (opt-in) dependency graphs showing downstream consumers. By learning more specific fields to Storywrangler, registered datasets can power performant dashboards and visual data essays, thanks to our query engine based on duckdb and parquet files. Every dataset is addressed as a three-level namespace: `catalog/domain/dataset_id` — e.g. `vcsi/wikimedia/ngrams` or `compstorylab/babynames/ngrams`. The institute can also take on long-term stewardship of datasets when needed, guaranteeing continued usability beyond any single research group. ## Dataset model: registry as pointer store The registry stores metadata only — a pointer to wherever the data lives on institutional storage. The API resolves that pointer at query time and reads via DuckDB's `read_parquet()`. This has the benefit of preserving data sovereignty and avoids duplicating TB-scale datasets on platform storage. In that sense, we meet researchers where they are; they keep credit for the work they put into wrangling datasets. The health of datasets will be monitored daily to keep track of the status of the data ecosystems. All current datasets are **external**: the submitting group owns the storage, the platform owns the query layer. Another benefit of this approach is to make sensitive data more shareable; for instance, users can submit encrypted [parquet](https://duckdb.org/docs/current/data/parquet/encryption) files, which the API could expose to other groups that possess the proper keys to read them. The Storywrangler API itself would be blind to the data, but nonetheless facilitate sharing of the sensitive data. But Storywrangler is not any kind of data catalog, the Dataset schema has been designed to be compatible with a set of tools out of the box. ## Open specifications and derived schemas The contract between submitters and the platform is defined in a standalone [Storywrangler-Specification](/specification) — a plain markdown document that anyone can read, discuss, and propose changes to. The spec is intentionally decoupled from any particular implementation, much like the [Parquet format spec](https://parquet.apache.org/documentation/latest/) or [OpenAPI](https://spec.openapis.org/oas/latest.html). The monorepo then derives concrete code from this spec. The [storywrangler-schemas](https://pypi.org/project/storywrangler-schemas/) package is a thin Python wrapper that turns the spec's prose into Pydantic models — it is the single source of truth shared by both the backend (validation at registration) and the [storywrangler-sdk](https://pypi.org/project/storywrangler/) (client-side construction). Keeping the spec open means that adding a new field or endpoint type starts as a conversation in the spec repo, not a pull request buried in implementation code. ## Instrument architecture Instruments are standalone computational libraries that get wired into the platform at three tiers. The library itself has no dependency on Storywrangler — it is published independently (e.g. [allotax](https://pypi.org/project/allotax/) on PyPI) and can be used directly in scripts or notebooks. A backend router then wraps the library behind dataset-aware API endpoints, handling data resolution and query routing. Finally, consumers — the SDK, frontends, or direct API calls — each surface the instrument in the way that fits their use case. This three-tier separation means adding a new instrument does not require changing the SDK or the specification. A new router that imports the library and exposes an endpoint is sufficient; the SDK's generic HTTP client can call it immediately. ## Data sovereignty by design Each layer of the platform is independently useful. Groups retain their pipeline outputs as parquet files regardless of the API's availability, and instruments remain runnable as standalone libraries. The API adds live querying and real-time evolution tracking on top of data that groups already own. The platform layer is additive, not extractive. --- # Versioning Storywrangler uses two versioning layers with different purposes. Understanding when to use each prevents both over-engineering (archiving every pipeline run) and under-engineering (losing reproducibility when it matters). ## The two-layer model Registration is designed for **daily use** — re-register freely whenever your pipeline runs. Versioned snapshots and archival are opt-in steps you take when reproducibility or citation is needed. ``` Dataset pipeline → parquet files land on disk → POST /register ← frictionless, fast iteration ↓ when the interface contract changes Registry snapshot (version="1.0.0") → immutable entry in the platform registry → reproducible queries against this interface contract ↓ when long-term preservation is needed Dataverse / archival → DOI-bearing, externally citable → lineage.archival_doi recorded in the registry entry ``` ## The `version` field Every `DatasetCreate` payload carries a `version` field that defaults to `"latest"`: ```python DatasetCreate( domain="babynames", dataset_id="ngrams", version="latest", # default — the mutable development slot ... ) ``` ### The mutable slot — `"latest"` Re-registering with `version="latest"` always overwrites the existing entry. This is the default and requires no thought during active development. Pipeline re-runs, metadata corrections, and coverage updates all use this slot. ### Semver strings — immutable snapshots Once you bump to a named version, that entry is **locked**. Re-registering the same version string returns `409 Conflict`. ```python # Create an immutable snapshot DatasetCreate( domain="babynames", dataset_id="ngrams", version="1.0.0", ... ) ``` The platform follows [Semantic Versioning](https://semver.org/). For datasets, the three increments map to interface changes rather than code changes: | Increment | Trigger | | --- | --- | | **PATCH** `1.0.x` | Bug fix in processing — same schema, corrected values | | **MINOR** `1.x.0` | New data added — new time range, new entities; old queries still work | | **MAJOR** `x.0.0` | Breaking interface change — column renamed, `endpoint_schema` or `transform` axes changed | A routine pipeline re-run that only adds new rows to existing parquet files does **not** require a version bump. The contract (schema, query axes, data location) is unchanged. ## The `schema_version` field Every registration automatically records which version of `storywrangler-schemas` was in effect. This is the software–data version coupling recommended by the [Research Data Alliance versioning guidelines](https://zenodo.org/records/13743876): it records which registration contract was in effect so consumers know whether newer fields are available. ```python # schema_version is auto-populated — do not set manually DatasetCreate( ... # schema_version="1.0.0" ← injected from importlib.metadata ) ``` ## Inspecting versions ### List all versions for a dataset ```bash GET /registry/babynames/ngrams/versions ``` ```json { "domain": "babynames", "dataset_id": "ngrams", "versions": [ { "version": "latest", "schema_version": "1.0.0", "created_at": "2025-03-01T..." }, { "version": "1.1.0", "schema_version": "1.0.0", "created_at": "2025-02-01T..." }, { "version": "1.0.0", "schema_version": "1.0.0", "created_at": "2025-01-01T..." } ], "total": 3 } ``` ### Retrieve a specific version ```bash GET /registry/babynames/ngrams?version=1.0.0 ``` Omitting `?version` always returns the most recently registered entry. ## Platform component versions The `/version` endpoint reports the runtime software stack: ```bash GET /version ``` ```json { "api": "1.0.0", "schemas": "1.0.0", "duckdb": "1.1.3", "allotax": "0.3.1" } ``` When using the allotaxonometer, the response `meta` block also includes `dataset_version` and `allotax_version` — so any result can be traced back to the exact data contract and computation engine that produced it. ## Archiving to Dataverse When a versioned snapshot is ready for long-term preservation and citation, archive it in [Harvard Dataverse](https://dataverse.harvard.edu/) (or any DOI-issuing repository) and record the DOI in `lineage.archival_doi`: ```python DatasetCreate( domain="babynames", dataset_id="ngrams", version="1.0.0", lineage=LineageConfig( repo="https://github.com/Vermont-Complex-Systems/babynames", archival_doi="10.7910/DVN/XXXXXX", # set after archiving ), ... ) ``` The presence of `archival_doi` signals that this version's data is durably stored externally and is citable in publications. The registry entry remains the lightweight interface record; Dataverse holds the canonical, immutable data copy. --- # Storywrangler Entity Standards v0.0.3 *v0.0.3 · 2026-04-19* ## Table of Contents 1. [Introduction](#1-introduction) 2. [Definitions](#2-definitions) 3. [Specification](#3-specification) - [Entity Identifier Systems](#31-entity-identifier-systems) - [Field Taxonomies](#32-field-taxonomies) - [Entity Mapping Requirements](#33-entity-mapping-requirements) - [Validation Rules](#34-validation-rules) - [Unresolved Entities](#35-unresolved-entities) - [API Endpoint Schemas](#36-api-endpoint-schemas) - [Dataset Registration Schema](#37-dataset-registration-schema) 4. [Extending the Standards](#4-extending-the-standards) 5. [Appendix A: Validation Algorithms](#appendix-a-validation-algorithms) 6. [Appendix B: Revision History](#appendix-b-revision-history) --- ## 1. Introduction The Storywrangler Entity Standards define accepted entity identifier systems and field taxonomies to enable interoperability across datasets in the Storywrangler ecosystem. ### 1.1 Scope This specification defines: - Accepted entity identifier systems - Accepted field taxonomy systems - Format requirements for identifiers and classifications - Validation rules - Entity and field mapping requirements for adapters - Dataset registration schema (storage formats, query axes, entity mapping, versioning) - API endpoint schema contracts (`types-counts`, `time-series`) This specification does NOT define: - Internal query implementation (SQL generation, caching, routing) - Processing algorithms - API transport contracts (HTTP methods, status codes, pagination) ### 1.2 Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. --- ## 2. Definitions ### Entity A distinguishable person, place, organization, concept, event, or work referenced in a corpus. ### Entity Identifier A persistent, unique identifier from a recognized identifier system (Wikidata, ORCID, OpenAlex, ROR, DOI, ISBN). ### Field Taxonomy A classification system for organizing knowledge domains, academic disciplines, or subject areas. ### Adapter Code component responsible for transforming pipeline outputs to include standardized entity identifiers and field classifications. ### Local Identifier A corpus-specific identifier used when no standard identifier exists. --- ## 3. Specification ### 3.1 Entity Identifier Systems #### 3.1.1 Wikidata Q-codes **Namespace:** `wikidata` **Format:** `wikidata:Q[0-9]+` **Usage:** People, places, concepts, events, works, organizations. **Resolution Base URL:** `https://www.wikidata.org/wiki/` **External Specifications:** - Wikidata Identifiers: https://www.wikidata.org/wiki/Wikidata:Identifiers - Wikidata Data Model: https://www.mediawiki.org/wiki/Wikibase/DataModel **Validation:** - MUST match regular expression: `^wikidata:Q[0-9]+$` - SHOULD verify entity exists in Wikidata **When to use:** - Default for all entities with Wikidata entries - Required for concepts, places, events, works - For people when ORCID is not available - For organizations when ROR is not available --- #### 3.1.2 ORCID **Namespace:** `orcid` **Format:** `orcid:[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{3}[0-9X]` **Usage:** Academic authors, researchers, scholars. **Resolution Base URL:** `https://orcid.org/` **External Specifications:** - ORCID Structure: https://support.orcid.org/hc/en-us/articles/360006897674 **Validation:** - MUST match regular expression: `^orcid:[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{3}[0-9X]$` - MUST pass ISO 7064 mod 11-2 checksum validation (see Appendix A.1) - SHOULD verify ORCID is registered **When to use:** - REQUIRED for academic authors when available - Preferred over OpenAlex and Wikidata for researchers with publications --- #### 3.1.3 OpenAlex **Namespace:** `openalex` **Format:** `openalex:[AWICSFP][0-9]+` **Usage:** Any entity type from the OpenAlex knowledge graph. The letter prefix encodes the entity type: | Prefix | Entity type | Example | |--------|-------------|---------| | `A` | Author | `openalex:A5002034958` | | `W` | Work (paper, preprint, book, dataset) | `openalex:W2741809807` | | `I` | Institution | `openalex:I114027177` | | `C` | Concept / field of study | `openalex:C41008148` | | `S` | Source (journal, repository, conference) | `openalex:S1983995261` | | `F` | Funder | `openalex:F4320332161` | | `P` | Publisher | `openalex:P4310319965` | **Resolution Base URL:** `https://openalex.org/` **External Specifications:** - OpenAlex API: https://docs.openalex.org/ - Author disambiguation: https://docs.openalex.org/api-entities/authors/author-disambiguation **Validation:** - MUST match regular expression: `^openalex:[AWICSFP][0-9]+$` - SHOULD verify entity exists via OpenAlex API **When to use:** - Any dataset derived from OpenAlex - Authors (`A`): when ORCID is unavailable; OpenAlex covers ~250M authors including those who have not self-registered - Works (`W`): when DOI is unavailable (preprints, grey literature, books) - Institutions (`I`): when ROR is unavailable - Concepts (`C`): preferred over `mag:` namespace for field classifications (see §3.2.3) **Notes:** - OpenAlex IDs are algorithmically assigned; author records may occasionally merge or split as disambiguation improves - OpenAlex is the actively maintained successor to Microsoft Academic Graph - Unlike ORCID, OpenAlex IDs are not self-certified — ORCID remains the preferred identifier for authors when available --- #### 3.1.4 ROR (Research Organization Registry) **Namespace:** `ror` **Format:** `ror:[a-z0-9]{9}` **Usage:** Research organizations, universities, institutes. **Resolution Base URL:** `https://ror.org/` **External Specifications:** - ROR Documentation: https://ror.readme.io/ - ROR API: https://ror.readme.io/docs/rest-api **Validation:** - MUST match regular expression: `^ror:[a-z0-9]{9}$` - SHOULD verify ROR ID exists in registry **When to use:** - REQUIRED for research institutions when available - Preferred over Wikidata for academic organizations --- #### 3.1.5 IPEDS (Integrated Postsecondary Education Data System) **Namespace:** `ipeds` **Format:** `ipeds:[0-9]{6}` **Usage:** US postsecondary education institutions (colleges, universities). **Resolution Base URL:** `https://nces.ed.gov/collegenavigator/?id=` **External Specifications:** - IPEDS Overview: https://nces.ed.gov/ipeds/ - IPEDS Database: https://nces.ed.gov/ipeds/use-the-data **Validation:** - MUST match regular expression: `^ipeds:[0-9]{6}$` - SHOULD verify IPEDS ID exists in NCES database **When to use:** - US higher education institutions - Course catalog data - Educational research datasets - Use alongside ROR when both available **Relationship to ROR:** - Many US institutions have both IPEDS and ROR IDs - IPEDS is US-specific, ROR is international - Prefer ROR for international interoperability - Include both when available **Examples:** - `ipeds:230764` (University of Vermont) - `ipeds:166027` (MIT) - `ipeds:110635` (Harvard University) **Notes:** - IPEDS IDs are 6-digit integers (with leading zeros preserved) - Only covers US postsecondary institutions - Maintained by National Center for Education Statistics (NCES) --- #### 3.1.6 DOI (Digital Object Identifier) **Namespace:** `doi` **Format:** `doi:10.[0-9]{4,}/[^\s]+` **Usage:** Published scholarly works, datasets, books with DOIs. **Resolution Base URL:** `https://doi.org/` **External Specifications:** - DOI Handbook: https://www.doi.org/doi-handbook/ - DOI Resolution: https://dx.doi.org/ **Validation:** - MUST match regular expression: `^doi:10\.[0-9]{4,}/[^\s]+$` - SHOULD verify DOI resolves **When to use:** - REQUIRED for published papers, articles, datasets with DOIs - Use alongside ORCID for author attribution - Preferred over URLs for citing scholarly works --- #### 3.1.7 ISBN (International Standard Book Number) **Namespace:** `isbn` **Format:** `isbn:[0-9]{13}` or `isbn:[0-9]{9}[0-9X]` **Usage:** Books (both print and digital editions). **Resolution Base URLs:** - WorldCat: `https://www.worldcat.org/isbn/` - Open Library: `https://openlibrary.org/isbn/` **External Specifications:** - ISBN International: https://www.isbn-international.org/ - ISBN Users' Manual: https://www.isbn-international.org/content/isbn-users-manual **Validation:** - MUST match one of: - ISBN-13: `^isbn:[0-9]{13}$` - ISBN-10: `^isbn:[0-9]{9}[0-9X]$` - MUST pass checksum validation (see Appendix A.2) - Hyphens MUST be removed before validation **When to use:** - REQUIRED for books with ISBNs - Use ISBN-13 when both formats exist - Reference books in course catalogs, literature corpora, citation contexts **Notes:** - ISBNs should be stored without hyphens - ISBN-10 can be converted to ISBN-13 (prefix with 978) - Different editions of same book have different ISBNs --- ### 3.2 Field Taxonomies Field and subject classifications enable thematic organization and discovery across datasets. Multiple classification systems are accepted to accommodate domain-specific needs and address coverage gaps in general-purpose taxonomies. **General Principle:** Adapters MUST provide at least one recognized taxonomy identifier. Adapters MAY provide multiple taxonomies for the same entity to enable cross-system mapping. #### 3.2.1 Wikidata Fields **Namespace:** `wikidata` **Format:** `wikidata:Q{id}` **Usage:** General-purpose field classifications across all domains. **External Specifications:** - Wikidata Academic Disciplines: https://www.wikidata.org/wiki/Q11862829 - SPARQL Query Service: https://query.wikidata.org/ **Validation:** - MUST match regular expression: `^wikidata:Q[0-9]+$` - SHOULD verify entity exists and represents an academic field or discipline **When to use:** - Default for general cross-domain classification - When no domain-specific taxonomy applies - For interdisciplinary topics well-represented in Wikidata **Limitations:** - May lack precision for specialized subfields - Coverage gaps in emerging fields - Potential geographic and language biases --- #### 3.2.2 arXiv Categories **Namespace:** `arxiv` **Format:** `arxiv:{category}` or `arxiv:{archive}.{subject-class}` **Usage:** Preprint classifications, particularly computer science, physics, mathematics, and quantitative fields. **External Specifications:** - arXiv Category Taxonomy: https://arxiv.org/category_taxonomy - arXiv Subject Classifications: https://arxiv.org/help/api/user-manual **Validation:** - MUST match pattern: `^arxiv:[a-z-]+(\.[A-Z]{2})?$` - SHOULD verify category exists in arXiv taxonomy **When to use:** - Papers from arXiv or similar preprint servers - Computer science, physics, mathematics research - When arXiv's fine-grained categories add precision **Hierarchy:** arXiv categories have implicit two-level hierarchy (archive.subject-class). Adapters MAY encode this explicitly in metadata. --- #### 3.2.3 Microsoft Academic Graph (MAG) Field IDs **Namespace:** `mag` **Format:** `mag:{id}` **Usage:** Scholarly publications classified in Microsoft Academic Graph or OpenAlex. **External Specifications:** - OpenAlex Concepts: https://docs.openalex.org/api-entities/concepts - MAG Field of Study (legacy): https://www.microsoft.com/en-us/research/project/academic/ **Validation:** - MUST match pattern: `^mag:[0-9]+$` - SHOULD verify field ID exists (via OpenAlex API) **When to use:** - Datasets derived from OpenAlex or legacy MAG - When leveraging MAG's hierarchical field structure - Papers with existing MAG classifications **Note:** Microsoft Academic Graph was retired in 2021. For new datasets use `openalex:C...` (§3.1.3) instead of `mag:`. The `mag:` namespace is retained for backwards compatibility with existing data. --- #### 3.2.4 Multiple Taxonomies An entity MAY be classified using multiple taxonomy systems simultaneously. **When to use multiple taxonomies:** - Dataset originates from system with native classification (e.g., arXiv papers include arXiv categories) - Enable cross-dataset queries by providing Wikidata mapping - Preserve domain-specific precision while maintaining interoperability **Format:** ```json { "fields": [ {"id": "arxiv:cs.CL", "primary": true, "confidence": 1.0}, {"id": "wikidata:Q21198", "primary": false, "confidence": 0.8} ] } ``` **Requirements:** - At least one taxonomy MUST be marked as `primary` - Confidence scores (0.0-1.0) SHOULD be provided when mapping is uncertain - Adapters SHOULD document mapping methodology --- #### 3.2.5 Local Field Classifications When no standard taxonomy adequately represents a field, discipline, or subject area: **Namespace:** `local` **Format:** `local:{corpus_id}:{field_id}` **Examples:** - `local:indigenous-knowledge:traditional_medicine` - `local:women-in-math:algebra_educators` **Requirements:** - MUST be used only when standard taxonomies have coverage gaps - MUST document field definitions in dataset metadata - SHOULD provide human-readable labels - SHOULD attempt mapping to standard taxonomies - MAY be upgraded to standard identifiers in future versions --- #### 3.2.6 Cross-Taxonomy Mapping Storywrangler provides utilities for mapping between taxonomy systems where feasible. **Mapping guarantees:** - Exact mappings provided where documented - Approximate mappings provided with confidence scores - No guarantee of perfect translation across all systems **Query behavior:** When querying by field, Storywrangler API: 1. Returns exact matches for specified taxonomy 2. MAY return approximate matches from other taxonomies 3. Includes confidence scores for cross-taxonomy matches **Adapters are not required to provide mappings** - Storywrangler handles cross-taxonomy queries using internal mapping tables. --- #### 3.2.7 Hierarchy and Relationships Many taxonomies encode hierarchical relationships (broader/narrower fields). **Approach:** - Wikidata: Use SPARQL queries with `P279` (subclass of) relationships - arXiv: Implicit hierarchy in archive.subject-class structure - MAG: Hierarchical field structure available via OpenAlex API **Adapters are not required to explicitly encode hierarchy.** Storywrangler leverages native taxonomy structures for hierarchical queries. **Optional:** Adapters MAY provide explicit hierarchy in metadata for clarity or performance optimization. --- ### 3.3 Entity Mapping Requirements #### 3.3.1 Adapter Obligations Adapters MUST: 1. Map entities to at least one standard identifier system 2. Validate identifier format using Section 3.1 specifications 3. Use priority rules defined in Section 3.3.2 Adapters SHOULD: 1. Verify identifiers exist in source registries 2. Provide confidence scores for mappings when uncertain 3. Document entity resolution methodology in pipeline code #### 3.3.2 Priority Rules When multiple identifier systems could apply: **For people:** 1. ORCID (if academic/researcher — self-certified ground truth) 2. `openalex:A...` (if researcher with publications and no ORCID) 3. Wikidata Q-code (for scholars, public figures, or historical persons not in OpenAlex) **For works:** 1. DOI (if available) 2. `openalex:W...` (for works without DOIs: preprints, grey literature, book chapters) 3. ISBN (if book) 4. Wikidata Q-code (otherwise) **For organizations:** 1. ROR (if research institution, preferred for international interoperability) 2. IPEDS (if US higher education institution) 3. `openalex:I...` (if institution is in OpenAlex but lacks ROR) 4. Wikidata Q-code (otherwise) **Note:** US higher education institutions SHOULD include both ROR and IPEDS when available. **For published works:** 1. DOI (if available) 2. ISBN (if book) 3. Wikidata Q-code (otherwise) **For concepts, places, events:** 1. Wikidata Q-code (required) **For fields/subjects:** - Use taxonomy most appropriate for dataset origin - Multiple taxonomies MAY be provided (see Section 3.2.4) #### 3.3.3 Multiple Identifiers An entity MAY have multiple identifiers from different systems. When providing multiple identifiers: - One MUST be designated as primary - Others MAY be listed as alternatives - Adapters SHOULD document why multiple identifiers are provided --- ### 3.4 Validation Rules #### 3.4.1 Format Validation All entity identifiers and field classifications MUST: 1. Include namespace prefix 2. Match the format specification for their system 3. Not include whitespace #### 3.4.2 Existence Validation Adapters SHOULD verify that identifiers exist in their source registries. When verification fails: - MAY proceed with format-valid identifier - SHOULD document validation status in metadata - MUST NOT proceed if identifier format is invalid --- ### 3.5 Unresolved Entities #### 3.5.1 Local Identifiers When an entity cannot be mapped to a standard identifier system, adapters MAY use local identifiers. **Namespace:** `local` **Format:** `local:{corpus_id}:{local_id}` Where: - `{corpus_id}` is the corpus identifier - `{local_id}` is a corpus-specific identifier **Example:** `local:women-in-math:person_042` **Constraints:** - MUST be used only when no standard identifier exists - SHOULD include confidence score indicating mapping quality - MAY be upgraded to standard identifiers in future versions #### 3.5.2 Documentation Requirements When using local identifiers, adapters SHOULD document: - Why no standard identifier exists - Entity resolution attempts made - Potential future resolution strategies --- ### 3.6 API Endpoint Schemas This section defines the output schemas for standardized API endpoints. Each dataset declares its endpoint type via `endpoint_schema.type` (see §3.7.4). The endpoint type determines the response shape and column semantics. #### 3.6.1 `types-counts` A rank distribution: a bag of (type, count) pairs ordered by frequency. Used for rank-based comparisons such as rank-turbulence divergence. **Default columns:** - `types` (VARCHAR): The token, label, or type value - `counts` (INTEGER): Frequency count Datasets with non-default column names MUST declare them via `endpoint_schema.type_column` and `endpoint_schema.count_column` (see §3.7.4). **Response format:** JSON array, ordered by count descending. ```json [ {"types": "John", "counts": 1234}, {"types": "Mary", "counts": 987}, {"types": "Michael", "counts": 856} ] ``` **Requirements:** - Response MUST be a JSON array (no wrapper objects) - Results MUST be ordered by count in descending order - The type column MUST be a text/varchar data type - The count column MUST be an integer data type **Typical query axes:** entity, time range, categorical filters (e.g. sex, granularity). Filter parameters are dataset-specific, declared via `transform` (see §3.7.5). --- #### 3.6.2 `time-series` Tabular rows from a flexible GROUP BY query. The caller chooses which dimensions to group by and which to filter on. Used for trend analysis and exploratory drill-down. **Default columns:** - `count` (INTEGER): The numeric measure to SUM The count column name MAY be overridden via `endpoint_schema.count_column`. There is no `type_column` for `time-series` — all non-count columns are grouping/filtering dimensions. **Response format:** JSON array of row objects. Column names match the dataset schema. ```json [ {"field": "Computer Science", "year": 2020, "count": 142857}, {"field": "Computer Science", "year": 2021, "count": 158432}, {"field": "Physics", "year": 2020, "count": 98765} ] ``` **Requirements:** - Response MUST be a JSON array of row objects - Results SHOULD be ordered by the time dimension ascending - The count column MUST be an integer data type - The dataset MUST declare `transform.time_dimension` - The dataset MUST declare at least one `transform.filter_dimensions` entry **Query interface:** Callers specify `group_by` (which dimensions appear in the SELECT/GROUP BY) and filter parameters (which become WHERE clauses). Multi-value filters (comma-separated) generate IN clauses. --- ### 3.7 Dataset Registration Schema This section defines the `DatasetCreate` object — the registration payload submitted to the Storywrangler registry. Registration is an upsert: safe to re-run after data or metadata changes. The `(domain, dataset_id, version)` tuple uniquely identifies a dataset entry. #### 3.7.1 Overview A registration declares: 1. **Where the data lives** — storage format and file path (`data_format`, `data_location`) 2. **What the API returns** — endpoint type and column names (`endpoint_schema`) 3. **How callers can slice the data** — time axis, categorical filters, hash buckets (`transform`) 4. **How entities are resolved** — local column → canonical identifier mapping (`entity_mapping`, `entities`) 5. **Who owns it and where it came from** — governance and provenance (`ownership`, `lineage`) 6. **What version it is** — mutable `latest` slot or immutable semver snapshots (`version`) The registry auto-derives additional metadata at registration time: - `data_schema` — column names and DuckDB types (from the parquet files) - `level_order` — hive nesting order with type tags and defaults (from the directory structure) - `manifest.availability` — time/entity coverage ranges (from the data) - `filter_values` — enumerable distinct values per filter dimension (from the data) - `hash_bucket` config — bucket counts per entity (from the directory structure) #### 3.7.2 Required Fields All registrations MUST include these fields: | Field | Type | Description | |---|---|---| | `catalog` | string | Producer identity — organisation or group registering this dataset | | `domain` | string | Owning service or router (e.g. `wikimedia`, `babynames`, `scisciDB`) | | `dataset_id` | string | Short identifier, unique within domain (e.g. `ngrams`, `revisions`) | | `data_location` | string or string[] | Path to the data on disk (see §3.7.3) | | `data_format` | enum | Storage format: `parquet` or `parquet_hive` (see §3.7.3) | | `description` | string | Human-readable description of the dataset | | `ownership` | object | Ownership metadata (see §3.7.8) | | `lineage` | object | Provenance metadata (see §3.7.8) | #### 3.7.3 Storage Formats Storywrangler accepts exactly two storage formats: ##### `parquet` — flat parquet Single file, flat directory of files, or explicit file list. `data_location` supports three forms: - Single file: `/data/babynames.parquet` - Flat directory: `/data/babynames/` (all `.parquet` files are read) - File list: `["/data/f1.parquet", "/data/f2.parquet"]` No directory structure is interpreted. All filtering is done via WHERE clauses on columns within the files. ##### `parquet_hive` — hive-partitioned parquet Directory tree where **every partition level uses `col=val/` naming** (Apache Hive convention). `data_location` MUST be the **root** of the hive tree — the directory directly above the first `col=val/` level. ``` /data/ngrams/ ← data_location points here ngram_size=1/ granularity=daily/ country=United States/ date=2024-01-01/ data_0.parquet ``` **Requirements:** - Every partition level MUST follow hive naming (`col=val/`) - Non-hive directory names (e.g. `1grams/`, `daily/`) are NOT supported - Partition levels are auto-discovered from the directory structure at registration time - Each discovered level is classified by matching against declarations in `transform` and `entity_mapping` (see §3.7.6) **Why hive-only:** DuckDB's `hive_partitioning=true` handles partition pruning automatically for any combination of WHERE conditions. This makes filtering uniform across both storage formats. --- #### 3.7.4 Endpoint Schema Declares what columns the API reads and returns. Describes the response structure only — query slicing belongs in `transform` (§3.7.5). ```json { "type": "types-counts", "type_column": "ngram", "count_column": "pv_count" } ``` | Field | Required | Description | |---|---|---| | `type` | REQUIRED | Endpoint type. MUST be one of: `types-counts`, `time-series`. See §3.6 | | `type_column` | OPTIONAL | `types-counts` only. Column holding token/type values. Defaults to `types` | | `count_column` | OPTIONAL | Column holding the numeric measure, or a list of selectable measure columns. Defaults to `counts` for `types-counts`, `count` for `time-series` | Datasets that use the default column names (`types`/`counts` or `count`) MAY omit `type_column` and `count_column`. `count_column` MAY be a list when the data carries several alternative measures of the same conceptual count (e.g. reddit's content type × weighting columns: `all_score_weighted`, `comments_unweighted`, ...). The first entry is the default; endpoints expose the choice via a `weight` query parameter validated against the list. Every listed column MUST exist in the data. Rank columns are not part of the menu — a stored rank reflects the pipeline's canonical measure and does not change with `weight`. **Constraints:** - `types-counts` datasets MUST declare either `entity_mapping` or `transform.filter_dimensions` (there must be at least one axis to slice on) - `time-series` datasets MUST declare `transform.time_dimension` and at least one `transform.filter_dimensions` entry --- #### 3.7.5 Transform Configuration Declares the query slice axes — how callers can filter the dataset at request time. ```json { "time_dimension": "date", "filter_dimensions": ["sex"], "hash_bucket": "ngram_bucket" } ``` | Field | Required | Description | |---|---|---| | `time_dimension` | OPTIONAL | Column name for time-range filtering (e.g. `year`, `date`). For `parquet_hive`, this is the hive partition column holding the time value | | `filter_dimensions` | OPTIONAL | Non-hive categorical columns inside parquet files where omitting the filter aggregates over all values (e.g. `["sex"]`). NOT needed for hive partition levels — those are auto-discovered | | `hash_bucket` | OPTIONAL | Hive partition column holding content-shard bucket IDs (e.g. `ngram_bucket`). Bucket counts per entity are auto-derived from the directory structure at registration. See §3.7.5.1 | | `hash_algorithm` | OPTIONAL | Hash algorithm for bucket routing. Currently only `murmur3_32` is supported. Defaults to `murmur3_32` | | `hash_seed` | OPTIONAL | Seed for the hash function. Defaults to `0` (matches DuckDB's `murmur3_32()` default) | **For `parquet_hive` datasets:** Hive partition levels do NOT need to be declared. They are auto-discovered from the directory structure and stored in `level_order` (§3.7.6). The minimal `transform` submission for a hive dataset is: ```json {"time_dimension": "date"} ``` ##### 3.7.5.1 Hash Buckets Hash buckets are content-sharded partitions used to split large datasets (e.g. n-gram files) into manageable file sizes. They are **routing-only** — not query axes, not exposed to end users. **Submission format:** The submitter declares only the column name: ```json "ngram_bucket" ``` **Derived config:** At registration, the platform walks the directory tree and derives: - `default_count` — the modal bucket count across all entity × partition combinations - `overrides` — entity/partition combinations that differ from the default **Query-time routing:** The query layer computes the target bucket using murmur3 (seed 0): ``` bucket = (murmur3_32(term, seed=0) & 0x7FFFFFFF) % count ``` - `& 0x7FFFFFFF` clears the sign bit (murmur3 returns signed int32; bucket IDs MUST be ≥ 0) - Seed 0 matches DuckDB's `murmur3_32()` default - `count` is resolved per entity from the derived config - `hash_algorithm` and `hash_seed` are stored in the schema for machine-readable contract declaration **SDK function:** The SDK provides `storywrangler.hashing.assign_bucket(term, num_buckets)` — the canonical implementation of this algorithm. Pipelines MUST use this function (or an exact reimplementation) when partitioning files into bucket directories. This ensures consistency between data production and query-time routing. --- #### 3.7.6 Level Order (Derived) For `parquet_hive` datasets, the registry auto-discovers the on-disk hive nesting order at registration time and stores it as `level_order`. This is the single source of truth for the dataset's directory structure. **Format:** Ordered array of level descriptors: ```json [ {"column": "ngram_size", "type": "partition", "default_value": 1}, {"column": "granularity", "type": "partition", "default_value": "daily"}, {"column": "country", "type": "entity", "default_value": "Afghanistan"}, {"column": "date", "type": "time", "default_value": "2020-01-01"} ] ``` **Type tags:** Each discovered hive level is classified by matching against declarations: | Type | Source | Description | |---|---|---| | `partition` | undeclared hive levels | Queryable partition axis with auto-default | | `entity` | `entity_mapping.local_id_column` | Entity resolution column | | `hash_bucket` | `transform.hash_bucket` | Content-shard routing column | | `time` | `transform.time_dimension` | Time-range filtering column | | `filter` | `transform.filter_dimensions` | Non-hive filter appearing as hive level | **`default_value`:** The first on-disk value (sorted alphabetically) for each level. Used by the query layer when a caller omits a partition parameter. **Classification rules:** - Levels matching `entity_mapping.local_id_column` → `entity` - Levels matching `transform.time_dimension` → `time` - Levels matching `transform.hash_bucket` → `hash_bucket` - Levels matching a `transform.filter_dimensions` entry → `filter` - All remaining levels → `partition` (with auto-default from first on-disk value) **Registration validation:** - Registration MUST fail with 422 if `transform.hash_bucket` names a column not found on disk **Backward compatibility:** `level_order` is absent (`null`) for datasets registered before this feature was introduced. Query-time code MUST fall back to recursive glob patterns when `level_order` is absent. --- #### 3.7.7 Entity Mapping Declares how a dataset-local column maps to canonical entity identifiers from §3.1. ```json { "local_id_column": "country", "entity_namespace": "wikidata" } ``` | Field | Required | Description | |---|---|---| | `local_id_column` | REQUIRED | Column in the dataset holding the entity identifier | | `entity_namespace` | OPTIONAL | Canonical namespace for the identifiers (see §3.1). Enables cross-dataset entity graph traversal | **Two resolution patterns:** 1. **Opaque local keys** — the column holds non-standard values (e.g. country names, state abbreviations). The submitter provides `entities` rows mapping each `local_id` to a canonical `entity_id`. `entity_namespace` is RECOMMENDED. 2. **Global-identifier columns** — the column already holds values from a recognised namespace (e.g. OpenAlex author URLs, DOIs). Set `entity_namespace` to declare the namespace. Entity rows are OPTIONAL — useful only for display names or format normalization. **Dual role in hive datasets:** For `parquet_hive`, `local_id_column` is both the entity resolution column AND the hive partition key. The directory level is `local_id_column=value/`. This is intentional — hive partitioning promotes a column to the path level; the name remains the column name. **Entity rows:** Submitted inline as `entities` in the registration payload, or via a separate batch endpoint. Each row contains: | Field | Required | Description | |---|---|---| | `local_id` | REQUIRED | Dataset-local identifier value | | `entity_id` | REQUIRED | Canonical entity ID (MUST match a format from §3.1) | | `entity_name` | REQUIRED | Human-readable name | | `entity_ids` | OPTIONAL | Alternate identifiers (e.g. `["iso:US", "local:babynames:united_states"]`) | **Auto-derivation:** If `entity_namespace` is omitted but `entities` rows are provided, the namespace is auto-derived from the entity_id prefixes when all rows share the same known namespace. --- #### 3.7.8 Ownership and Lineage ##### Ownership ```json { "owner_group": "vcsi", "contact": "compstorylab@uvm.edu", "status": "active" } ``` | Field | Required | Description | |---|---|---| | `owner_group` | REQUIRED | Lab or research group identifier | | `contact` | REQUIRED | Email or GitHub handle of the current maintainer | | `status` | OPTIONAL | Lifecycle state: `active` (default), `needs_successor`, or `archived` | ##### Lineage ```json { "sources": {"geo": {"united_states": "https://www.ssa.gov/..."}}, "derived_from": ["wikimedia/ngrams"], "consumers": ["storywrangler/allotax"], "repo": "https://github.com/Vermont-Complex-Systems/babynames" } ``` | Field | Required | Description | |---|---|---| | `repo` | REQUIRED | Git repository URL for the producing pipeline | | `sources` | OPTIONAL | External raw data URLs, keyed by dimension then location | | `derived_from` | OPTIONAL | Intra-registry upstream datasets as `domain/dataset_id` | | `consumers` | OPTIONAL | Downstream users — stories, tools, or scripts | | `archival_doi` | OPTIONAL | DOI from an archival system (e.g. Harvard Dataverse) for long-term preservation | --- #### 3.7.9 Versioning The `version` field controls dataset mutability: - **`latest`** (default) — mutable development slot. Each re-registration overwrites the previous entry. Safe to re-register freely during development. - **Semver strings** (e.g. `1.0.0`) — immutable snapshots. Re-registering the same version string MUST return 409 Conflict. **Semver semantics:** - **PATCH** — bug fixes (same schema, corrected values) - **MINOR** — new data (new time range, new entities — backward compatible) - **MAJOR** — breaking schema changes (column rename, endpoint_schema change) Semver interpretation follows https://semver.org/. --- #### 3.7.10 Manifest (Derived) Pre-computed coverage metadata, never read at query time. Used for discovery, UI display, and SDK consumers. The name is borrowed from Apache Iceberg's concept of a manifest. ```json { "availability": { "United States": {"daily": {"min": "2024-01-01", "max": "2026-04-20", "types": 2648755}}, "Canada": {"daily": {"min": "2024-01-01", "max": "2026-04-20", "types": 1893002}} }, "partition_index": [ {"identifier": "Cat", "revision_count": 142, "first_edit": "2001-01-01"} ] } ``` | Field | Derived? | Description | |---|---|---| | `availability` | Yes — auto-populated at registration | Time coverage summary: MIN/MAX of the time dimension, grouped by entity and partition dimensions. Entity-first format when `entity_mapping` is present; flat otherwise | | `availability.*.types` | Yes — types-counts datasets only | Vocabulary size (distinct type count) at the latest available date, per entity × partition combination. A representative ceiling hint for `topN`-style parameters, not an exact figure for arbitrary date ranges | | `partition_index` | No — submitter-provided | Enumerable partition list with optional per-partition stats. Stored separately from summary responses | **`availability` auto-population:** When `transform.time_dimension` is set, the registry computes availability by scanning the data files at registration time. Submitters SHOULD NOT compute this manually. For datasets with `endpoint_schema.type = "types-counts"`, each availability leaf also gains a `types` count (best-effort — a leaf may hold bounds only if the count could not be read). --- #### 3.7.11 Complete Examples ##### Minimal flat parquet (no time axis) ```json { "catalog": "vcsi", "domain": "Vermont-Zoning-Atlas", "dataset_id": "zoning_bylaws", "data_location": "/data/vt/zoning_bylaws.parquet", "data_format": "parquet", "description": "Vermont municipal zoning bylaws.", "endpoint_schema": {"type": "types-counts"}, "entity_mapping": {"local_id_column": "town", "entity_namespace": "wikidata"}, "ownership": {"owner_group": "vcsi", "contact": "compstorylab@uvm.edu"}, "lineage": {"repo": "https://github.com/Vermont-Complex-Systems/vt-zoning-atlas"} } ``` ##### Hive-partitioned with entity mapping and hash buckets (wikimedia ngrams) ```json { "catalog": "vcsi", "domain": "wikimedia", "dataset_id": "ngrams", "data_location": "/netfiles/wikimedia_snapshots/wikigrams", "data_format": "parquet_hive", "description": "Wikipedia n-grams by frequency, date, and location.", "endpoint_schema": { "type": "types-counts", "type_column": "ngram", "count_column": "pv_count" }, "transform": { "time_dimension": "date", "hash_bucket": "ngram_bucket" }, "entity_mapping": { "local_id_column": "country", "entity_namespace": "wikidata" }, "ownership": {"owner_group": "vcsi", "contact": "compstorylab@uvm.edu"}, "lineage": { "sources": {"url": "https://dumps.wikimedia.org/other/enterprise_html/"}, "repo": "https://github.com/Vermont-Complex-Systems/wikipedia-parsing" } } ``` Hive levels auto-discovered: `ngram_size → granularity → country → date` (with `ngram_bucket` nested under `country`). ##### Time-series endpoint (scisciDB) ```json { "catalog": "vcsi", "domain": "scisciDB", "dataset_id": "field-venue-metrics", "data_location": "/netfiles/compethicslab/scisciDB/field-venue-metrics", "data_format": "parquet_hive", "description": "Precomputed paper counts by S2 field, venue, year, and metric type.", "endpoint_schema": { "type": "time-series", "count_column": "count" }, "transform": { "time_dimension": "year", "filter_dimensions": ["field", "venue"] }, "ownership": {"owner_group": "compethicslab", "contact": "compstorylab@uvm.edu"}, "lineage": { "sources": {"semantic_scholar": {"s2_papers": "https://api.semanticscholar.org/datasets/v1/release/"}}, "repo": "https://github.com/jstonge/scisciDB" } } ``` Hive level `metric_type` is auto-discovered as a `partition` level. The query layer injects its default value when callers omit it. --- ## 4. Extending the Standards ### 4.1 Proposing New Systems To propose a new entity identifier system or field taxonomy: 1. Open GitHub Discussion in storywrangler-standards repository 2. Provide specification following format in Section 3: - Namespace - Format with regular expression - Usage description - Resolution base URL (if applicable) - External specifications - Validation rules 3. Demonstrate: - Persistent, stable identifiers - Open access for validation/resolution - Active governance - Community need (affected datasets) ### 4.2 Governance The Technical Steering Committee reviews proposals quarterly. **Approval criteria:** - Majority vote from TSC - Technical feasibility demonstrated - Community need established - Maintenance commitment identified **Upon approval:** 1. Specification added to next minor version 2. Implementation in storywrangler-sdk required 3. Migration guide published 4. Announcement to community --- ## Appendix A: Validation Algorithms ### A.1 ORCID Checksum (ISO 7064 mod 11-2) The final character of an ORCID identifier is a check digit calculated using the ISO 7064 mod 11-2 algorithm: 1. Remove the `orcid:` prefix and all hyphens 2. Take the first 15 digits 3. Initialize total = 0 4. For each digit: - total = (total + digit) × 2 5. remainder = total mod 11 6. result = (12 - remainder) mod 11 7. If result = 10, check digit is 'X', otherwise it is the string representation of result The identifier is valid if the calculated check digit matches the final character. --- ### A.2 ISBN Checksum Validation #### A.2.1 ISBN-13 Checksum ISBN-13 uses a weighted sum modulo 10: 1. Remove `isbn:` prefix and all hyphens 2. Take all 13 digits 3. Multiply odd-position digits (1st, 3rd, 5th...) by 1 4. Multiply even-position digits (2nd, 4th, 6th...) by 3 5. Sum all results 6. Check digit = (10 - (sum mod 10)) mod 10 The ISBN is valid if the calculated check digit matches the 13th digit. #### A.2.2 ISBN-10 Checksum ISBN-10 uses modulo 11: 1. Remove `isbn:` prefix and all hyphens 2. Take first 9 digits 3. For each digit at position i (1-indexed): - Multiply digit by (11 - i) 4. Sum all results 5. remainder = sum mod 11 6. Check digit = 11 - remainder 7. If check digit = 10, use 'X' The ISBN is valid if the calculated check digit matches the 10th character. #### A.2.3 ISBN-10 to ISBN-13 Conversion To convert ISBN-10 to ISBN-13: 1. Prefix with "978" 2. Take first 9 digits of ISBN-10 3. Calculate new ISBN-13 check digit using A.2.1 --- ## Appendix B: Revision History ### Version 0.0.3 (2026-05-18) **Added Dataset Registration Schema (§3.7):** - Storage formats: `parquet` (flat) and `parquet_hive` (hive-partitioned with `col=val/` at every level) - `endpoint_schema`: output shape declaration (`type`, `type_column`, `count_column`) - `transform`: query slice axes (`time_dimension`, `filter_dimensions`, `hash_bucket`) - `entity_mapping`: local column → canonical entity ID resolution (connects to §3.1) - `level_order`: auto-derived hive nesting order with type tags and defaults - `manifest`: auto-derived availability and submitter-provided partition_index - `ownership` and `lineage`: governance and provenance metadata - `version`: mutable `latest` slot and immutable semver snapshots - Three complete registration examples (flat parquet, hive with entities, time-series) **Expanded §3.6 API Endpoint Schemas:** - Renamed §3.6.1 from "Top N-Grams Endpoint" to `types-counts` to match endpoint type name - Added §3.6.2 `time-series` — tabular rows from flexible GROUP BY queries - Documented custom column name declarations via `type_column` / `count_column` **Updated §1.1 Scope:** - Added dataset registration schema and endpoint schema contracts to "defines" list - Refined "does NOT define" — "Internal data formats" → "Internal query implementation" --- ### Version 0.0.2 (2026-03-24) **Added entity identifier systems:** - OpenAlex (`openalex:[AWICSFP][0-9]+`) — covers all OpenAlex entity types: authors (A), works (W), institutions (I), concepts (C), sources (S), funders (F), publishers (P) **Updated priority rules (§3.3.2):** - People: ORCID > `openalex:A...` > Wikidata - Works: DOI > `openalex:W...` > ISBN > Wikidata - Organizations: ROR > IPEDS > `openalex:I...` > Wikidata **Updated §3.2.3:** `mag:` namespace retained for backwards compatibility; new datasets should use `openalex:C...` --- ### Version 0.0.1 (2025-11-09) Initial release. **Included entity identifier systems:** - Wikidata Q-codes - ORCID - ROR - DOI - ISBN **Included field taxonomies:** - Wikidata fields - arXiv categories - Microsoft Academic Graph (MAG) field IDs **Initial governance:** Technical Steering Committee established --- # Allotaxonometer The allotaxonometer compares two ranked lists — word frequencies, name counts, topic distributions — across time, place, or any declared dimension. It measures rank-turbulence divergence (RTD) between the two lists and produces a contribution breakdown: which elements shifted the most, and in which direction. The visual output is an allotaxonograph — a mirror plot showing the top drivers of the divergence. Below, it compares English Wikipedia (US) on two days of very different collective attention — **election day** (2024-11-06) and the **Charlie Kirk** assassination (2025-09-10) — computed live in your browser from the API's ranked n-grams. The diamond plots every n-gram by its rank on each day; the wordshift ranks the terms that drove the two days apart. That figure is the [allotaxonometer-ui](https://www.npmjs.com/package/allotaxonometer-ui) npm package — the same Svelte components you can drop into your own app. Install it: ```bash npm install allotaxonometer-ui ``` Feed it two ranked type/count lists (for example the two systems returned by `GET /storywrangler/top-ngrams`), then render the dashboard: ```svelte {#if graph.dat} {/if} ``` The core rank-turbulence-divergence math lives in [allotaxonometer-core](https://github.com/Vermont-Complex-Systems/allotaxonometer-core), a Rust crate the package calls via WebAssembly. The companion instrument, [wordshift](/tools/wordshift), decomposes *why* two lists diverge into per-word contributions. --- # Wordshift Wordshift decomposes the difference in average word score (sentiment, frequency, or any per-word signal) between two text collections into word-level contributions — which words drove the change, through increased or decreased usage, and whether they pulled the score up or down. The reference implementation is the [wordshift](https://pypi.org/project/wordshift/) Python package, usable directly in scripts and notebooks. Install it: ```bash pip install wordshift ``` Give it two `{word: frequency}` maps and a bundled labMT lexicon; it returns each word's signed contribution to the change in average sentiment, plus the component sums a shift graph needs: ```python import wordshift # type2freq_1, type2freq_2: {word: frequency} maps, e.g. two days of ranked # n-grams from GET /storywrangler/top-ngrams result = wordshift.weighted_avg_shift( type2freq_1, type2freq_2, lexicon="labMT_English", # bundled labMT happiness lexicon top_n=50, # cap the returned per-word entries ) result["entries"][:5] # the words that drove the sentiment change ``` On the platform you do not have to fetch and feed the two systems yourself. `GET /storywrangler/wordshift` (or `wiki.wordshift(...)` in the SDK) resolves the dataset, loads both date-or-entity systems, and runs the shift server-side, the same way `/rtd` does for divergence. The core math lives in [wordshift-core](https://github.com/Vermont-Complex-Systems/wordshift-core), a Rust crate, mirroring the [allotaxonometer](/tools/allotaxonometer)'s `allotaxonometer-core`. --- # Wikimedia pipeline The Wikipedia enterprise dump comprises over 100Gb of compressed files, daily. Arguably, it counts as "big data" by academic standards. As is often the case, it is not just about size; one needs to extract the data from the raw html, making some decisions along the way about what should stay and what should go. Typically, this involves deciding what you count as content and what is boilerplate. Some insights only come from experimentation from downstream tasks. In our case, our downstream products included counting ngrams of varying sizes (e.g. words, bigrams, trigrams), before feeding that to our instruments that build on principles of heavy-tail distributions. This pipeline is also a story about how large data problems have become tractable on accessible hardware. Crunching 1 TB of n-gram counts on a consumer laptop was unthinkable a decade ago; today it is routine, thanks to open formats like Apache Arrow and out-of-core execution engines like DuckDB. This case study follows the full Storywrangler workflow: we start with the pipeline itself, walk through how its outputs are registered with the platform using the `storywrangler` SDK, and then look at how the Storywrangler API serves over 500 GB of Parquet data from a SvelteKit application. Here is what the allotaxonometer page of the app looks like: For context, on loading a pair of single days, it'll load two "types-counts" parquet files of around 300–400 MB each, access the `top N = types` (at first we do 100K, then 1M), before computing the rank turbulence divergence (RTD). You can think of RTD as something like entropy: a metric that surfaces which types in comparable systems have experienced the most turbulence, controlling for the heavy-tail nature of the system. Note that as we increase date ranges to 16 days (2 times 8 days), we are loading as many 300–400 MB files as there are days, summing around 4 GB of data without much hassle. ## The data pipeline This is how we go from 100Gb of daily dumps to about 165G of parquets for the daily level, hive partitioned on each date such that each file is about 300-400Mb. This is nice because duckdb knows how to skip days based on that hive partition, then within each file we sort based on types such that we can find the rank of a given type for a given time window. ## The registration process You have a dataset — a table of (type, count) pairs, possibly sliced by time, geography, or other dimensions. You register it by telling the API: - **Where the data lives** — a file path to parquet files - **What shape it has** — which column is the "type", which is the "count", and what time column exists - **What format** — flat parquet or hive-partitioned That's enough for the `/allotax` endpoint to load and compare any two slices of your data. The wikimedia ngrams registration adds entity mapping on top of that minimum. ```python register({ "catalog": "vcsi", "domain": "wikimedia", "dataset_id": "ngrams", "data_location": "/netfiles/wikimedia_snapshots/wikigrams", "data_format": "parquet_hive", "description": "Wikipedia n-grams by frequency, date, and location with entity mappings and ranks", "entity_mapping": { "entity_namespace": "wikidata", "local_id_column": "country", }, "entities": get_entities(), "endpoint_schema": { "type": "types-counts", "type_column": "ngram", "count_column": "pv_count", }, "transform": { "time_dimension": "date", # Hive levels (ngram_size, granularity, country, date) are auto-discovered. # hash_bucket: "ngram_bucket", # optional: content-sharded partition }, "lineage": { "sources": { "url": "https://dumps.wikimedia.org/other/enterprise_html/" }, "repo": "https://github.com/Vermont-Complex-Systems/wikipedia-parsing", }, "ownership": { "owner_group": "vcsi", "contact": "compstorylab@uvm.edu" }, }) ``` Two things this dataset declares beyond the minimum: non-default column names (`ngram`, `pv_count`) and entity mapping. Hive partition levels (ngram_size, granularity, country, date) are auto-discovered from the directory structure. ## What you gain by registering entities Without entities, callers pass raw column values directly — `?geo=US&geo2=CA`. It works, but the API treats them as opaque strings. ``` # filter-only (no entity_mapping — pass raw column values directly) GET /storywrangler/allotax ?domain=babynames&dataset=babynames-simple &geo=US&geo2=CA&sex=F&sex2=F &dates=1990&dates2=1980 ``` With entities, you map your local IDs to a shared namespace (e.g. Wikidata). This buys you: - **Cross-dataset queries** — "United States" means the same thing in babynames, wikimedia, and storywrangler ngrams. A frontend can let users pick an entity once and query all datasets. - **Discovery** — the registry knows which entities exist in your dataset, their names, aliases (`iso:US`), and date coverage. A UI can populate a dropdown without hitting your data. - **Provenance** — you can attach source URLs and availability ranges per entity, so users know where each slice came from and how far back it goes. The call pattern changes accordingly — instead of a raw column value, the caller passes a canonical ID: ``` # entity-mapped (canonical ID resolved to local column value at query time) GET /storywrangler/allotax ?domain=wikimedia&dataset=ngrams &entity=wikidata:Q30&entity2=wikidata:Q145 &dates=2024-10-01,2024-10-31&dates2=2024-10-01,2024-10-31 &granularity=daily ``` In short: filter-only is quick to submit; entities make your dataset a first-class citizen that composes with others. --- # scisciDB pipeline scisciDB is a pipeline for studying the science of science at scale. It ingests metadata from Semantic Scholar and OpenAlex — over 200 million papers — and pre-aggregates them into compact, queryable parquet files. The goal: let researchers explore publication trends by field of study, venue, topic, and metric type through Storywrangler's time-series endpoint, then drill down to select subsets of texts for deeper analysis. ## From raw data to pre-aggregated parquet The raw data lives in DuckLake (DuckDB + PostgreSQL metadata catalog) on VACC institutional storage. The pipeline, orchestrated by Dagster, runs on SLURM and pre-computes dimensional aggregations. Instead of querying 200M paper rows at request time, we materialize compact cross-tabs: - **field-venue-metrics**: S2 field of study x venue x year x metric_type x count - **field-topic-metrics**: S2 field of study x OpenAlex primary topic x year x metric_type x count Each is written as hive-partitioned parquet (partitioned by `field`), so DuckDB prunes to a single directory when filtering by field. ```sql WITH base AS ( SELECT list_filter(s2fieldsofstudy, x -> x.source = 's2-fos-model')[1].category AS field, venue, year, has_abstract, has_fulltext FROM s2_papers WHERE s2fieldsofstudy IS NOT NULL AND year IS NOT NULL AND year BETWEEN 1900 AND 2025 ) SELECT field, year, venue, 'total' AS metric_type, COUNT(*) AS count FROM base WHERE field IS NOT NULL GROUP BY field, year, venue UNION ALL SELECT field, year, venue, 'has_abstract' AS metric_type, COUNT(*) AS count FROM base WHERE field IS NOT NULL AND has_abstract = true GROUP BY field, year, venue UNION ALL SELECT field, year, venue, 'has_fulltext' AS metric_type, COUNT(*) AS count FROM base WHERE field IS NOT NULL AND has_fulltext = true GROUP BY field, year, venue ``` The `metric_type` dimension is synthetic — it doesn't exist in the raw data. The pipeline creates it by unioning total, has_abstract, and has_fulltext counts. This avoids computing these at query time. ## The time-series endpoint Unlike `types-counts` (which returns a ranked bag of tokens), `time-series` returns tabular rows from a flexible GROUP BY query. The caller chooses which dimensions to group by and which to filter on. ``` # Field trends over time GET /scisciDB/metrics?group_by=field,year&metric_type=total # Drill into one field's venues GET /scisciDB/metrics?group_by=venue,year&field=Computer+Science&metric_type=total # Compare metric types GET /scisciDB/metrics?group_by=year,metric_type&field=Computer+Science # Multi-value filter (top venues) GET /scisciDB/metrics?group_by=venue,year&venue=Nature,Science,PLOS+ONE&metric_type=total ``` The same endpoint serves all these queries — `group_by` controls the SELECT and GROUP BY, while filter params become WHERE clauses. Multi-value filters (comma-separated) generate IN clauses. ## Registration Once the parquet files are on disk, a submit script registers them with the Storywrangler API. The registration declares the endpoint type, column names, and query axes. ```python register({ "catalog": "vcsi", "domain": "scisciDB", "dataset_id": "field-venue-metrics", "data_location": "/netfiles/compethicslab/scisciDB/field-venue-metrics", "data_format": "parquet_hive", "description": "Precomputed paper counts by S2 field, venue, year, and metric type.", "endpoint_schema": { "type": "time-series", "count_column": "count", }, "transform": { "time_dimension": "year", "filter_dimensions": ["field", "venue"], # metric_type is a hive partition level — auto-discovered with default "has_abstract" # (alphabetically first). The query layer injects this default when callers omit it. }, "lineage": { "sources": {"semantic_scholar": {"s2_papers": "https://api.semanticscholar.org/datasets/v1/release/"}}, "repo": "https://github.com/jstonge/scisciDB", }, "ownership": { "owner_group": "compethicslab", "contact": "compstorylab@uvm.edu", }, }) ``` Key fields in the registration: - **`endpoint_schema.type = "time-series"`** tells the router to use `load_time_series()` instead of `load_system()` - **`filter_dimensions`** (field, venue) are non-hive columns safe to omit — omitting aggregates over all values - **`metric_type`** is a hive partition level, auto-discovered from the directory structure. The query layer injects a default when callers omit it, preventing accidental double-counting - Adding a new dataset (e.g. field-topic-metrics) is just another registration — same endpoint, same router, zero new code ## types-counts vs time-series Storywrangler supports two endpoint types, each suited to a different analytical workflow: - **types-counts**: Returns `{types: [...], counts: [...]}` — a ranked distribution. Used by `/allotax` to compare two systems via rank-turbulence divergence. The query is always: GROUP BY type_column, ORDER BY count DESC. - **time-series**: Returns `[{col1: v, ..., count: n}]` — tabular rows from a flexible GROUP BY. The caller picks the dimensions. Used for exploration, trend analysis, and drill-down before selecting subsets of texts. Both share the same infrastructure: registry-driven column names, WHERE clause generation from filter params, hive partition pruning. The difference is the query shape and what consumers do with the result. --- # Roadmap What the registry does today, what is missing, and what is planned. This page exists so potential adopters can see the gaps honestly rather than discovering them after onboarding. ## Current state The registry today is a metadata catalog: it stores pointers to datasets, validates their format at registration, and serves them to instruments. The core read/write path is operational. What is missing is mostly the connective tissue — lineage, ownership, resilience, and enforcement. | Area | Today | Gap | Priority | | --- | --- | --- | --- | | Dataset registration | Working | — | — | | Lineage | Field exists in schema | Not enforced; no traversal API | P2 | | Ownership & succession | `ownership` field in schema | Transfer endpoint and policy not yet implemented | P1 | | Identifier enforcement | Format accepted as string | Malformed IDs reach Silver undetected | P1 (via SDK) | | Endpoint schema contract | Field populated | Not validated at query time | P2 | | Catalog resilience | PostgreSQL only | Catalog unreachable when API is down | P3 | | Column descriptions | `{col: type}` only | No human-readable field documentation | P4 | | Gold layer | Not registered | Derived artifacts and ML outputs are invisible | P5 | | Storage backend | netfiles (institutional NFS) | Slow reads; UVM-only access; blocks immutable Frozen DuckLake snapshots | Planned | ## P1 — Ownership and succession When a student leaves, their dataset registration persists but institutional knowledge is lost. There is no mechanism for the institute to take custody and hand off to a new maintainer. The `ownership` field is now part of the `Dataset` model and registration payload. Four sub-fields, submitted as `"ownership": {...}`: - `owner_group` — lab or research group (`"compethicslab"`, `"VCSI"`) - `contact` — email or GitHub handle of the current maintainer - `status` — `active` | `needs_successor` | `archived` - `storage_risk` — durability signal: `managed` > `institutional` > `cloud` > `personal`. Datasets on `personal` or `cloud` storage automatically flag `needs_successor`. Paired with a `PATCH /admin/registry/{domain}/{dataset_id}/transfer` endpoint to change ownership. This is also the on-ramp for promoting a dataset to managed hosting when a student leaves without handing off. ## P2 — Lineage The registry knows where data lives but not where it came from or what depends on it. A schema change in `wikimedia/ngrams` currently has unknown blast radius. Three fields to add: - `derived_from` — list of `domain/dataset_id` references this dataset was built from (`["wikimedia/ngrams", "wikimedia/revisions"]`) - `produced_by` — pipeline or script that generated this dataset (git SHA, Dagster asset key, or script path) - `consumers` — opt-in list of known downstream users (stories, tools, partner groups) that would otherwise be invisible. Dataset-to-dataset downstream lineage is computable by inverting `derived_from`; `consumers` covers what that query cannot see. Paired with `GET /registry/{domain}/{dataset_id}/dependents` — lists all datasets that declare `derived_from` containing this one. Enables impact analysis before schema changes without a graph database: simple list traversal is sufficient at this scale. ## P3 — Catalog resilience If PostgreSQL is down, `GET /registry/` is unreachable. Groups lose discoverability even though their data is physically accessible. Proposed fix: a scheduled export job that writes the full registry to a parquet snapshot on netfiles: ``` /netfiles/compethicslab/registry/ snapshots/date=YYYY-MM-DD/registry.parquet latest.parquet ``` Groups can then query the catalog directly via DuckDB without going through the API. ## P4 — Column descriptions `data_schema` today is `{column: type}` only. No descriptions, no sensitivity flags. This limits discoverability for researchers who don't know what `rank` or `cnt` means in the ngrams schema. Proposed extension — backward compatible, old format detected and migrated on read: ```json { "rank": {"type": "BIGINT", "description": "Frequency rank of the n-gram on this date"}, "cnt": {"type": "BIGINT", "description": "Raw occurrence count"}, "freq": {"type": "DOUBLE", "description": "Normalized frequency (cnt / total tokens)"} } ``` ## P5 — Gold layer registration Derived datasets and ML artifacts are currently unregistered. There is no way to know which version of `wikimedia/ngrams` a given embedding was built from, or what stories depend on what Gold artifacts. No separate model registry is needed at this scale. Gold artifacts register via the same `submit.py` pattern, with `derived_from` pointing to their Silver inputs: | Asset | `data_format` | `derived_from` | | --- | --- | --- | | `wikimedia/article-embeddings` | `parquet_hive` | `["wikimedia/revisions"]` | | `wikimedia/topic-classifier` | `duckdb` | `["wikimedia/article-embeddings"]` | | `wikimedia/ngram-topics` | `parquet_hive` | `["wikimedia/ngrams", "wikimedia/topic-classifier"]` | A trained model is just another registered dataset whose `data_location` points to model weights on netfiles. If model versioning becomes critical, that is when to reconsider. ### Managed datasets When a student leaves without a successor, or when a dataset is small and stable enough, VCSI could take custody and move the data to platform-controlled storage. Managed ingestion would have three paths: **static clone** (copy parquet files, update `data_location`), **pipeline adoption** (clone the source repo, schedule via Dagster), or **PostgreSQL ingest** (for very small, highly-queried datasets). The `storage_risk: "managed"` value is reserved for this case. ### Internal tables Some datasets are small enough and queried frequently enough that storing them as parquet files adds unnecessary indirection. The planned design is to allow a third storage class — **internal tables** — where data is ingested directly into the platform's PostgreSQL database and served without DuckDB. The registry entry would declare `data_format: "postgres"` and `data_location` would identify the table rather than a file path. The query layer would route to a PostgreSQL cursor instead of `read_parquet()`. No datasets currently use this path. ## Planned — Storage backend: netfiles → S3 and Frozen DuckLake The current data serving path reads parquet files directly from netfiles, UVM's institutional NFS. This creates three compounding problems: - **Performance.** netfiles is an archival system optimised for sequential writes, not the random column reads DuckDB issues at query time. Every API request pays the NFS latency tax. - **Accessibility.** `data_location` paths are only reachable from UVM-networked machines. External collaborators and cloud compute cannot access the data without a VPN or an intermediary copy. - **Frozen snapshots.** [Frozen DuckLake](https://ducklake.select/2025/10/24/frozen-ducklake/) snapshots reference parquet files by HTTP/S3 URL — they require the files to be HTTP-accessible. netfiles paths cannot be referenced this way, so versioned snapshots cannot freeze against them. ### What Frozen DuckLake actually is A Frozen DuckLake is a tiny `.ducklake` file — a DuckDB database containing table schemas and pointers to parquet files stored in S3. It does **not** copy the data: the parquet files stay in place and the catalog freezes which exact files existed at that moment. The catalog file itself is kilobytes; the storage cost is negligible. Queries attach it like any DuckDB database: ``` ATTACH 'ducklake:s3://storywrangler-snapshots/babynames/ngrams/1.0.0.ducklake' ``` Because the catalog references S3 URLs and S3 object versioning prevents silent overwriting, a frozen snapshot is trustworthy in a way that a `data_location` pointer to a submitter's local disk is not: the submitter cannot accidentally break or mutate it. ### Migration path The migration is submitter-driven and incremental — no flag day. DuckDB handles S3 URIs transparently; the query layer is unchanged. Submitters update one line in their `submit.py` and upload their parquet files to the platform S3 bucket: ```python # before data_location = "/netfiles/gsm-storywrangler/babynames/ngrams/" # after data_location = "s3://storywrangler-data/babynames/ngrams/" ``` The registry stores the new path; the API serves queries against S3 from that point forward. Existing netfiles registrations continue to work during transition. ### Frozen DuckLake on top Once a dataset's `data_location` is an S3 URI, the platform can generate a frozen snapshot at version registration time automatically. At `POST /register` with a semver version string, the platform: 1. Enumerates all `.parquet` files at `data_location` (already done for introspection) 2. Calls `ducklake_add_data_files()` to register those S3 URLs in a `.ducklake` catalog 3. Uploads the catalog to `s3://storywrangler-snapshots/{domain}/{dataset_id}/{version}.ducklake` 4. Stores the URL in `RegistryEntry.ducklake_path` (already reserved in the schema) Query layer change is minimal: versioned requests check `ducklake_path` and use `ATTACH 'ducklake:...'` instead of `read_parquet(s3://...)`. The `version="latest"` slot always reads the live `data_location` — no snapshot is generated for it. ### Cost S3 Standard is approximately $0.023/GB/month, with S3 Intelligent-Tiering reducing cost further for infrequently accessed snapshot versions. For a typical research dataset in the tens of GB range, storage is $1–5/month. Request costs are negligible at academic traffic volumes. The `.ducklake` catalog files themselves are kilobytes — essentially free. ### Schema reservation `RegistryEntry.ducklake_path` is already a nullable column in the registry schema. It is `null` for all current entries (netfiles-backed and `latest` versions). No existing queries are affected. When the migration lands, the column is populated automatically by the registration endpoint — submitters never set it. ## Open questions These are unresolved policy questions, not technical gaps. Documenting them here so collaborators can see what is still being decided. - **Should stories be registered assets?** A story consuming `wikimedia/ngrams` is a downstream dependency. Tracking it would complete the lineage picture. But stories are frontend code, not data — the boundary is unclear. - **Storage class enforcement?** Should the platform hard-reject registration of `personal` storage, or only warn? Hard enforcement reduces friction from bad registrations; soft warning may be ignored. - **Spec validation strictness.** Format validation (regex + checksum) should be a hard reject at registration. Existence checks (live ORCID registry, Wikidata SPARQL) are expensive and should be warnings only. The spec already makes this distinction (`MUST` vs `SHOULD`). - **Who governs the registry?** The succession mechanism will exist once P1 lands, but the policy is unresolved: who approves new registrations, holds admin access, and can archive or transfer a dataset when a student leaves without handing off? This needs a named role before external groups onboard. - **PII and data sensitivity.** `storage_risk` covers durability but not sensitivity. Some datasets contain or are derived from identified individuals. Does the registry need a `sensitivity` field? Who determines classification, and does it gate access to `data_location`? --- # API reference: auth Generated from the live OpenAPI spec of the Storywrangler API. ## POST /auth/login Login Exchange username + password for the user's API key. ## GET /auth/me Me Return the authenticated user's profile. --- # API reference: registry Generated from the live OpenAPI spec of the Storywrangler API. ## POST /registry/register Register Dataset Register a new dataset or update an existing one (upsert). ## GET /registry/domains List Valid Domains List accepted domain names for dataset registration. **Example response** ```json { "domains": [] } ``` ## GET /registry/ List Registered Datasets List all registered datasets (latest version per dataset). **Example response** ```json { "datasets": [ { "catalog": "vcsi", "domain": "babynames", "dataset_id": "ngrams", "data_location": "/data/babynames/names.parquet", "data_format": "parquet", "description": "US baby name frequencies by state, year, and sex.", "created_at": "2024-01-15T10:00:00Z", "updated_at": "2024-01-15T10:00:00Z" } ], "total": 1 } ``` ## GET /registry/{domain}/{dataset_id} Get Dataset Info Get metadata for a specific registered dataset. Defaults to the latest version. Pass `?version=1.0.0` to retrieve an immutable snapshot. Pass `?full=true` to include filter_values and partition_index. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | path | string | yes | | | | `dataset_id` | path | string | yes | | | | `version` | query | string or null | no | | | | `full` | query | boolean | no | `false` | | **Example response** ```json { "domain": "babynames", "dataset_id": "ngrams", "data_location": "/data/babynames/names.parquet", "data_format": "parquet", "description": "US baby name frequencies by state, year, and sex.", "manifest": { "data_schema": { "year": "int32", "sex": "varchar", "types": "varchar", "counts": "int64" } }, "lineage": {}, "endpoint_schema": { "type": "types-counts", "time_dimension": "year", "filter_dimensions": [ "sex" ] }, "created_at": "2024-01-15T10:00:00Z", "updated_at": "2024-01-15T10:00:00Z" } ``` ## GET /registry/{domain}/{dataset_id}/adapter Get Adapter Info List entity mapping rows for a dataset (entity_id ↔ local_id). **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | path | string | yes | | | | `dataset_id` | path | string | yes | | | **Example response** ```json [ { "local_id": "united_states", "entity_id": "wd:Q30", "entity_name": "United States", "entity_ids": [ "wd:Q30" ] }, { "local_id": "california", "entity_id": "wd:Q99", "entity_name": "California", "entity_ids": [ "wd:Q99" ] } ] ``` ## GET /registry/{domain}/{dataset_id}/validate-sources Validate Dataset Sources Validate that all source URLs for a dataset are still accessible. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | path | string | yes | | | | `dataset_id` | path | string | yes | | | **Example response** ```json { "domain": "babynames", "dataset_id": "ngrams", "summary": { "total_urls": 2, "accessible": 2, "inaccessible": 0, "all_accessible": true }, "sources": { "national": { "url": [ { "url": "https://example.org/names.zip", "status": "accessible", "status_code": 200, "method": "HEAD" } ] } } } ``` ## GET /registry/{domain}/{dataset_id}/versions List Dataset Versions List all registered versions for a dataset, newest first. Returns the full version history: 'latest' (the mutable dev slot, if present) plus any immutable semver snapshots (e.g. '1.0.0', '2.0.0'). **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | path | string | yes | | | | `dataset_id` | path | string | yes | | | --- # API reference: storywrangler Generated from the live OpenAPI spec of the Storywrangler API. ## GET /storywrangler/top-ngrams Top Ngrams Top types by count for any registered types-counts dataset. The generic form of the per-domain `/{domain}/top-ngrams` endpoints: the dataset is selected by query params and filter dimensions are passed by their registered column names — `?ngram_size=1&granularity=daily` for wikimedia, `?n=1&lang=en` for reddit, `?sex=M` for babynames. Discover them via `GET /registry/{domain}/{dataset_id}`; missing partition dims get the dataset's registered defaults. With `dates2`, returns two systems keyed by date range for a temporal comparison (same shape as the per-domain endpoints). mongodb pass-through datasets accept single dates only. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | query | string | no | `"wikimedia"` | Domain owning the dataset | | `dataset` | query | string | no | `"ngrams"` | Dataset ID within the domain | | `dates` | query | string or null | no | | Date/year range for system 1. Single value '2024-10-01' or range '2024-10-01,2024-10-31'. Omit to load all time (datasets without a time dimension take no dates; mongodb datasets require a single date). | | `dates2` | query | string or null | no | | Optional second range for a temporal comparison. | | `entity` | query | string or null | no | | Global entity ID (e.g. 'wikidata:Q30') or local ID. Omit for datasets without entity_mapping. | | `weight` | query | string or null | no | | Count measure — one of the dataset's endpoint_schema.count_column entries. Defaults to the first registered measure. | | `limit` | query | integer | no | `100` | Max types per system (0 = no limit). manifest.availability's `types` gives the vocabulary ceiling. | **Response** - `data` (array of object) — Type/count entries sorted by count descending. With dates2, replaced by two arrays keyed by each date range (e.g. '2024-10-01_2024-10-07'). - `types` (string) — The type (n-gram, name, ...) - `counts` (number) — Total count over the date range under the selected weight. Matches the declared column type: integer for integer-counted datasets (wikimedia, bluesky, twitter), float for weighted measures (reddit). - `metadata` (object) — Request metadata echoed back - `domain` (string) — Dataset domain - `dataset` (string) — Dataset ID - `dataset_version` (string) — Registered dataset version served - `entity` (string) — Entity ID used (null for entity-less datasets) - `filters` (object) — Filter dimensions applied, defaults included - `weight` (string) — Count column used **Example response** ```json { "data": [ { "types": "the", "counts": 12345678 }, { "types": "of", "counts": 9876543 } ], "metadata": { "domain": "wikimedia", "dataset": "ngrams", "dataset_version": "1.0.0", "entity": "wikidata:Q30", "filters": { "ngram_size": 1, "granularity": "daily" }, "weight": "pv_count" } } ``` **Usage notes** - `filters`: Filter dimensions are dataset-specific query params using registered column names (?ngram_size=1&granularity=daily for wikimedia, ?n=1&lang=en for reddit, ?sex=M for babynames). Discover them via GET /registry/{domain}/{dataset_id} (level_order / transform.filter_dimensions). - `comparison`: Pass dates2 for a two-system temporal comparison; the response keys the two arrays by their date ranges instead of 'data'. mongodb pass-through datasets (twitter) accept single dates only. - `dates_range`: A range entirely outside the slice's availability is a 400 naming the actual bounds (from manifest.availability) — no need to look them up before querying. ## GET /storywrangler/term-series Term Series Per-date time series for a single type in any registered types-counts dataset. Returns counts, and rank/freq when the dataset declares `rank_column`/ `freq_column`. Fast path: hash-bucket lookup on the type-first sparkline companion (resolved from lineage); slow fallback: a scan of the date-first tree for types outside the precomputed vocabulary. mongodb datasets are served through the same endpoint (find + range, no aggregation). `include=` (or `include=all`) attaches a type-documents companion's ranked source documents per date (the provenance behind each entry). **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | query | string | no | `"wikimedia"` | Domain owning the dataset | | `dataset` | query | string | no | `"ngrams"` | Dataset ID within the domain | | `type` | query | string | yes | | The type/term to look up. Case-sensitive. | | `entity` | query | string or null | no | | Global entity ID (e.g. 'wikidata:Q30') or local ID. Omit for datasets without entity_mapping. | | `dates` | query | string or null | no | | Date range: a single date '2024-06-01' or 'start,end' like '2024-01-01,2024-12-31'. Omit for full history. | | `weight` | query | string or null | no | | Count measure — one of the dataset's endpoint_schema.count_column entries. Defaults to the first. | | `sparkline_dataset` | query | string or null | no | | Deprecated. The type-first sparkline companion is resolved from lineage; pass a dataset_id only to override, or '' to disable the fast path (dist-tree scan only). | | `include` | query | string or null | no | | Comma-separated provenance role(s) to attach per date (e.g. 'articles'), or 'all' for every declared companion. Roles are resolved from the primary's lineage; a raw type-documents dataset id also works (deprecated). | | `include_dates` | query | string or null | no | | Comma-separated exact dates to attach provenance for (e.g. the two comparison dates '2026-01-20,2026-01-21'). Narrows the ?include= documents only; the series itself keeps its dates range. Omit to attach documents for every date in range. | **Response** - `type` (string) — The type/term looked up (echoed back). - `latest_available_date` (string) — Most recent date with data for the resolved slice. - `series` (array of object) — One entry per date, chronological. `rank`/`freq` appear only when the dataset declares rank_column/freq_column. With `?include=`, each entry also carries a key named for the requested provenance role (e.g. `articles`) holding a ranked `[[document, score], ...]` list. - `date` (string) - `counts` (integer) — The selected measure for this type on this date. - `rank` (integer) — Rank under the chosen measure (omitted when undeclared). - `freq` (number) — Normalized frequency (omitted when undeclared). **Example response** ```json { "type": "trump", "latest_available_date": "2026-01-04", "series": [ { "date": "2025-12-05", "counts": 5352, "rank": 811, "freq": 0.000117 }, { "date": "2025-12-06", "counts": 4332, "rank": 847, "freq": 0.000112 } ] } ``` **Performance** - `fast_path`: Types in the sparkline vocabulary are a hash-bucket point lookup on the type-first companion (~tens of ms). - `slow_fallback`: Types outside it fall back to a scan of the date-first tree, bounded to the requested range and directory-pruned; an undated request is clamped to the slice availability, so the scan is never an open walk of the whole tree. A dataset that is itself type-first (orientation:type-first, e.g. a term-bucketed tree) has no slow path: every request is a bucket point lookup and a miss is an empty series. - `mixed_batch`: Batch requests scan only the types the sparkline missed: vocabulary types return fast regardless, out-of-vocabulary types add one scan for just those. - `mongodb`: Pass-through datasets (twitter) serve the range as a plain find + time filter + sort — a range read, not an aggregation. - `include`: ?include= adds one bucket-routed read per provenance companion; omit it for the tidy counts/rank/freq series. include_dates= narrows the documents to specific dates (e.g. the two comparison dates a UI actually renders) without touching the series range. **Usage notes** - `dataset`: Selected by ?domain=&dataset= (the caller-facing types-counts dataset). The type-first sparkline fast path is resolved automatically from lineage.derived_from + orientation:type-first — no param needed. ?sparkline_dataset= remains as a deprecated override. Filter dims (?n=&lang=, ?ngram_size=&granularity=) use the dataset's registered column names. - `include`: ?include= (or include=all) attaches a type-documents companion's ranked source documents per date, nested under a key named for the role (e.g. 'articles': [[url, score], ...]). Roles are declared on the companion and resolved via lineage; a raw companion dataset id also works (deprecated). - `formats`: Works across parquet_hive, flat parquet, and mongodb pass-through (per-term range reads). rank/freq are present only when the dataset registers rank_column/freq_column; otherwise the series is counts-only. ## GET /storywrangler/term-series/batch Term Series Batch Batch term-series — a map of type → series in one request. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | query | string | no | `"wikimedia"` | Domain owning the dataset | | `dataset` | query | string | no | `"ngrams"` | Dataset ID within the domain | | `types` | query | string | yes | | Comma-separated types, e.g. 'trump,covid,the'. Case-sensitive. | | `entity` | query | string or null | no | | Global entity ID or local ID. Omit for datasets without entity_mapping. | | `dates` | query | string or null | no | | Date range: a single date or 'start,end'. Omit for full history. | | `weight` | query | string or null | no | | Count measure — defaults to the first registered. | | `sparkline_dataset` | query | string or null | no | | Deprecated. The type-first sparkline companion is resolved from lineage; pass a dataset_id only to override, or '' to disable the fast path. | | `include` | query | string or null | no | | Comma-separated provenance role(s) to attach per date, or 'all'. Resolved from the primary's lineage; a raw type-documents dataset id also works (deprecated). | | `include_dates` | query | string or null | no | | Comma-separated exact dates to attach provenance for (e.g. the two comparison dates '2026-01-20,2026-01-21'). Narrows the ?include= documents only; the series itself keeps its dates range. Omit to attach documents for every date in range. | ## GET /storywrangler/allotax Allotaxonometer Compares two type-frequency systems using the allotaxonometer (rank-turbulence divergence). Each system is defined by a dataset, an optional entity, a date range, and optional filter values. The two systems may differ on any combination of axes: - **entity vs entity** — e.g. US Wikipedia vs UK Wikipedia - **dates vs dates** — e.g. October vs November - **filter-only** — e.g. `sex=M` vs `sex2=F` (skipping entity registry) > **Filter dimensions** — look up a dataset's available filter dimensions via > `GET /registry/{domain}/{dataset_id}` (`transform.filter_dimensions`). > Pass them as extra query params using the `dim` / `dim2` suffix convention: > `?sex=M&sex2=F` compares boy vs girl babynames, `?geo=US&geo2=CA` compares countries. > Entity registration is optional when a filter dimension serves as the comparison axis. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | query | string | no | `"wikimedia"` | Domain owning the dataset | | `dataset` | query | string | no | `"ngrams"` | Dataset ID within the domain | | `entity` | query | string or null | no | | Global entity ID for system 1, e.g. 'wikidata:Q30' (United States). Optional — omit for datasets using filter_dimensions as the comparison axis. | | `entity2` | query | string or null | no | | Global entity ID for system 2, e.g. 'wikidata:Q145' (United Kingdom). Optional — omit for datasets using filter_dimensions as the comparison axis. | | `dates` | query | string or null | no | | Date/year range for system 1. Single value '2024-10-01' or range '2024-10-01,2024-10-31'. Omit to load all time. | | `dates2` | query | string or null | no | | Date/year range for system 2. Omit to load all time. | | `alpha` | query | string | no | `"1.0"` | RTD alpha parameter (number or 'inf') | | `alphas` | query | string or null | no | | Comma-separated alphas for multi-alpha mode, e.g. '0.5,1.0,inf' | | `weight` | query | string or null | no | | Count measure for both systems — one of the dataset's endpoint_schema.count_column entries. Defaults to the first registered measure. | | `ngram_limit` | query | integer | no | `10000` | Max types to load per system before computing | | `wordshift_limit` | query | integer | no | `200` | Truncate wordshift output to top N entries | **Response** - `normalization` (number) — Normalization constant for the rank-turbulence divergence - `delta_sum` (number) — Sum of normalized divergence elements — the actual D_alpha^R value - `diamond_counts` (array) — 2D rank-space histogram used to render the diamond plot - `max_delta_loss` (number) — Maximum delta-loss value (used for color-scale normalization) - `ncells` (integer) — Number of cells along one side of the diamond grid; use to size the band scale - `maxlog10` (number) — Largest log10(rank) across both systems, rounded up to at least 1; use to label diamond axes - `alpha` (number) — Alpha parameter used in the computation - `balance` (number) — Balance measure between the two systems (0.5 = equal, >0.5 = system 2 dominates) - `wordshift` (array of object) — Top contributing types, sorted by absolute divergence contribution. - `type` (string) — The n-gram / token - `rank1` (integer) — Rank in system 1 (0 = absent) - `rank2` (integer) — Rank in system 2 (0 = absent) - `score` (number) — Signed divergence contribution (positive = system 2 favours this type) - `meta` (object) — Request metadata echoed back - `system1` (object) — System 1 parameters: entity, dates, filters, type count - `system2` (object) — System 2 parameters: entity, dates, filters, type count - `domain` (string) — Dataset domain - `dataset` (string) — Dataset ID - `granularity` (string) — Granularity used **Example response** ```json { "normalization": 0.9871, "diamond_counts": [ [ 0, 1, 0 ], [ 2, 5, 3 ], [ 1, 4, 2 ] ], "max_delta_loss": 0.0421, "alpha": 1, "balance": 0.523, "wordshift": [ { "type": "COVID", "rank1": 850, "rank2": 45, "score": 0.0189 }, { "type": "election", "rank1": 1200, "rank2": 78, "score": 0.0142 }, { "type": "the", "rank1": 1, "rank2": 2, "score": -0.0021 } ], "meta": { "system1": { "entity": "wikidata:Q30", "dates": "2024-10-01,2024-10-31", "filters": {}, "types": 50000 }, "system2": { "entity": "wikidata:Q145", "dates": "2024-11-01,2024-11-30", "filters": {}, "types": 48000 }, "domain": "wikimedia", "dataset": "ngrams", "granularity": "daily" } } ``` ## GET /storywrangler/rtd Rank Turbulence Divergence Lightweight rank-turbulence divergence between two dates for a single entity. Returns per-term signed divergence contributions (wordshift) without the full allotaxonometer overhead (no diamond plot, no balance). Designed for fast on-the-fly comparisons (~80ms). Positive divergence = term is more prominent on the target date. Both `dates` and `dates2` must be provided. The frontend should compute valid dates from manifest.availability metadata. **Filter dimensions** are passed as extra query params (not listed above). Look up available filters via `GET /registry/{domain}/{dataset_id}` (`transform.filter_dimensions`). Use the `dim` / `dim2` suffix convention: `?sex=M` filters both systems, `?sex=M&sex2=F` compares across filter values. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | query | string | no | `"wikimedia"` | Domain owning the dataset | | `dataset` | query | string | no | `"ngrams"` | Dataset ID within the domain | | `entity` | query | string or null | no | | Global entity ID, e.g. 'wikidata:Q30' (United States) | | `dates` | query | string or null | no | | Target date, e.g. '2026-02-17' | | `dates2` | query | string or null | no | | Reference date, e.g. '2026-02-10' | | `alpha` | query | string | no | `"0.25"` | RTD alpha parameter (number or 'inf') | | `alphas` | query | string or null | no | | Comma-separated alphas for multi-alpha mode, e.g. '0.25,1.0,inf' | | `weight` | query | string or null | no | | Count measure for both systems — one of the dataset's endpoint_schema.count_column entries. Defaults to the first registered measure. | | `ngram_limit` | query | integer | no | `10000` | Max types to load per system (0 = no limit) | | `wordshift_limit` | query | integer | no | `10000` | Max wordshift entries to return (0 = no limit) | ## GET /storywrangler/wordshift Weighted Avg Wordshift Weighted-average sentiment word shift between two type-frequency systems. Scores each system's vocabulary with a bundled labMT happiness lexicon and returns each word's signed contribution to the change in average sentiment, plus the six component sums needed to render a shift graph. This is the sentiment analogue of `/rtd`: same data-loading path, a different instrument. **System 1 is the baseline; system 2 is read as a shift away from it.** The two systems may differ on any axis (like `/allotax`): - **entity vs entity** — e.g. US Wikipedia vs UK Wikipedia - **dates vs dates** — e.g. October vs November (omit `entity2` to reuse the entity) - **filter-only** — e.g. `?sex=M&sex2=F` Positive `shift_score` = the word pushed system 2's average sentiment up relative to system 1. `s_avg_1`/`s_avg_2` are the two weighted-mean scores. > **Lexicon** — wikipedia ngrams are English (enwiki), so `labMT_English` > is correct for every entity. Override `lexicon` only for corpora in > another language. labMT scores single words, so keep `ngram_size=1` > (the default); higher n-gram sizes match nothing and return an empty shift. > **Filter dimensions** — look up available filters via > `GET /registry/{domain}/{dataset_id}` (`transform.filter_dimensions`) and > pass them with the `dim` / `dim2` suffix convention. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | query | string | no | `"wikimedia"` | Domain owning the dataset | | `dataset` | query | string | no | `"ngrams"` | Dataset ID within the domain | | `entity` | query | string or null | no | | Global entity ID for system 1, e.g. 'wikidata:Q30' (United States). Optional — omit for datasets using filter_dimensions as the comparison axis. | | `entity2` | query | string or null | no | | Global entity ID for system 2, e.g. 'wikidata:Q145' (United Kingdom). Omit to reuse system 1's entity (e.g. date-vs-date). | | `dates` | query | string or null | no | | Date/year range for system 1. Single value '2024-10-01' or range '2024-10-01,2024-10-31'. Omit to load all time. | | `dates2` | query | string or null | no | | Date/year range for system 2. Omit to load all time. | | `lexicon` | query | string | no | `"labMT_English"` | labMT language lexicon: labMT_English, labMT_French, labMT_German, labMT_Spanish, labMT_Portuguese, labMT_Russian, labMT_Chinese, labMT_Hindi, labMT_Indonesian, labMT_Korean (short names like 'English' also accepted). | | `reference_value` | query | string or null | no | | Score partition point: omit for system 1's frequency-weighted mean (the baseline), 'average' (equivalent), or a number like 5.0 (labMT's neutral midpoint). | | `weight` | query | string or null | no | | Count measure for both systems — one of the dataset's endpoint_schema.count_column entries. Defaults to the first registered measure. | | `ngram_limit` | query | integer | no | `10000` | Max types to load per system before computing (0 = no limit) | | `wordshift_limit` | query | integer | no | `200` | Truncate per-word output to the top N by |shift| (0 = all). Component sums are always computed over the full vocabulary. | | `stop_lens` | query | string or null | no | | Neutral-word filter 'lo,hi' (e.g. '4,6'): drop words whose labMT score falls in the closed interval [lo, hi] before computing the shift, renormalizing over the survivors. Omit to keep every scored word. Bounds may be given in either order. | | `stop_words` | query | string or null | no | | Comma-separated words to exclude from the shift entirely regardless of score (e.g. 'rt,amp'). Omit to exclude nothing. | **Response** - `entries` (array of object) — Per-word contributions, sorted by absolute shift score descending. Truncated to wordshift_limit; component sums are always over the full vocabulary. - `type` (string) — The word (labMT-scored type) - `p_diff` (number) — Change in relative frequency, p_2 - p_1 (over the surviving scored vocabulary) - `s_diff` (number) — Change in score, s_2 - s_1 — always 0 for a single lexicon - `p_avg` (number) — Mean relative frequency, 0.5 * (p_1 + p_2) - `s_ref_diff` (number) — Deviation of the word's score from the reference, s - s_ref - `shift_score` (number) — Normalized signed contribution (positive = pushed system 2's average sentiment up) - `component_sums` (object) — Cumulative sign-quadrant contributions for the stacked total bars (Shifterator's component sums, normalized). pos_s/neg_s carry the s_diff term and are 0 for a single lexicon. - `pos_s_pos_p` (number) - `pos_s_neg_p` (number) - `neg_s_pos_p` (number) - `neg_s_neg_p` (number) - `pos_s` (number) - `neg_s` (number) - `total_diff` (number) — Unnormalized sum of raw shift scores (s_avg_2 - s_avg_1) - `norm` (number) — Normalization denominator (Σ|shift|) applied to all scores - `s_avg_1` (number) — Frequency-weighted mean labMT score of system 1 (Φ_avg) - `s_avg_2` (number) — Frequency-weighted mean labMT score of system 2 - `reference_value` (number) — Reference score partitioning positive/negative regimes (system 1's weighted mean unless overridden) - `normalization` (string) — Normalization scheme; always 'variation' - `meta` (object) — Request metadata echoed back - `system1` (object) — System 1 parameters: entity, dates, filters, type count - `system2` (object) — System 2 parameters: entity, dates, filters, type count - `lexicon` (string) — labMT language lexicon used to score both systems - `weight` (string) — Count column used - `stop_lens` (array of number) — Neutral-word lens applied as [lo, hi], or null. Words scored inside [lo, hi] were dropped before computing the shift. - `stop_words` (array of string) — Words excluded from the shift (sorted), or null. - `domain` (string) — Dataset domain - `dataset` (string) — Dataset ID - `dataset_version` (string) — Registered dataset version served - `wordshift_version` (string) — wordshift package version that computed the shift **Example response** ```json { "entries": [ { "type": "happy", "p_diff": 0.00042, "s_diff": 0, "p_avg": 0.0011, "s_ref_diff": 3.32, "shift_score": 0.0184 }, { "type": "crisis", "p_diff": -0.00031, "s_diff": 0, "p_avg": 0.0008, "s_ref_diff": -2.67, "shift_score": 0.0121 } ], "component_sums": { "pos_s_pos_p": 0.31, "pos_s_neg_p": -0.12, "neg_s_pos_p": -0.09, "neg_s_neg_p": 0.24, "pos_s": 0, "neg_s": 0 }, "total_diff": 0.043, "norm": 1.87, "s_avg_1": 5.42, "s_avg_2": 5.46, "reference_value": 5.42, "normalization": "variation", "meta": { "system1": { "entity": "Australia", "dates": "2026-07-18", "filters": { "ngram_size": 1, "granularity": "daily" }, "types": 4540 }, "system2": { "entity": "Canada", "dates": "2026-07-18", "filters": { "ngram_size": 1, "granularity": "daily" }, "types": 4531 }, "lexicon": "labMT_English", "weight": "count", "stop_lens": [ 4, 6 ], "domain": "wikimedia", "dataset": "ngrams", "dataset_version": "1.0.0", "wordshift_version": "0.1.1" } } ``` **Usage notes** - `0`: F - `1`: i - `2`: l - `3`: t - `4`: e - `5`: r - `6`: s - `7`: - `8`: a - `9`: r - `10`: e - `11`: - `12`: o - `13`: f - `14`: f - `15`: - `16`: b - `17`: y - `18`: - `19`: d - `20`: e - `21`: f - `22`: a - `23`: u - `24`: l - `25`: t - `26`: . - `27`: - `28`: T - `29`: h - `30`: e - `31`: - `32`: c - `33`: o - `34`: n - `35`: v - `36`: e - `37`: n - `38`: t - `39`: i - `40`: o - `41`: n - `42`: a - `43`: l - `44`: - `45`: l - `46`: a - `47`: b - `48`: M - `49`: T - `50`: - `51`: n - `52`: e - `53`: u - `54`: t - `55`: r - `56`: a - `57`: l - `58`: - `59`: l - `60`: e - `61`: n - `62`: s - `63`: - `64`: i - `65`: s - `66`: - `67`: a - `68`: p - `69`: p - `70`: l - `71`: i - `72`: e - `73`: d - `74`: - `75`: c - `76`: l - `77`: i - `78`: e - `79`: n - `80`: t - `81`: - - `82`: s - `83`: i - `84`: d - `85`: e - `86`: - `87`: b - `88`: y - `89`: - `90`: p - `91`: a - `92`: s - `93`: s - `94`: i - `95`: n - `96`: g - `97`: - `98`: s - `99`: t - `100`: o - `101`: p - `102`: _ - `103`: l - `104`: e - `105`: n - `106`: s - `107`: = - `108`: 4 - `109`: , - `110`: 6 - `111`: - `112`: — - `113`: - `114`: t - `115`: h - `116`: e - `117`: - `118`: e - `119`: n - `120`: d - `121`: p - `122`: o - `123`: i - `124`: n - `125`: t - `126`: - `127`: d - `128`: o - `129`: e - `130`: s - `131`: - `132`: n - `133`: o - `134`: t - `135`: - `136`: b - `137`: a - `138`: k - `139`: e - `140`: - `141`: i - `142`: n - `143`: - `144`: a - `145`: - `146`: d - `147`: e - `148`: f - `149`: a - `150`: u - `151`: l - `152`: t - `153`: - `154`: s - `155`: o - `156`: - `157`: c - `158`: a - `159`: l - `160`: l - `161`: e - `162`: r - `163`: s - `164`: - `165`: s - `166`: t - `167`: a - `168`: y - `169`: - `170`: i - `171`: n - `172`: - `173`: c - `174`: o - `175`: n - `176`: t - `177`: r - `178`: o - `179`: l - `180`: . - `181`: - `182`: m - `183`: e - `184`: t - `185`: a - `186`: . - `187`: s - `188`: t - `189`: o - `190`: p - `191`: _ - `192`: l - `193`: e - `194`: n - `195`: s - `196`: - `197`: / - `198`: - `199`: m - `200`: e - `201`: t - `202`: a - `203`: . - `204`: s - `205`: t - `206`: o - `207`: p - `208`: _ - `209`: w - `210`: o - `211`: r - `212`: d - `213`: s - `214`: - `215`: e - `216`: c - `217`: h - `218`: o - `219`: - `220`: w - `221`: h - `222`: a - `223`: t - `224`: e - `225`: v - `226`: e - `227`: r - `228`: - `229`: w - `230`: a - `231`: s - `232`: - `233`: a - `234`: p - `235`: p - `236`: l - `237`: i - `238`: e - `239`: d - `240`: , - `241`: - `242`: f - `243`: o - `244`: r - `245`: - `246`: r - `247`: e - `248`: p - `249`: r - `250`: o - `251`: d - `252`: u - `253`: c - `254`: i - `255`: b - `256`: l - `257`: e - `258`: - `259`: c - `260`: a - `261`: p - `262`: t - `263`: i - `264`: o - `265`: n - `266`: s - `267`: . --- # API reference: wikimedia Generated from the live OpenAPI spec of the Storywrangler API. ## GET /wikimedia/revisions List Revision Articles List articles with extracted revision histories. Uses the pre-computed article_index from manifest.partition_index, populated at registration time by the submit script. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `min_revisions` | query | integer | no | `1` | Minimum revision count filter | | `limit` | query | integer | no | `100` | Max articles to return | **Response** - `articles` (array of object) — Articles with extracted revision histories - `identifier` (string) — Article identifier (slug) - `revision_count` (integer) — Number of revisions extracted - `total` (integer) — Total number of matching articles returned **Example response** ```json { "articles": [ { "identifier": "Cat", "revision_count": 142 }, { "identifier": "Dog", "revision_count": 98 } ], "total": 2 } ``` ## GET /wikimedia/revisions/{identifier} Get Revision Deltas Delta-encoded revision history for one article. Returns one entry per revision. The first revision (revision_idx=0) contains the full token map. Subsequent revisions contain only changed tokens (value 0 = token removed). **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `identifier` | path | string | yes | | | **Response** - `revisions` (array of object) — Ordered revision history (oldest first). First entry is the full token map; subsequent entries contain only changed tokens. - `revision_id` (string) — Wikipedia revision ID - `name` (string) — Article title - `date_modified` (string) — ISO 8601 modification date - `revision_comment` (string) — Edit summary - `categories` (array) — List of article categories - `token_diff` (string) — JSON-encoded delta map: token → new count (0 = removed) **Example response** ```json { "revisions": [ { "revision_id": "1234567890", "name": "Cat", "date_modified": "2024-01-15", "revision_comment": "/* Breeds */ Added Persian section", "categories": [ "Cats", "Mammals", "Pets" ], "token_diff": "{\"cat\": 3, \"breed\": 5, \"persian\": 1}" } ] } ``` ## GET /wikimedia/precomputed-rtd Precomputed Rtd Precomputed Rank Turbulence Divergence for a single entity and date. Returns the top divergent terms between two consecutive time periods, sorted by absolute divergence descending. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `entity` | query | string | yes | | Global entity ID, e.g. 'wikidata:Q30' | | `date` | query | string | yes | | Reference date (YYYY-MM-DD) | | `date_delta` | query | integer | no | `1` | Days between the two compared systems | | `granularity` | query | string | no | `"daily"` | Granularity: daily | weekly | monthly | | `ngram_size` | query | integer or null | no | | N-gram size (1 = unigrams, 2 = bigrams) — the registered column name. | | `n` | query | integer | no | `1` | Deprecated alias for ngram_size. | | `alpha` | query | number | no | `0.17` | RTD alpha parameter | | `limit` | query | integer | no | `200` | Top N terms by absolute divergence | ## GET /wikimedia/semantic-timeseries Semantic Timeseries Daily lexicon-scored time series for one country's pageview-weighted corpus. Returns the full history: one entry per day with the labMT happiness score (avg_happs), ousiometric power/danger/structure scores, and the pageview-weighted word-count denominators behind each average. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `country` | query | string | no | `"United States"` | Country name as stored in the data (e.g. 'United States'), or 'All' for the global pageview-weighted corpus | ## GET /wikimedia/semantic-ngrams Semantic Ngrams Per-word labMT counts for one country and day — the word shift graph's input. Returns the row for (country, date): the daily lexicon scores plus `count`, a map of labMT word → pageview-weighted count for every labMT word that appeared that day. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `country` | query | string | no | `"United States"` | Country name as stored in the data (e.g. 'United States'), or 'All' for the global pageview-weighted corpus | | `date` | query | string | yes | | Date (YYYY-MM-DD) | | `granularity` | query | string | no | `"daily"` | daily, weekly, or monthly | --- # API reference: open-academic-analytics Generated from the live OpenAPI spec of the Storywrangler API. ## GET /open-academic-analytics/academic-research-groups Get Academic Research Groups Get the UVM faculty roster with OpenAlex IDs and research group metadata. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `inst_ipeds_id` | query | string or null | no | | Filter by institution IPEDS ID, e.g. '231174' for UVM | | `payroll_year` | query | integer or null | no | | Filter by payroll year, e.g. 2023 | ## GET /open-academic-analytics/authors Get All Authors Get all authors with current age, last publication year, and research group status. ## GET /open-academic-analytics/coauthors/{author_name} Get Coauthors For Author Get coauthor data for a specific author. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `author_name` | path | string | yes | | | | `filter_big_papers` | query | boolean | no | `false` | Filter out papers with >25 coauthors | | `limit` | query | integer or null | no | | Limit number of results | ## GET /open-academic-analytics/embeddings Get Embeddings Data Papers with UMAP embeddings and department metadata for visualization. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `limit` | query | integer | no | `6000` | Number of papers to sample | ## GET /open-academic-analytics/papers/{author_name} Get Papers For Author Get processed papers for a specific author. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `author_name` | path | string | yes | | | | `filter_big_papers` | query | boolean | no | `false` | Filter out papers with >25 coauthors | | `limit` | query | integer or null | no | | Limit number of results | ## GET /open-academic-analytics/training/{author_name} Get Training Data Aggregated training data for change point analysis. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `author_name` | path | string | yes | | | --- # API reference: scisciDB Generated from the live OpenAPI spec of the Storywrangler API. ## GET /scisciDB/metrics Get Metrics Flexible time-series query over a registered scisciDB metrics dataset. Specify which dimensions to aggregate with `group_by`, and pass any declared filter or partition dimensions as extra query params to narrow the result. Comma-separated values are supported for multi-value filtering (IN clause). Partition dimensions (e.g. metric_type) have safe defaults injected when omitted from both group_by and filters — prevents accidental cross-partition aggregation (e.g. summing total + has_abstract would double-count). Examples: ?group_by=field,year&metric_type=total ?group_by=venue,year&field=Computer+Science&metric_type=total ?group_by=venue,metric_type&venue=Nature,Science&field=Computer+Science **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `group_by` | query | string | yes | | Comma-separated columns to GROUP BY, e.g. 'field,year' | | `dataset` | query | string | no | `"field-venue-metrics"` | Registered dataset ID within scisciDB | | `start_year` | query | integer or null | no | | | | `end_year` | query | integer or null | no | | | | `exclude_nulls` | query | boolean | no | `true` | Exclude rows where any group_by column is NULL | | `top_n` | query | integer or null | no | | Return only the top N groups by total count (non-time dimensions) | | `limit` | query | integer | no | `1000` | | --- # API reference: health Generated from the live OpenAPI spec of the Storywrangler API. ## GET /health/status Get Current Status Current health status for all datasets. Returns the most recent probe result for each (domain, dataset_id). ## GET /health/status/history Get Status History Daily health history for all datasets over the past N days. Returns one row per (domain, dataset_id, day) with the worst status that day and the average latency. Powers the heatmap grid. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `days` | query | integer | no | `90` | | ## GET /health/status/{domain}/{dataset_id} Get Dataset Health Detail Recent check history for a single dataset. **Parameters** | Name | In | Type | Required | Default | Description | | --- | --- | --- | --- | --- | --- | | `domain` | path | string | yes | | | | `dataset_id` | path | string | yes | | | | `limit` | query | integer | no | `100` | | --- # API reference: platform Generated from the live OpenAPI spec of the Storywrangler API. ## GET /version Platform Version Return the versions of all platform components. Useful for debugging, reproducibility, and pinning API clients to a known software stack. The `schemas` version records the registration contract in effect; `duckdb`, `allotax`, and `wordshift` versions govern query results.