Querying Datasets#

The Storywrangler platform is a set of API endpoints that expose the metadata of datasets contributed by the community. This means reading the registry is cheap, as it is only metadata. The required metadata is specified according to the Storywrangler Specifications, which provide a formal standard by which we declare what metadata is necessary and the whereabouts of the data.

The workflow is discovery-first: read the registry to learn what a dataset can answer, then build the query from what you found. Queries built from the registry work the first time; queries built from assumptions come back empty.

Run it live#

The fastest way to learn the query workflow is to run it. These notebooks are executable copies of everything on this page:

• Getting started: registry exploration, dataset deep dive, first time series.
• Multiplatform: the same term across Twitter, Reddit, and Wikimedia, merged into one analysis.
• Instruments: allotaxonometer and wordshift on registered datasets.

They also live in the repo under notebooks/.

Step 1 - Discover datasets#

GET /registry/                     → all datasets (latest version each)

The per-dataset response contains everything needed to build a valid query. The registry can be glanced over from the API references page. For each domain, we list available datasets with query parameters and their description.

With the SDK, the same catalog is a dataframe away:

from storywrangler import Storywrangler

client = Storywrangler()
client.registry.list().df()

Datasets are identified by three-level naming: catalog/domain/dataset_id, e.g. vcsi/wikimedia/ngrams.

Step 2 - Dive into a dataset#

GET /registry/{domain}/{dataset}   → one dataset's metadata

The dataset-scoped client wraps this metadata and answers the three questions you need before querying:

wiki = client.dataset("wikimedia", "ngrams")

wiki.filters        # what axes can I slice on, with valid values and defaults
wiki.availability   # what time range exists, per entity and granularity
wiki.adapter.df()   # how do I name the entity: local names ↔ namespaced IDs

Filter names are per-dataset, not platform-wide: wikimedia/ngrams slices on ngram_size, reddit/ngrams on n, because that is what each pipeline registered. The registry is the ground truth, so check filters rather than guessing.

For entity datasets, adapter maps human-readable names to namespaced identifiers such as wikidata:Q30 (United States). Query endpoints accept either form.

Step 3 - Query#

Datasets registered with the types-counts endpoint schema get the generic query endpoints out of the box, no bespoke code needed:

# what was trending in a window
wiki.top_ngrams(
    entity="wikidata:Q30",
    dates="2025-01-01", dates2="2025-01-10",
    limit=10,
)

# one term through time
wiki.term_series(
    "Trump", entity="United States",
    dates="2024-09-01,2026-06-01",
).df()

Every response object has .df() to turn it into a pandas dataframe. The full parameter reference for each endpoint, including per-dataset extras such as include="articles" on Wikimedia, lives in the API reference.

Instruments#

The same discovery-first workflow drives the analysis endpoints: the allotaxonometer (rank-turbulence divergence between two systems), its lightweight rtd variant, and wordshift. They take the same entity, date, and filter arguments as the query endpoints. The instruments notebook runs all of them.

When results come back empty or slow#

• Empty result, no error. The query was valid SQL over data that is not there. Check that the dates fall inside availability for that entity, and that the entity name matches the adapter.
• 422 on a filter. The value is not in the registered filter_values for that dataset. Read the error message: it lists the valid values.
• Slow query. Constrain the date range. Full-history scans over large corpora can take minutes; availability tells you the tightest range worth asking for.

Agents get the same discovery workflow through the MCP server and llms.txt exports, so anything on this page works from Claude or any MCP client as well.