# Querying Datasets

The Storywrangler platform is a set of API endpoints that expose the metadata of datasets contributed by the community. This means reading the registry is cheap, as it is only metadata. The required metadata is specified according to the Storywrangler [Specifications](/specification), which provide a formal standard by which we declare what metadata is necessary and the whereabouts of the data.

The workflow is discovery-first: read the registry to learn what a dataset can answer, then build the query from what you found. Queries built from the registry work the first time; queries built from assumptions come back empty.

## Run it live

The fastest way to learn the query workflow is to run it. These notebooks are executable copies of everything on this page:

- [Getting started](https://colab.research.google.com/drive/1j6nxG0iMFaGXBoK9iaxc-NQwckcQgz9F?usp=sharing): registry exploration, dataset deep dive, first time series.
- [Multiplatform](https://colab.research.google.com/drive/1I-PPb4kLXbVWApx-jn77pZOU0NVfIweV?usp=sharing): the same term across Twitter, Reddit, and Wikimedia, merged into one analysis.
- [Instruments](https://colab.research.google.com/drive/1cZq56-RbilZgaog39Zx-_qShDL_iML_u?usp=sharing): allotaxonometer and wordshift on registered datasets.

They also live in the repo under [`notebooks/`](https://github.com/Vermont-Complex-Systems/storywrangler/tree/main/notebooks).

## Step 1 - Discover datasets

```
GET /registry/                     → all datasets (latest version each)
```

The per-dataset response contains everything needed to build a valid query. The registry can be glanced over from the [API references](https://storywrangler.uvm.edu/api-reference) page. For each domain, we list available datasets with query parameters and their description.

With the SDK, the same catalog is a dataframe away:

```python
from storywrangler import Storywrangler

client = Storywrangler()
client.registry.list().df()
```

Datasets are identified by three-level naming: `catalog/domain/dataset_id`, e.g. `vcsi/wikimedia/ngrams`.

## Step 2 - Dive into a dataset

```
GET /registry/{domain}/{dataset}   → one dataset's metadata
```

The dataset-scoped client wraps this metadata and answers the three questions you need before querying:

```python
wiki = client.dataset("wikimedia", "ngrams")

wiki.filters        # what axes can I slice on, with valid values and defaults
wiki.availability   # what time range exists, per entity and granularity
wiki.adapter.df()   # how do I name the entity: local names ↔ namespaced IDs
```

Filter names are per-dataset, not platform-wide: `wikimedia/ngrams` slices on `ngram_size`, `reddit/ngrams` on `n`, because that is what each pipeline registered. The registry is the ground truth, so check `filters` rather than guessing.

For entity datasets, `adapter` maps human-readable names to namespaced identifiers such as `wikidata:Q30` (United States). Query endpoints accept either form.

## Step 3 - Query

Datasets registered with the `types-counts` endpoint schema get the generic query endpoints out of the box, no bespoke code needed:

```python
# what was trending in a window
wiki.top_ngrams(
    entity="wikidata:Q30",
    dates="2025-01-01", dates2="2025-01-10",
    limit=10,
)

# one term through time
wiki.term_series(
    "Trump", entity="United States",
    dates="2024-09-01,2026-06-01",
).df()
```

Every response object has `.df()` to turn it into a pandas dataframe. The full parameter reference for each endpoint, including per-dataset extras such as `include="articles"` on Wikimedia, lives in the [API reference](https://storywrangler.uvm.edu/api-reference).

## Instruments

The same discovery-first workflow drives the analysis endpoints: the [allotaxonometer](/tools/allotaxonometer) (rank-turbulence divergence between two systems), its lightweight `rtd` variant, and [wordshift](/tools/wordshift). They take the same entity, date, and filter arguments as the query endpoints. The [instruments notebook](https://colab.research.google.com/drive/1cZq56-RbilZgaog39Zx-_qShDL_iML_u?usp=sharing) runs all of them.

## When results come back empty or slow

- **Empty result, no error.** The query was valid SQL over data that is not there. Check that the dates fall inside `availability` for that entity, and that the entity name matches the `adapter`.
- **422 on a filter.** The value is not in the registered `filter_values` for that dataset. Read the error message: it lists the valid values.
- **Slow query.** Constrain the date range. Full-history scans over large corpora can take minutes; `availability` tells you the tightest range worth asking for.

Agents get the same discovery workflow through the [MCP server and llms.txt exports](/llms), so anything on this page works from Claude or any MCP client as well.