Storywrangler
Navigation
Claude generated — content in progress

Getting started

Storywrangler is a text-analysis data platform from the Vermont Complex Systems Institute. It serves n-gram frequencies, time series, and rank-turbulence divergence (allotaxonometry) over large parquet datasets — Wikipedia page views, Reddit comments, US baby names, zoning bylaws, academic publication data — through a single FastAPI service backed by DuckDB.

Core concepts

  • Domain — a top-level data family with its own router and endpoints (wikimedia, reddit, babynames, storywrangler, open-academic-analytics, scisciDB, vt-zoning-atlas). GET /registry/domains lists the valid ones.
  • Dataset — a registered parquet source inside a domain, identified as {domain}/{dataset_id} (e.g. wikimedia/ngrams). The registry stores its location, layout, slice axes, and introspected metadata.
  • Registry — the catalog. GET /registry/ lists every dataset with its level_order (hive nesting), filter_values (valid values per dimension), and endpoint_schema (output shape). This is the ground truth for what is queryable — always check it before constructing queries.
  • Instruments — the analysis endpoints layered on datasets: top n-grams, per-term time series, and the allotaxonometer (rank-turbulence divergence between two systems).
  • Entities — datasets are partitioned by an entity (a country, a subreddit, a town). Entities use namespaced identifiers such as wikidata:Q30 (United States). GET /registry/{domain}/{dataset_id}/adapter maps local IDs to canonical entity IDs and human-readable names.

Connecting

The API base URL comes from the STORYWRANGLER_URL environment variable and defaults to http://localhost:8000. Interactive OpenAPI docs live at the API's /docs; a machine-readable spec at /openapi.json.

This documentation site is also machine-readable: /llms.txt returns everything as plain markdown, and /sections.json lists every section with a per-section /{slug}/llms.txt export — designed for LLM agents working with the platform.

Read endpoints (registry lookups, domain queries) are public. Registration and admin operations require a Bearer API key — see authentication.

The Python SDK

from storywrangler import Storywrangler

client = Storywrangler(base_url="http://localhost:8000", api_key="<your-key>")

# Dataset-scoped client (recommended)
wiki = client.dataset("wikimedia", "ngrams")
wiki.filters        # {'ngram_size': {'default': 1, 'valid': [1, 2]}, 'granularity': {...}}
wiki.availability   # date ranges per entity, from manifest.availability

result = wiki.allotax(
    entity="wikidata:Q30", entity2="wikidata:Q145",
    dates="2026-05-01", dates2="2026-05-01",
    ngram_size=1, granularity="daily",
)

The SDK validates filter names and values against the registry before sending a request, so errors surface with actionable messages instead of empty results.

GET /version reports the API, schemas, DuckDB, and allotax versions in effect — useful for reproducibility notes in papers and pipelines.

Where to go next

  • Querying datasets — the discovery-first query workflow, instrument endpoints, and performance guidance.
  • Registering a dataset — how to publish a new parquet dataset to the platform, including the hive-partitioning convention.
  • Why Storywrangler? — the motivation behind the platform.