Storywrangler is a decentralized data catalog for complex system instruments and data governance

By registering your datasets to the Storywrangler platform, you gain access to instruments out of the box, while improving data discoverability, ownership, and lineage tracking. Built at the Vermont Complex Systems Institute to study collective attention as ecological time series.

Get started →

Text as ecological signal

Storywrangler hosts a set of tools to facilitate the study of large-scale text corpora. Text produced on social media (Bluesky, Reddit, Twitter), news outlets, Wikipedia, and higher education is treated as ecological time series: living records of how collective attention shifts across populations and over time.

Text sources flow into the Storywrangler platform and produce analytical instruments like time series and allotaxonometer visualizations

The registered datasets are exposed out of the box via the storywrangler API as well as the python SDK through the available instruments.

Query the registry →

Academia's tragedy of the commons

Storywrangler is a digital commons where participants nurture a collective data garden, learn about each other's work, and share complex system tools. Everyone benefits from shared datasets, but the work of maintaining them falls on whoever contributed. So academic data tends to get deposited, then go quiet. Storywrangler improves the return on contributing data: you get the instruments now and a dataset that outlives your PhD, while the community gains a living map of ownership and provenance.

See Manifesto → The Turing Way on digital commons ↗

Key features

Instrument-driven endpoints

The allotaxonometer and other VCSI tools become available the moment a dataset is registered. The schema contract is the wiring — no per-dataset integration work required.

Performance-Oriented

Built on Parquet and DuckDB to scale from megabytes to terabytes. Columnar storage, partition pruning, and in-process analytical queries keep latency low without infrastructure overhead.

Interoperability by Design

Datasets declare their query axes and output shape once. The platform bridges heterogeneous identifier namespaces — Wikidata, OpenAlex, local IDs — through a unified entity graph.

Selective Sharing & Succession

Fine-grained access control means datasets that cannot be fully open can still be shared: expose aggregate results only, a filtered row subset, or full access per collaborator. Ownership succession ensures datasets survive student turnover.

Discoverable Analysis

Search the registry to find datasets that are already instrument-ready. The schema contract tells you not just where the data lives, but what analyses are immediately available — discovery and reproducibility are the same guarantee.

Lineage & Impact

Downstream groups that build on your data register their dependency in the registry. Their work appears in your impact record automatically — research credit propagates without either group coordinating directly.

Storywrangler's architecture

Submitters write datasets as parquet files to shared storage and register their metadata via a simple POST request. The API validates schema compatibility and availability, wires datasets to instruments automatically where applicable, and records ownership, lineage, and discoverability.

Submitter's Pipeline

write

Storywrangler Catalog

Schema validation on register
Instrument wiring
Ownership & lineage tracking

Storage: Parquet

Columnar Parquet files owned and managed by submitters. Supports flat files and hive-partitioned trees.

Web Applications

Feeds downstream applications like complex-stories and wikimedia.uvm.edu, and surfaces queryable endpoints for any registered dataset.

Register a dataset

Beta release — manual account provisioning

Account creation is not yet self-serve. To get access, contact the VCSI team to have an account created and your API key issued. The key should then be stored in your API_KEY environment variable.

Register a parquet dataset with a single POST and, if you provided a valid payload, the instruments become automatically available. Using the python SDK with the uv package manager:

uv init --python 3.12 # create environment
uv sync # creates the ~/.venv
uv add storywrangler

The registering process is about specifying the payload:

from storywrangler import Storywrangler
client = Storywrangler(api_key="<your-key>")
client.registry.register({    
    "catalog": "vcsi",
    "domain": "wikimedia",
    "dataset_id": "ngrams",
    "data_format": "parquet",
    "data_location": "/netfiles/wikimedia/wiki-ngrams.parquet",
    "description": "Wikipedia n-gram frequencies by country and date.",
    "endpoint_schema": {"type": "types-counts"},
    "transform": {"filter_dimension": "country", "time_dimension": "date"},
    "ownership": {"owner_group": "vcsi", "contact": "compstorylab@uvm.edu"},
    "lineage":   {"repo": "https://gitlab.com/compstorylab/wikipedia-parsing"}
})

Once your data is registered, share its analysis with anyone on earth:

client.instrument.rtd(
    domain="wikimedia", dataset="ngrams",
    country1="Arlington",country2="Burlington",
    dates="2026-07-16"
)