Storywrangler is a decentralized data catalog for complex system instruments and data governance
By register your datasets to the Storywrangler platform, you first gain access to cool instruments out of the box. Built at the Vermont Complex Systems Institute to study collective attention as ecological timeseries, while improving data discoverability, ownership, and lineage tracking.
Text as ecological signal
Storywrangler is hosting a set of tools to facilitate the study of large-scale text corpora. Text produced on social media (Bluesky, Reddit, Twitter), news outlets, Wikipedia, and higher education are treated as ecological time series — living records of how collective attention shifts across populations and over time.
Storywrangler's architecture
Submitters write datasets as parquet files to shared storage and register their metadata via a simple POST request. The API validates schema compatibility and availability, wires datasets to instruments automatically where applicable, and records ownership, lineage, and discoverability.
Submitter's Pipeline
Storywrangler Catalog
- Schema validation on register
- Instrument wiring
- Ownership & lineage tracking
Storage: Parquet
Columnar Parquet files owned and managed by submitters. Supports flat files and hive-partitioned trees.
Web Applications
Feeds downstream applications like complex-stories and wikimedia.uvm.edu, and surfaces queryable endpoints for any registered dataset.
Key features
Storywrangler is a digital commons where participants nurture a collective data garden, learn about each others' work, and share complex system tools.
Instrument-driven endpoints
The allotaxonometer and other VCSI tools become available the moment a dataset is registered. The schema contract is the wiring — no per-dataset integration work required.
Performance-Oriented
Built on Parquet and DuckDB to scale from megabytes to terabytes. Columnar storage, partition pruning, and in-process analytical queries keep latency low without infrastructure overhead.
Interoperability by Design
Datasets declare their query axes and output shape once. The platform bridges heterogeneous identifier namespaces — Wikidata, OpenAlex, local IDs — through a unified entity graph.
Selective Sharing & Succession
Fine-grained access control means datasets that cannot be fully open can still be shared: expose aggregate results only, a filtered row subset, or full access per collaborator. Ownership succession ensures datasets survive student turnover.
Discoverable Analysis
Search the registry to find datasets that are already instrument-ready. The schema contract tells you not just where the data lives, but what analyses are immediately available — discovery and reproducibility are the same guarantee.
Lineage & Impact
Downstream groups that build on your data register their dependency in the registry. Their work appears in your impact record automatically — research credit propagates without either group coordinating directly.
Registering your first data pipeline
Beta release — manual account provisioning
Account creation is not yet self-serve. To get access, contact the VCSI team to have an account created and your API key issued. The key should then be stored in your API_KEY environment variable.
The registration process is a simple POST request documented here.
We also provide an SDK to ease the use of the platform, which we recommend to install with uv (or pip):
uv init --python 3.12 # create environment
uv sync # creates the ~/.venv
uv add storywranglerOnce you have your username and password, call Storywrangler.login() (or POST /auth/login)
to get your api_key. Save that key — on subsequent runs you can pass it
directly or store it in the API_KEY environment variable to avoid logging in again. Using the python SDK:
# Import the SDK and the client module
from storywrangler import Storywrangler, DatasetCreate
# Connect to the Storywrangler API
client = Storywrangler(api_key='YOUR_API_KEY')
# A basic request to verify connection is working
me = client.users.whoami()
# Create a dataset
dataset = DatasetCreate(
catalog="vcsi",
domain="babynames",
dataset_id="ngrams",
data_location="/mydata/babynames.parquet",
data_format="parquet",
description="Babynames frequencies by year and sex in the US.",
endpoint_schema={"type": "types-counts"},
transform={"time_dimension": "year", "filter_dimensions": ["sex"]},
ownership={"owner_group": "vcsi", "contact": "vcsi@uvm.edu"},
lineage={"repo": "https://github.com/Vermont-Complex-Systems/babynames"}
)
# Register
client.registry.register(dataset)You can find a walkthrough of the types-counts API schema that the allotaxonometer expects in Registering a dataset. In this case, we are telling the API that the dataset has the following shape and is available at the data/ location:
types,counts,year,sex
John,4394,1925,M
Robert,2559,1925,M
Axell,1956,1925,M
Donald,1565,1925,M
Peter,1464,1925,M
...Provided the registration is successful, you can now share your analysis of babynames with anyone on earth using:
result = client.instrument.rtd(
domain="babynames",
dataset="ngrams",
entity="wikidata:Q30",
dates="1925",
dates2="2025",
sex="M",
)
print(result['wordshift'][:5]Running this command you'll see:
[{'type': 'Jackson',
'rank1': 676.0,
'rank2': 74.0,
'divergence': 0.00013999022173321902},
{'type': 'Duvall',
'rank1': 10309.5,
'rank2': 428.0,
'divergence': 0.0001209265034150297},
{'type': 'Bunny',
'rank1': 564.0,
'rank2': 5765.0,
'divergence': -9.604679669786964e-05},
{'type': 'Weaver',
'rank1': 8522.5,
'rank2': 736.0,
'divergence': 9.389480911305102e-05},
{'type': 'Bowl',
'rank1': 254.0,
'rank2': 1169.0,
'divergence': -8.660948752039387e-05}]
Under the hood, we are versioning the interaction of the allotaxonometer tool and the submitted babynames pipeline for reproducibility.
For more details on entity mapping, discoverability, and storage options, see Registering a dataset.