Agents & MCP#

---
name: storywrangler-submitter
description: Interactive guide for getting data onto the Storywrangler platform — writing or improving a DatasetCreate payload, deciding between serving through an existing endpoint type or disseminating behind a bespoke endpoint, validating before registration, and verifying what the server derived. Use whenever a user wants to submit, register, share, or publish a dataset on Storywrangler, needs help writing the registration payload, asks whether their data is submittable, or wants to fix or optimize an existing submission — even if they never say the word "submission". Not for building the extract/transform pipeline that produces the data.
---

# Submitting datasets to Storywrangler

The payload is built in conversation with its author: propose, confirm, then
build. Start from the smallest valid payload and add only what earns its place.

## The shared core (every submission)

Whatever else happens, every payload needs:

- **identity**: `catalog`, `domain`, `dataset_id` — check the registry first
  (`GET /registry/domains`, `list-datasets`): an existing entry means update,
  not create.
- **location**: `data_location` + `data_format` (`parquet` unless the data is
  a hive-partitioned tree).
- **governance**: `description`, `ownership.owner_group`, `ownership.contact`,
  `lineage.repo`.

An author with a single `data.parquet` should get from "here's my file" to a
valid payload in two or three questions. Governance is the only part they must
be asked; identity can usually be proposed from context.

## The fork: existing endpoint type, or dissemination?

One question then decides how much more of the contract matters:

**Existing endpoint type** — the data's shape matches a recurring endpoint
type (`types-counts`: rank distributions; `time-series`: tabular measures),
so generic endpoints can serve it directly. Declare `endpoint_schema` plus
what the type requires:

- `types-counts` — columns default to `types`/`counts`; declare
  `type_column`/`count_column` only when the data's names differ (declare
  around the data, never rename it). Requires one comparison axis:
  `entity_mapping` or `transform.filter_dimensions`.
- `time-series` — requires `transform.time_dimension` and at least one
  `filter_dimension`.

**Dissemination** — the platform hosts the data behind an endpoint written
*for* it (e.g. `/wikimedia/semantic-timeseries`, essentially
`SELECT * WHERE country = ?`). The endpoint hardcodes the data's shape, so no
`endpoint_schema` is needed and the author doesn't have to learn those fields
— the core payload is enough to register. Serving needs the bespoke endpoint
to exist: a PR to the Storywrangler repo, which can come after registration.

When the data plainly reduces to (type, count) rows or GROUP-BY-able
measures, recommend the endpoint-type path; wide bespoke shapes (score
columns, JSON maps) point to dissemination. Recommend, don't decide.

## Performance declarations (both paths, all opt-in)

Everything beyond the core exists for one reason: Storywrangler's query layer
is a thin DuckDB layer over parquet, and each declaration lets it read less.
Offer each as one question with its payoff attached:

- `transform.time_dimension` → time-range pruning, `?dates=` queries, and
  coverage auto-derived at registration.
- `transform.filter_dimensions` → categorical slicing on in-file columns
  (omitting the parameter = aggregate over all values).
- `entity_mapping` (+ `entities` rows when IDs are opaque) → `?entity=`
  resolution and cross-dataset joins — bespoke endpoints use it too.
- `data_format="parquet_hive"` → partition pruning at scale. Only now does
  layout matter: every level `col=val/`, `data_location` = the tree root. If
  the layout doesn't conform, propose a fix (e.g. a rename script) and wait —
  never restructure the author's data unasked.
- `transform.hash_bucket` → term-first lookup routing. Just the column name;
  the pipeline must assign rows with `storywrangler.hashing.assign_bucket()`.

Declare the minimum — the server derives the rest at registration
(`level_order`, `filter_values`, availability, bucket counts). Never
enumerate hive levels, never compute availability client-side. And guard the
boundary: `endpoint_schema` = what comes back; `transform` = which rows are
read.

## Running the conversation

- **Look at only what the current decision needs.** A single file →
  `DESCRIBE` it and move on. A directory tree's structure matters only once
  hive partitioning is on the table. Don't survey a deep tree to answer a
  question nobody asked.
- **One decision per question**, with a recommendation and its consequence.
  Use AskUserQuestion when available.
- **Stop when it's enough** — an optional field the author doesn't need is
  noise, not thoroughness.
- Field-level reference lives in the docs/MCP (`get-documentation`) — fetch
  it rather than guessing.

## Close the loop

- `validate-submission` (MCP) on every draft: real schema contract, mirrored
  server guards, on-disk layout checks. Show payload and result together;
  errors become the next question, not silent fixes.
- Register only on the author's explicit go. `version="latest"` is the
  mutable slot — routine re-runs are not version bumps; semver strings are
  immutable snapshots (409 on re-register), bumped only on interface changes.
  Never set `schema_version`.
- Verify: `GET /registry/{domain}/{dataset_id}` and check the derived fields.
  Empty `level_order`/`availability` means introspection failed (unreachable
  path, naming violation) — not success, even on a 200.

## Scope

- **No data yet** → settle the fork and (if applicable) the endpoint type
  before the pipeline is built — the shape is cheap to design in and
  expensive to retrofit. Pipeline craft itself is the `pipelines` guide
  (`get-documentation pipelines`), not this skill.
- **Existing submission to review or fix** → start at validate; walk back
  only where the errors point.
- **Generic ETL with no submission in it** → out of scope.

---
name: storywrangler-sveltekit
description: Use the Storywrangler registry to access the Vermont Complex Systems datasets and instruments from a SvelteKit frontend using remote functions. Discover what data is available, its shape, and how to use it out of the box, directly or through text-based instruments for studying collective attention such as the allotaxonometer and wordshift. Load this when building a SvelteKit app against Storywrangler to enforce type-safe queries, handle instrument errors, and keep the user in the loop on which dataset and whether to use an instrument. For deep query craft load storywrangler-analyst; for submitting data load storywrangler-submitter.
---

# Using Storywrangler from a SvelteKit app

Storywrangler is a data platform with two layers: a **registry of datasets**
(corpora like wikimedia n-grams, reddit, babynames) and **instruments** —
computational analyses layered on those datasets, like the allotaxonometer and
wordshift. Your app picks a dataset and either reads its data directly or runs
an instrument over it, then draws the result.

You never carry the specifics in your head — they live on the **MCP**, which has
two discovery surfaces:

- **Docs** (`list-sections`, then `get-documentation`) — the platform guides and
  the instrument / endpoint reference: what an instrument does, its parameters,
  response shape, examples, and use cases. The endpoint reference lives under
  `api-reference/…` (e.g. `api-reference/storywrangler`). Start here to
  understand *what exists and how it works*.
- **Registry** (`list-datasets`, then `get-dataset`) — the live catalog: which
  datasets exist, and for one dataset its queryable dimensions, valid filter
  values, and date availability. This is *what data exists and which values are
  valid*.

Two boundaries keep this skill from drifting:

- **SvelteKit mechanics** (remote functions, query vs prerender, validation,
  rendering) belong to the Svelte skills and MCP. That API is still evolving —
  get the current code from them, don't reproduce it here.
- **Instrument and dataset specifics** belong to the MCP surfaces above. Don't
  recite a fixed list of instruments or guess params — look them up.

What this skill owns is the Storywrangler practical layer, and keeping the
**user in the loop** on the decisions that are theirs: which dataset, and
whether to use an instrument at all (and which).

## Start from the registry — don't assume a dataset

When a request is vague about *which* data ("show storywrangler data", "add a
wordshift"), don't silently default to a corpus. List what exists with
`list-datasets`, surface the options, and let the user choose. Then `get-dataset`
gives that dataset's queryable dimensions, valid filter values, entity mapping,
date availability, and `endpoint_schema` (its output shape). Confirm the dataset
and the slice before wiring anything.

## Data directly, or an instrument?

Not every app needs an instrument — often the user just wants to read and show
the dataset's own data (counts, ranks, time series). An **instrument** is a
computational analysis layered on top (the allotaxonometer, wordshift, …).

Don't assume one, and don't recite a fixed menu — the available instruments and
their use cases live in the MCP. If the user isn't sure what to build, look them
up (`get-documentation`, `api-reference/…`) and **propose** the ones whose
documented use case fits their question; let them choose.

Once the target is settled, the lookup splits across the two MCP surfaces: the
endpoint's **parameter contract** (params, response shape, examples) comes from
the API reference, and the dataset's **valid values** (which entities exist,
which dates are available, which filter dims) come from `get-dataset`. Read
both; don't guess.

## Wiring the call — the Storywrangler-specific parts

Deliver the call as a SvelteKit remote function, but the *how* (the remote-
function API, argument validation, rendering) is the Svelte skills' job — and
that API is still moving, so take the current shape from them rather than from
memory. What stays true on the Storywrangler side:

- **Public read, no auth.** The data and instrument endpoints need no key to
  consume.
- **Map the error surface to UI.** Endpoints answer **404** (no data for that
  entity/date), **400** (bad filter value), **503** (instrument library down).
  Turn each into a real state, not a forever-spinner.
- **Discovery-first before hardcoding.** Resolve entities and check date
  availability (via the MCP, or `storywrangler-analyst`) before baking an
  `entity` or `date` in — otherwise the page 404s for no reason.
- **Large, sanitized payloads.** Responses can be big; bound them with the
  endpoint's limit params (documented in the API reference). The server rewrites
  non-finite numbers — `NaN` → `null`, `±Infinity` → the strings `"Infinity"` /
  `"-Infinity"` — so treat divergence / measure fields as possibly-string.