Storywrangler
Navigation

Weighted Avg Wordshift

Rust
get /storywrangler/wordshift

Weighted-average sentiment word shift between two type-frequency systems.

Scores each system's vocabulary with a bundled labMT happiness lexicon and returns each word's signed contribution to the change in average sentiment, plus the six component sums needed to render a shift graph. This is the sentiment analogue of /rtd: same data-loading path, a different instrument.

System 1 is the baseline; system 2 is read as a shift away from it. The two systems may differ on any axis (like /allotax):

  • entity vs entity — e.g. US Wikipedia vs UK Wikipedia
  • dates vs dates — e.g. October vs November (omit entity2 to reuse the entity)
  • filter-only — e.g. ?sex=M&sex2=F

Positive shift_score = the word pushed system 2's average sentiment up relative to system 1. s_avg_1/s_avg_2 are the two weighted-mean scores.

Lexicon — wikipedia ngrams are English (enwiki), so labMT_English is correct for every entity. Override lexicon only for corpora in another language. labMT scores single words, so keep ngram_size=1 (the default); higher n-gram sizes match nothing and return an empty shift.

Filter dimensions — look up available filters via GET /registry/{domain}/{dataset_id} (transform.filter_dimensions) and pass them with the dim / dim2 suffix convention.

Query Parameters

domain string

Domain owning the dataset

dataset string

Dataset ID within the domain

entity

Global entity ID for system 1, e.g. 'wikidata:Q30' (United States). Optional — omit for datasets using filter_dimensions as the comparison axis.

entity2

Global entity ID for system 2, e.g. 'wikidata:Q145' (United Kingdom). Omit to reuse system 1's entity (e.g. date-vs-date).

dates

Date/year range for system 1. Single value '2024-10-01' or range '2024-10-01,2024-10-31'. Omit to load all time.

dates2

Date/year range for system 2. Omit to load all time.

lexicon string

labMT language lexicon: labMT_English, labMT_French, labMT_German, labMT_Spanish, labMT_Portuguese, labMT_Russian, labMT_Chinese, labMT_Hindi, labMT_Indonesian, labMT_Korean (short names like 'English' also accepted).

reference_value

Score partition point: omit for system 1's frequency-weighted mean (the baseline), 'average' (equivalent), or a number like 5.0 (labMT's neutral midpoint).

weight

Count measure for both systems — one of the dataset's endpoint_schema.count_column entries. Defaults to the first registered measure.

ngram_limit integer

Max types to load per system before computing (0 = no limit)

wordshift_limit integer

Truncate per-word output to the top N by |shift| (0 = all). Component sums are always computed over the full vocabulary.

stop_lens

Neutral-word filter 'lo,hi' (e.g. '4,6'): drop words whose labMT score falls in the closed interval [lo, hi] before computing the shift, renormalizing over the survivors. Omit to keep every scored word. Bounds may be given in either order.

stop_words

Comma-separated words to exclude from the shift entirely regardless of score (e.g. 'rt,amp'). Omit to exclude nothing.

Available Datasets

Pass these as additional query parameters using the actual column names from each dataset.

babynames/ngrams

Baby names by popularity, year, and location with entity mappings

No filter dimensions.

bluesky/ngrams

Date-first ngram distributions for Bluesky — all count measures per (ngram, date), by language and ngram size, bundled one ISO week per file under year/month for single-date distribution and rank-divergence queries. Includes repost/quote attention-weighted measures (count_all). Companion to the hash-bucketed bluesky/sparklines per-term time series.

n default: 1 · 1, 2
lang default: en · af, als, am, an, ar …and 171 more
reddit/ngrams

Reddit ngrams

n default: 1 · 1, 2
lang default: af · af, als, am, an, ar …and 162 more
twitter/ngrams

Storywrangler Twitter n-grams (1/2/3-grams across 169 languages), served from MongoDB. One document per (word, day) with with-RT and no-RT counts, ranks, and frequencies.

No filter dimensions.

vt-zoning-atlas/ngrams

Vermont zoning bylaw word frequencies by town.

No filter dimensions.

wikimedia/ngrams

Wikipedia n-grams by frequency, date, and location with entity mappings and ranks

ngram_size default: 1 · 1, 2
granularity default: daily · daily, monthly, weekly

Response

entries array

Per-word contributions, sorted by absolute shift score descending. Truncated to wordshift_limit; component sums are always over the full vocabulary.

component_sums object

Cumulative sign-quadrant contributions for the stacked total bars (Shifterator's component sums, normalized). pos_s/neg_s carry the s_diff term and are 0 for a single lexicon.

total_diff number

Unnormalized sum of raw shift scores (s_avg_2 - s_avg_1)

norm number

Normalization denominator (Σ|shift|) applied to all scores

s_avg_1 number

Frequency-weighted mean labMT score of system 1 (Φ_avg)

s_avg_2 number

Frequency-weighted mean labMT score of system 2

reference_value number

Reference score partitioning positive/negative regimes (system 1's weighted mean unless overridden)

normalization string

Normalization scheme; always 'variation'

meta object

Request metadata echoed back

curl -X GET https://api.storywrangler.uvm.edu/storywrangler/wordshift \