Category: Data & Technology

Living on the Upper West Side, the bright lights of Times Square and Broadway feel close to home, a 15-minute B/C subway ride away. My kids are big fans of The Lion Guard on Disney+, so I knew The Lion King on Broadway would be a must-see for the family. Tickets aren’t cheap, so after striking out repeatedly on the lottery, I started looking for a programmatic way to find the best deal.

Digging through the fetch network logs, I found two useful endpoints powering the Broadway Direct interface:

1. api/events/getbymonth
2. api/eventinventory/{event_id}

The first endpoint returns a list of all shows within a supplied month. Beyond basic information like the show date, time, and location, the endpoint also returns how many tickets are still available for purchase (AvailabilityCount).

{
  "ID": 1032655,
  "InventoryLastUpdated": null,
  "SecondaryMarketEventId": null,
  "EventTemplateID": 13510,
  "EventTemplateName": "The Lion King",
  "EventTemplateStatementDescriptor": "The Lion King Tickets",
  "EventTemplateCode": "546654",
  "EventTemplateAlertMessage": "",
  "EventTemplateRequireAlertAcknowledgement": false,
  "LocalEndDate": "2025-03-02T16:00:00",
  "EndDate": "2025-03-02T21:00:00",
  "HasProducts": false,
  "Name": "The Lion King (New York, NY)",
  "AlternateName": "",
  "AwayTeam": null,
  "UseHeavyOnSale": false,
  "RedirectURL": null,
  "ShouldRedirect": false,
  "RequireRecaptcha": false,
  "UrlSafeName": "the-lion-king-new-york-ny-1032655",
  "Date": "2025-03-02T18:00:00",
  "LocalDate": "2025-03-02T13:00:00",
  "LocalOnSaleDate": "2024-06-05T10:00:00",
  "LocalOnSaleEndDate": "2025-03-02T12:30:00",
  "LocalAPIOnSaleDate": "2024-06-05T10:00:00",
  "HoursToEvent": 13,
  "TimeTBD": false,
  "DateTBD": false,
  "DateTBDDescription": null,
  "IsPresale": false,
  "IsBeforeSaleDate": false,
  "AvailabilityCount": 178,
  "AvailabilityColor": "Medium",
  "HideFeesUntilCheckout": false,
  "linkedEvent": null,
  "ShouldShowCheckInventoryLaterText": false,
  "eventTemplateType": "Standard",
  "UpfrontFeeDisclosure": true
}

Supplied with an event id, the second endpoint returns an array of price classes for a particular show. For example, a show on March 2 offered tickets across seven price classes: $287.50, $242.50, $169.50, $150.50, $135.50, $116.75, and $113.75.

{
  "Price": 105,
  "DisplayPrice": 116.75,
  "Class": "p5",
  "PriceLevelID": 53271,
  "DiscountedPrice": 105,
  "DiscountedDisplayPrice": 116.75,
  "DisplayRetailPrice": 116.75,
  "CommissionAmount": 0,
  "CompositePriceID": "105.00-105.00-False",
  "GASortOrder": 13,
  "DisplayName": "12",
  "FeeTotal": 11.75,
  "PriceKey": "53271--",
  "Commissionable": false,
  "Zone": "Mezz G-H",
  "RequirePackagePurchase": false,
  "HasPackages": false,
  "PackagePriceMaps": [],
  "SaleTypeName": "Standard",
  "ShowSaleTypeSeatIcon": false,
  "ExternalSystemData": null
}

Using these two endpoints (getbymonth + eventinventory), I pieced together a dataset representing all price classes for all shows in March, April, and May of 2025.

Then, to visualize the data, I collapsed the price classes to find the minimum, median, and maximum price point per show:

Based on this data, the cheapest ticket is usually $143, the median ticket is usually $181, and the most expensive ticket is usually $260. That means for a family of four, you should expect to shell out at least $500 to $700 for a single Lion King show during the spring season.

This whole exercise paints me as a cheap dad, concerned about price first and fun and childhood memories second, which is not really the case. That said, it’s satisfying to know I’m getting a good deal. 😉

Here’s the R code used to scrape the endpoints, build the dataset, and make the chart:

Web scraping is a useful tool for data practitioners, to state the obvious. Often, scraping is most valuable when performed on a scheduled basis, to incorporate new or refreshed values into the dataset.

In the past, I’ve paid a (small) monthly fee to PythonAnywhere to run scraping jobs. However, there’s a better, free alternative offered by a familiar platform: GitHub Actions. While GitHub Actions is largely designed for code deployment automation (testing pull requests, deploying merged pull requests to production) it can also be used to run jobs, including web scraping jobs.

This post walks through the implementation of a simple GitHub action, which scrapes the headline mortgage rates posted on Freddie Mac’s home page daily.

Setup

To get started, create a directory called .github/workflows in your repository. Within the .github/workflows directory, create a .yml file. This will contain the details of the action workflow.

The .yml file structure has two basic parts: on:specifies when the job should run, and jobs: defines what steps should be taken.

This action has been scheduled to run daily:

on:
  workflow_dispatch:
  schedule:
    - cron: '0 8 * * *'

Copy the scraper.yml, and modify as needed for your use case. Update the .py script and requirements.txt file in the root directory accordingly. This tutorial, as well as the official GitHub documentation, are good resources for building on this template.

Here, the python script is grabbing the mortgage rates posted on Freddie Mac’s homepage and saving them to a new .csv file.

Over time, enough snapshots accumulate to do something meaningful with this data!

Analysis

Let’s get a quick sense of how mortgage rates have changes since the action was first configured on December 18, 2021.

This .R script reads, joins, and cleans the saved files, and creates a trend plot:

The takeaway? It’s clear that mortgage rates are rapidly rising from their historically-low levels, propelled by the expectation of rate hikes by the Fed to counter inflation.

Thanks for following along. Check out this repo for all the components of the walkthrough.

Background

When the Church of Jesus Christ of Latter-Day Saints was organized in a small town in New York in 1830, there were only six members. Today, the Mormon Church has grown to over 16 million members, with congregations in 160 countries.

Leaders of the church have taught that this extraordinary growth is fulfillment of the Old Testament prophecy. Daniel 2:31–45 describes a stone “cut out of the mountain without hands” which would roll forth to fill the whole earth. Like the stone, the Church is prophesied to spread and fill “every nation, kindred, tongue, and people.” (See D&C 42:58)

Given the ambitious scope of that prophecy, it’s no wonder that many parties, inside and outside the organization, are interested in measuring and tracking the growth of the LDS Church. While high-level membership metrics are shared bi-annually by church leadership, country or state-specific trends are not provided. This project is an attempt to more precisely measure church growth by tracking changes in the number and distribution of meetinghouses and wards over time.

Wards are the basic organizational unit of the Church, e.g., a congregation.

DATA SOURCE

To help members or visitors locate worship services nearby, the Church provides a meetinghouse locator tool. After entering an address, the user is shown nearby meetinghouses and hours of ward worship services.

Since there are many thousands of meetinghouses owned by the Church across the world, it would be very difficult to collect meetinghouse and ward details manually. However, using the back-end web service that powers the meetinghouse locator, it’s possible to query the full list of meetinghouses, along with the ward units assigned to those meetinghouses. You can find a copy of the code used to extract and clean the data here.

Note: The meetinghouse locator is publicly available online and is not restricted to authenticated users. Consequently, the underlying meetinghouse data is presumed to be open and available for collection and analysis.

DATA Structure

There are currently two data outputs: (1) a list of the ~19,000+ meetinghouses owned or operated by the Church and (2) a list of ~30,000+ wards or other organizational units “assigned” to those meetinghouses. Below is a simple example to make this relationship clear:

Meetinghouses Table

Meetinghouse Assignments Table

In other words, the first table says there’s a meetinghouse (church building) on 6695 South 2200 West in West Jordan, UT. The second table says that there are two wards that meet in that building, the Colonial Park Ward and the Meadowland Ward.

Use Cases

The data described above represents a snapshot in time. It describes the distribution of meetinghouses and wards around the world at the moment the data is compiled, and could be used to answer these kinds of questions:

• How many meetinghouses are there currently in Sao Paulo, Brazil?
• How many wards are there currently in Coconut Creek, Florida?

While these are interesting questions, the bigger prize is understanding how the number of meetinghouses and wards in each country, state, or zip code is changing over time. To accomplish this, we need to capture and compare snapshots at regular intervals.

For example, by comparing the list of meetinghouses in January 2020 and June 2021, we could infer which meetinghouses are new, where they are located, which meetinghouses have been closed, and where they are located. Ultimately, this should serve as a kind of (imperfect) proxy for growth or migration effects within the church. My intention is to capture monthly snapshots of this data, and then stitch it together to analyze trends in growth (or decline).

Data download

Visit ldsmeetinghouses.com or the GitHub repo for the latest files for download.

Have a question, suggestion, or idea? Create a new issue via GitHub here.

Inspired in part by Python’s Beautiful Soup, the R package rvest makes it delightfully easy to scrape data from the web. As part of the Tidyverse collection of packages, rvest fits nicely within the broader data workflow:

In this post I’ll walk through an example of using rvest to compile a dataset of NBA player salaries. To follow along, create a free RStudio Cloud account to write and run R code and bookmark the SelectorGadget tool to help identify HTML/CSS tags.

Background

Professional athletes are paid handsomely for their highly-specialized skill sets, and NBA players are no exception. Take Steph Curry, the highest-paid player during the 2018-2019 season. During that season he brought in a cool $37M in salary.

ESPN publishes annual salary data going back to the 1999-2000 season. Rather than manually copying this data, which is spread across hundreds of web pages, we can write a script to compile the data automatically using rvest. The data can then be used to explore variation in salary over time, by team, and by position.

Page Structure

The first step in any web scraping project is to become familiar with the target site URL structure. In this case, ESPN has a different page for each season. For example, the link below contains the player salaries for the 2018-2019 season:

http://www.espn.com/nba/salaries/_/year/2019

Within each season, the salaries are spread across several sub-pages, with 40 players listed on each sub-page:

http://www.espn.com/nba/salaries/_/year/2019/page/2

This means our code will need to dynamically determine the number of sub-pages to loop through when scraping the player salaries from each season.

Code Walkthrough

With that background, let’s dive into the code! To get started, we’ll need three packages: rvest, tidyverse, and stringr:

Next, we’ll define a vector of season URLs to loop over:

Next comes the bulk of the code required to perform the scraping. Here we loop over each of the season URLs. By extracting the text from the .page-numbers element on each page, we can dynamically determine the number of sub-pages for each season.

The html_table() command from the rvest package detects and extracts the table of salaries from each sub-page.

The last step is to clean the raw scraping output by adding column names, removing extra rows, and splitting the “name” and “position” fields into two columns, since they were stored as a single column in the ESPN tables, separated by a comma.

The final dataset has 9,456 rows. Below are the first 10:

Visualization

With the hard part out of the way, let’s quickly explore the trend in player salaries. We’ll visualize the distribution of salary by season:

Notice how the right tail has grown over time. It appears that most of the growth in average player salary is being drive by salaries paid to superstars. This observation is consistent with a similar analysis of NBA salaries done by Dimitrije Curcic.

“The average salary in the NBA has increased more than 7x since the 1990-91 season. Median salary has slower growth than the average; this could suggest that the financial gap between the top talents and the rest is getting larger.”

Here’s the code used to create the chart.

Web scraping is an important tool for data analysis because it enables data collection at scale. Rvest streamlines the process, allowing you to quickly create novel datasets for analysis, like NBA player salaries.

Complete R Script: link
Final Dataset: link