The Washington Post: Reader Response Kit

Introduction

The Washington Post is a news organization that produces a lot of interactive journalism that is rich in graphics and data.

Many teams at the Post want to build interactive stories or games that store data keyed to a Post user that can be aggregated and returned. The Reader Response Kit stores and aggregates reader responses empowering various interactive story formats and games, like redistricting minigolf. It contextualizes the reader's response with other Post users' responses, giving the reader a sense of how their response compares to others, and ensures application state persistence across devices for logged-in users.

The goal of the Reader Response Kit is to provide a simple, flexible, and powerful way for news-developers or graphics teams to create a new project where they can easily request and aggregate a readers’ responses. We wanted to broaden the pool of who can contribute to the Post's interactive journalism by decreasing the costs of development and standardizing the tools used.

My role

I was one of four engineers on a nimble team that was tasked with exploring the concept behind, and building the Reader Response Kit. I made contributions to all parts of the project, but focused primarily on the Admin interface and the client-side SDK. Along with my team, I made contributions to the design of the architecture, and made important contributions about the workflow of the Reader Response Kit.

I am particularly proud of bringing in rapid-prototyping strategies to the team for this project, helping us to "Build a little, test a little, fix a little", a design methodology that emphasizes iterative development and testing. Instead of building a complete system from the outset, components are built and tested individually, allowing for early detection and correction of any design flaws. This approach can lead to faster and more cost-effective development by reducing the need for costly redesigns later in the process.

Ethos

Before we started building the Reader Response Kit, we laid down some foundational principles that tried to encapsulate the goals of the project.

• 🛠️ Enable teams to build freely: We wanted to empower our graphics and news-devs teams to build whatever they wanted without having to worry about the underlying infrastructure.
• 🔒 Bake in security by design: Ingesting user submitted data comes with security risks. We wanted to make sure that the Reader Response Kit was designed with security in mind.
• 🔄 Solve a small number of hard problems: We wanted to keep the scope of the project focused. It was important to us that the Reader Response Kit was not a monolithic framework that tried to solve many problems, but rather a simple tool that could be extended to solve more complex problems.
• 🚀 Build for scale: The Reader Response Kit is used on a variety of platforms and devices. We wanted to make sure that the Reader Response Kit was designed to be scalable and could handle the load of a large number of users.

Interface

The first problem we had to design for was the interface. We needed to create a simple and easy to use way for our graphics and news-devs teams to bootstrap their projects and configurations for the same. We debated creating a CLI tool that would create YAML configurations for activating the Reader Response Kit, but decided against it. Instead, we chose to create a KeystoneJS based admin for interfacing with the Reader Response Kit. This allowed our stakeholders, many of whom were not technical, to create and access configurations for the Reader Response Kit from the Washington Post's okta portal.

This admin provided our stakeholders with a simple way to create and manage existing projects. Creation of a new project entailed filling out a form with the project's name, description, and a description of the project's expected data schema (which is used to validate and sanitize the data submitted).

Backend Infrastructure

Powering the Reader Response Kit is a backend infrastructure that is responsible for ingesting, storing, and aggregating the reader responses. It is written in Node and uses a variety of AWS services to store, process, and serve the data. A high level overview of the architecture is shown below.

• API: Lambdas coupled with AWS API Gateway that handle authentication, insertion and retrieval of reader responses.
• Step Function: To aggregate reader responses across multiple projects, we use AWS Step Functions to orchestrate a workflow that ingests data from multiple projects, aggregates the data, and stores the result in the S3 bucket.
• Storage: We use S3 buckets that are thoroughly cached for speedy retrieval of aggregated reader responses. Project configurations are stored in a PostgreSQL database.

Client-side SDK

The graphics and news-devs teams at the Post use a React-based app to create all interactive reporting on the Washington Post. To facilitate easy integration of the Reader Response Kit into their projects, we created a Node.js based SDK that can be added to any project. The SDK is published to npm and can be installed using yarn or npm, and automatially handles the nitty-gritty of authenticating the User with the Washington Post's subscription service, and retrieving the User's responses from the Reader Response Kit backend.