Scaling at Twitter with Drupal & Thrift

Adapted from our SXSW talk: http://schedule.sxsw.com/2014/events/event_IAP26451

We gave this talk at SXSW but we would like to adapt it for the DrupalCon audience. The main thrust of the presentation is how to connect to remote services using Apache Thrift, how Twitter does this today, and what the benefits may be. This talk is currently geared towards and advanced engineering audience and was presented in a long workshop format.

In this workshop, presented by Twitter's Gazebo team, participants will learn:

1. Why Twitter chose Drupal and how they use it
2. Apache Thrift and the advantages of using it
3. How to use Thrift to connect Drupal to a remote service

By the end of this workshop, participants will have created a Drupal app that uses Thrift to connect to remote service.

Twitter relies on its websites to help it reach every person on the planet. At this scale, Twitter needs a standard way to build, maintain, and operate these websites. The “Gazebo” was introduced as a shared platform and now powers about.twitter.com, engineering.twitter.com, business.twitter.com, and more.

A publishing tool and CMS for non-engineers, this Drupal-based platform allows Twitter to scale consistently across all its sites, leverage existing internal services, and maintain the required level of security.

The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between various languages. It is used at Twitter to provide interfaces for internal services to allow teams to easily connect and exchange data within our existing platform.

Thrift can seem difficult to setup and implement when compared to alternatives like REST, but there are a number of advantages. Because Thrift generates both client and server interfaces, the result is more consistency and safety with regard to data types and exception handling. In addition to HTTP, Thrift supports other protocols allowing services with large volumes of data to be routed more appropriately.

Participants will create new Drupal modules using the Form API and thrift module to capture form values and send those values to a remote service. These new modules will be used to illustrate transmitting data via Thrift and storing values in a remote key/value store instead of MySQL.

Schedule info
Experience level: 
Advanced
Drupal Version: 
Drupal 7.x