Platform Engineering Tech Talk @TwitterSeattle

Every day Twitter’s distributed, real-time platform utilizes millions of cores to process hundreds of billions of requests and serve petabytes of data. The Platform Engineering team’s mission is to improve the scalability and capabilities of this infrastructure, while maintaining world-class availability. We continually make changes to allow us to reach more customers, handle more data, and process data faster with ever more powerful tools. Join us for an evening of food, drinks, and talks that offer a peek into the engineering challenges in our production systems.

Agenda:

Tech Talks:

Pod balancer, a balancing algorithm for Twitter storage clusters: Twitter's storage cluster contains thousands of servers, millions of shards and petabytes of data. We require a system that minimizes both time to recovery and overall cost of the cluster. We have developed a new algorithm, the Pod Balancer, which achieves these goals by sub-dividing the cluster into logical units called Pods. A Pod serves as a logic association between shards of data and physical machines. This allows us to divide and conquer the problem of balancing the whole cluster, so that we can have near optimal balance, without ever having to consider the placement of > 1M shards of data across > 1K machines. It also provides a constraint on the maximum amount of data movement that can be caused by a single machine failing or being added to the cluster. Finally, we have introduced two new metrics Replica Distribution Factor & Shard Overlapped Factor, which help us quantify how effectively we are minimizing time to recovery. We have deployed the Pod balancer to our production clusters and have observed an improvement in the cluster balance of 6x. This enables us to run the same workload with significantly fewer machines without sacrificing time to recovery. The net result is that we are saving the company money on an ongoing basis.

Flying Faster with Heron: Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real- time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage – all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This talk will present the design and implementation of the new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and we share our experiences from running Heron in production.

Open Source Data Processing at Scale: Twitter builds large-scale data processing pipelines on top of libraries owned by the Core Data Libraries team. A large number of production analytics jobs power business insights or product dashboards, and our team aims to improve the correctness and performance of these jobs with as little disturbance and as few regressions as possible. We also open source our work and commit to mature public APIs, so we must balance our changes against disruption the open source community. Learn about our big successes (and problems!) optimizing and hardening these production pipelines under constraints and how we contribute what we learn back to the open source world.

*Note: Press is not admitted into these events; please e-mail press@twitter.com if you have any questions. Twitter reserves the right to refuse entry to this event. Twitter also reserves the right to remove individuals from Twitter's premises for disruptive or disorderly behavior, or for any other reason in Twitter's sole discretion.