Curated by Igor Brigadir and Vicki Boykis.
An annotated look through the release of the Twitter algorithm, through the context of engineering and recsys, with notes from repo creators on significance of specific parts of the code. Since it can be hard to parse through so much code and derive meaning and context, we do it for you!
This code focuses on the services used to build the Home timeline For You feed, the algorithmic tab that is now served first on both web and mobile next to the Following feed.
We're happy to take changes that add and contextualize Twitter's recommendations algorithm as it's been released over the past week. To contribute, please submit a PR with good formatting and grammar and lots of links to references where relevant. We're especially happy for feedback from tweeps or former tweeps who can tell us where we got it wrong.
One thing that's immediately obvious is that this is not the entire codebase or even a working majority of it. Missing from this codebase are
- many flows that process,enrich, and refine model input data
- YAML configuration metafiles which could tell us quite a bit about how the code actually works. There are only 7 of them, the rest have been redacted.
- Most code related to spinning up the actual infrastructure
- Git commit history that shows us how some of this code has evolved
- A large portion of the trust and safety codebase, which Twitter has noted they've left out for now
An important high-level concept discussed in the Spaces releasing this code was in-network and out-of-network. In-network tweets are those from people you follow, out-of-network is everyone else. A blend of 50%/50% are offered in the daily ~1500 tweets run through rankers.
What was released? The majority of the code and algorithms, but not the data or parameters or configurations or build tools of the Recommneder Systems behind "For You" timeline recommendations. The Candidate Retrieval code was also not released, and neither was the Trust and Safety components, and the Ads components - those remain closed off. No User Data or credentials were inside the repositories and code comments were sanitized (or at least, none were obviously there on first look).
Twitter Algo Repo || Twitter ML Algo Repo || Blog Post
There is a very, very old post from 2013 on High Scalability which gives some context to how these systems were initially constructed.
As context, Twitter initially ran all workloads on-prem but has been moving to Google Cloud.. In 2019, Twitter began by migrating to BigQuery and DataFlow from a data and analytics perspective. Before the move to BigQuery, much of the data was stored in HDFS using Thrift. It currently lives in BigQuery and is processed for many of the pipelines described below using DataFlow, GCP's Spark/Scalding-processing equivaent platform.
The released code comes in a variety of languages. The most common languages used at Twitter are:
- Used in Lucene for search indexing
- Used in the GraphJet library
- Python for the machine learning models in the stack, includes both PyTorch and Tensorflow (legacy) code
- Bazel for building Scala and Java services
- Starlark for bazel configuration
- Thrift, a cross-platform framework for RPC calls originally developed at Facebook(Meta)
- Hadoop - Twitter still runs one of the largest installs of Hadoop out there
- Finagle is a service written in Scala with Java and Scala APIs used to manage RPCs
- Snowflake is a service that generates unique identifiers for each tweet based on timestamp, worker number, and sequence number
The typical recommender system pipeline has four steps: candidate generation, ranking, filtering, and serving. Twitter has many pipelines for performing verious parts of this this across the overall released codebase.
- Candidate generation occurs when you have millions or billions of potential items in your source data based on user-item interactions. This piece usually includes collaborative filtering or neural algorithms to reduce the size of the candiate dataset for downstream tasks.
- These need to then be ranked against each other, filtered against business logic and blended and served to the user in a given surface area, in this case the
For Youfeed in the Twitter timeline.
- The system starts with 500 million tweets posted on a daily basis.
The input data comes from:
- Kafka
- Twitter Eventbus
- GCS
- Vertica
- Manhattan, a real-time multitenant distributed database that was initially developed as a serving layer on top of Hadoop and includes both observability and other metrics.
In migrating to GCP, the current data ingest looks something like this:
- Streaming Dataflow jobs to apply deduping
- Perform real-time aggregation and sink data into BigTable
That data is then made available to the candidate generation phase. There is not much about the actual data, even what a schema might look like, in the repo.
(also called "features" in the chart)
- GraphJet - A realtime Java graph processing library that allows for in-memory processing on a single server and focuses on providing content recommendations. Paper here. Recommendations are provided based on shared interests, correlated activities, and a number of other input signals. GraphJet maintains a realtime bipartite interaction graph that keeps track of user–tweet interactions over the most recent n hours and reads from Kafka. Each individual GraphJet serever can ingest one million graph edges per second and compute 500 recommendations/second.
They describe the reasons specifically for creating an in-memory DB in the GraphJet paper:
The precursor to GraphJet was WTF, Who to Follow, which focused only on recommending users to other users., using Cassovary, an in-memory graph processing engine built specifically for WTF, also built on the JVM.
GraphJet implements two random walk algorithms:
- Circle of Trust (internal to Twitter) and
- SALSA (Stochastic Approach for Link-Structure Analysis).
+A large portion of the traffic to GraphJet comes from clients who request content recommendations for a partic- ular user.
Graphjet includes CLICK, FAVORITE, RETWEET, REPLY, AND TWEET as input node types and keeps track of left (input) and right(output) nodes.
At the end of the candidate generation phase, 1500 Tweets are available for serving to your feed.
- The largest candidate generator is Earlybird, a Lucene based real-time retrieval engine. There is an Earlybird paper.
- Recap The "Heavy Ranker" is a parallel masknet. Majority of the code for this is in the ML repo. The ranker itself is run after the candidate generators.
Outputs are predictions on how user will respond to the tweet:
- probability the user will favorite the Tweet
- probability the user will click into the conversation of this tweet and reply or like a Tweet
- probability the user will click into the conversation of this Tweet and stay there for at least 2 minutes. +probability the user will react negatively (requesting "show less often" on the Tweet or author, block or mute the Tweet author)
- probability the user opens the Tweet author profile and Likes or replies to a Tweet
- probability the user replies to the Tweet
- probability the user replies to the Tweet and this reply is engaged by the Tweet author
- probability the user will click Report Tweet
- probability the user will ReTweet the Tweet
- probability (for a video Tweet) that the user will watch at least half of the video
All of these are combined and weighted into a score. Hyperparameters for the model and weighting are here.
For more details on the model, see the Architecture overview.
- The Light Ranker
-
Coming into this stage from the light ranker, there are other heuristics that are used to filter out more tweets after scoring.
-
Remove out-of-network competitor site URLs from potential offered candidate Tweets
- The timeline mixer has a ratio where verified blue checkmark tweets are offered twice as more if they're out-of-network and four times as more if they're in-network.
These are Twitter specific terms and names that keep coming up across different code bases and blog posts.
- Twepoch - A "magic number"
1288834974657L, which is a timestamp for2010-11-04T01:42:54Zthe date that Twitter introduced the Snowflake ID system, used as Twitter's own "Unix Epoch" - Snowflake - Twitter's system for assigning unique IDs to tweets, users, lists, DMs, media etc.
- WTF - Who to follow
- DDG - Duck Duck Goose, Twitter's A/B Testing Platform.
- Earlybird - Twitter's Lucene based real-time search index. Notes and a blog post here.
- "Unregretted user minutes" - the metric Twitter publicly states is the thing they are optimizing for. It is unknown how exactly they measure this.
Cases of potential bias, manipulation, favouritism, hacks, etc. The focus on this repository is on the concrete, techincal aspects of the code, not speculating on anything twitter may or may not have done. That exercise is left to the reader, however, there are some technical aspects that should still be described about these popular accusations, this is a section for those. Unfortunately, much of the configuration that would contain specific instances of interventions is not in the code.
It was long speculated youtube links get massively deboosted, and Spaces links massively boost Tweets in recommendations. There are no specific references to this in the code. However, there are filters that could be configured for this, referencing OutOfNetworkCompetitorURLFilter for example.
The Elon Musk / Democrat / Republican Code: Now Removed. One of the first widely shared cases, falsely assuming this is something that directly affects Recommendations when it was actually for internal A/B testing, to monitor for effects (DDG is Duck Duck Goose, the A/B Testing Platfom). It was also mentioned in the space and denied there. However, a former twitter employee also offered an alternative explanation (A/B Testing measures behavior, so one way or another Twitter is tuning your TL, indirectly).
There are two mentions related to Ukraine in the Twiter Algo repo. Whereas one of them is a flag for Ukraine-related misinformation used for moderation, or warning labels, there is another safety label for Twitter Spaces called UkraineCrisisTopic. Here are some facts about these labels and their function:
- Each safety label "describes a particular policy violation, and usually leads to reduced visibility of the labeled entity in product surfaces"
SafetyLabelresults in tweet interstitial or notice, are publicly documented here previously and specifically for Armed Conflicts here.- All other Twitter Spaces safety labels are related to misinformation, NSFW, toxic or harmful content, DMCA takedowns, etc.
- 2 hours after it was released, Twitter removed feature flags that specifically higlighted Elon's account
-
@karlhigley's blog and thread of threads are very accessible things about Recommender Systems in practice.
-
A good Recommender Systems entry point is the Google Machine Learning for Recommender Systems course, it also has a good glossary of terms.
-
The biggest academic recsys community is ACM Recsys and state-of-the-art recommender systems research is usually openly available in the proceedings. A lot of the presentations are on youtube.
-
Admittedly out of date, but still useful, is the RecSys Wiki.
-
The latest edition of the Recommender Systems Handbook is also a good book that covers the field well.