Building the Retweet Button

I joined the Twitter engineering team in March 2009 and spent 5 years at the company, as it grew from ~30 to ~3000 people. It was a seminal experience in my career and the first few years generated some “interesting” stories about engineering work.

Over the past decade-plus I have found myself telling these stories to coworkers so many times that I figured I should write them down. I want to write them especially for folks who have entered the industry more recently, as it can sometimes be hard to comprehend how things were in the “old days”.

Before I go further, I want to provide a disclaimer: these are my accounts of what happened 10+ years ago. Where possible I have cross-referenced with available sources and people, but there are probably some details missing or wrong. Feel free to email me if you find such an error.

Chapter 1: RT

I joined Twitter engineering in March of 2009. The company had, I think, 33 people, about 20 in engineering. The engineering organization was mostly flat, though was informally divided into four clusters: those working on product, operations, infrastructure+storage+performance (“Infrastormance” as it was later called) and search+anti-spam+related.

My first project when I joined was the retweet button. For the youngsters out there, in 2009 there was no native retweet functionality in the product. The idea had been invented by users who would take a tweet like “eating a sandwich” and literally copy/paste into a tweet of their own, saying “RT @rk eating a sandwich”. Third-party clients built it in, and then Twitter decided to incorporate it.

The first thing to say was that putting the retweet functionality into Twitter itself, rather than leaving the core platform “dumb” and letting clients do stuff on their own was not an obvious choice internally. The team was conflicted on it. When I joined there had been a 1-man team, Chris Wetherell working on it for 9 months already (as a contractor).

I was tasked with taking the UI Chris had built, building a backend,  generally making it work and launching it. Easy enough. As I started working on it I immediately had product questions, which quickly led into the morass of “what are we doing anyway?” In addition to ambivalence about whether it should be in the core product, there were a lot of product questions - “should it be called retweet or share”? “who should see them?”

The biggest concern was referred to as “strangers in the house” which was a short-hand for the concept that up until this point, Twitter timelines had been 100% curated by the end user. You would absolutely never see a tweet from someone you had not followed. Except, of course for the manual / third-party retweets, but that was different.

One of the interesting parts about those tweets is that the user photo was of the retweeter, not the original tweet. This led to debates about how the native RT product should treat user profiles, specifically which user’s photo should be shown. I think we might have even built a version which had both of them overlaid with each other.

It’s also kinda crazy to remember, but the company at the time, was ambivalent about calling anything “tweets”. In the code they were called statuses and the old-guard was somewhat resistant to the cutesy name ’tweet'.

I did not see the RT project through to the end. After a few months, Matt Knox joined the team and I handed the implementation work off to him (in favor of me working on the project described in the next chapter) but still collaborated with him on some of the technical design questions. 

In summary (not going to go too deep here), to launch the native RT functionality, we had to tackle a handful of backend projects

First, we had to alter the way tweets were stored to allow some to be retweets, adding a field called source_status_id, which used to hold reference to the original tweet.

Small digression here. I really wish I could confirm the above with Mike Limon, who was our one-man DBA team at the time, but he sadly passed away a few years ago. Fuck COVID.

After changing how we stored tweets to keep references to the original, we had to change the way tweet timelines were stored in memcache. Discussion of how that worked deserves its own blog post.

After getting the storage and timelines part sorted and updating the UI and APIs to render, Matt and I had to figure out a rollout process for the native-RT functionality. 2009 was before the idea of feature flags was widely known. The eng team at flickr blogged about it later that year, which created a lot more awareness.

However, twitter was ahead of the curve enough that we had not one, but two implementations. One called “Darkmode”, which was for turning off features on the site, mostly to shed load and/or because they were broken. And another call Releaser.

At the time, Darkmode was binary and Releaser could work from static lists (show this feature to all these users) or by percentages. Before shipping the RT button, Matt unified these two implementations into one called Decider¹, making a few improvements as well. Releaser only used 2 decimal digits of precision (1/100). That was extended to 4 (1/10,000).

The first launch challenge we encountered was that the RT functionality had two sides- posting and reading and we wanted to test both. 

We wanted to first expose it to ourselves, but this posed a problem. What should these retweets look like to those for whom the feature was off? One of us thought of the clever idea to render them as if they were manual RTs. So if I had the feature I could post a native-RT. I you didn’t have it, you would see what looked like a non-native-RT.

Getting that in place, we rolled it out to just those of us working on the feature. Then the eng team, then all twitter employees.

The next launch challenge was this– if we turned it on for 1% of the user-base, the chances of overlap, where both retweeter and reader had the feature, could be small. So what Matt did was, starting with Twitter employees, do a graph walk, finding folks most closely connected in the following graph. Then rather than turn on by percentage, we used that list for the first several thousand users to grant access.

There were also two different Decider flags - one for seeing native Retweets and one for creating them. Extra care was taken to be sure that folks could see them before they could create them.

After rolling it out to the first several thousand users via a graph walk, we went back to the standard percentage-based rollout. Somewhere in here we added a bunch of API partners so they could test their clients.

As was typical at the time - some users were annoyed. And they tweeted about being annoyed. It seemed bad at the time, but in retrospect, what is Twitter if not the perfect tool for 1) talking about twitter,  2) complaining about trivial things and 3) soon forgetting that anything changed at all.