Rewriting Twitter for web best practices

TwitterFE screenshot

Last week I decided to rewrite the Twitter.com front-end on Google App Engine to incorporate modern front-end programming best practices, exceptional performance, and establish a solid platform for further development. TwitterFE.com is a fully-functional read-only clone of Twitter.com designed to make your web browser sing. I created the site as an example of web development best practices anyone can integrate into their web presence.

The new web front-end on TwitterFE.com features localized templates, expressive markup, distinct URL structures, integrated site search, geo-distributed dynamic and static servers, and more available features than Twitter.com. In this post I will outline some of the changes I’ve applied to the Twitter front-end reproduction as they apply to general front-end web development.

  1. Global audience
  2. Unique usage models
  3. Consolidate URLs
  4. Expressive markup
  5. Split the page load
  6. Know your cache settings
  7. Review access controls
  8. Expose site search
  9. Summary

Global audience

Twitter most popular countries

I added a localization framework to the Twitter front-end to enable site content delivered in multiple languages. According to Google Trends Twitter’s top regional languages are English, Portuguese, Japanese, Chinese, German, and Spanish speaking regions. I isolated the site’s template strings and translated all key phrases into Spanish. Visitors with an Accept-Language header of es will receive template strings in Spanish.

TwitterFE Spanish language example

The most difficult part of localization is isolating your template strings and choosing common wording across the site. Twitter uses the terms “person,” “user,” “account,” and more to reference a profile owner. Websites need to pick common concepts to explain their site interactions before requesting translations.

Modern websites rely on crowd-sourcing to translate a site into new languages. Porting a web application to your native language is a point of pride for many communities. Something as simple as “favourites” instead of “favorites” for the Brits could help create identity around a product in other countries. Facebook Translations and Google in Your Language are just two examples of large localization efforts led by an engaged community.

Unique usage models

Twitter.com has three main visitor interactions: anonymous visitor, annotator, and author. The current Twitter website loads resources for all possible interaction types, weighing down the page and interfering with the intended experience. I tore the site down and started from scratch, building up each interaction model starting with the anonymous visitor.

Anonymous visitor

Twitter logged-out view

The anonymous, public visitor is anyone browsing Twitter content in a non signed-in state. This audience typically makes up the majority of site traffic and includes both humans and search engines. Websites need to clearly and concisely communicate content to this new audience who is likely inexperienced with your site while quickly while smartly driving business objectives such as member signups or advertising.

Annotator

Twitter Al Gore example

The annotator is a logged-in member browsing site content with an opportunity for annotation. They may want to discover new social network friends, mark content as a favorite, or otherwise engage with an existing content. Annotations are typically short and asynchronous, posting new associations between an account holder and a unique content identifier. The majority of pages presented to logged-in users on Twitter.com follow this annotation interaction model.

Author

Twitter status update field

Twitter is a message authoring platform. Logged-in users may publish new text updates from their homepage while browsing other subscribed content. Authors type into a text area and receive real-time feedback on authoring limitations while they type. The author commits new content to Twitter’s servers after hitting a submit button, and the service responds with a confidence indicator for accepted updates.

Consolidate URLs

How many possible URLs represent the same content on your website? Websites should avoid duplicate content spread across multiple paths, subdomains, and protocols. There should be one strong public-facing match for your distinct content.

Which one of the following URLs represents the profile page of Twitter CEO Evan Williams (username ev).

  • http://twitter.com/ev
  • http://twitter.com/ev/
  • https://twitter.com/ev
  • http://explore.twitter.com/ev
  • http://m.twitter.com/ev
  • and more…

Websites need to pick a winner and funnel visits into that best representation of content. Search engines are crawling multiple versions of Twitter right now and splitting authority between many different options.

Be aware of URL propagation when you introduce new subdomains and schemes with relative URLs inherited from a common template. You might be buying new servers to keep up with a crawl load across your millions of pages as a result.

Expressive markup

TwitterFE uses an xHTML vocabulary to express content, CSS for positioning and styling, and JavaScript for progressive enhancement and interactions. Gone are the table-based layouts of Twitter.com and its heavy DOM footprint. Resources are split into dynamic and static content and served from geographically-distributed datacenters for optimal performance. Twitter currently stores static assets such as profile pictures on Amazon S3 and does not use a distributed CDN to address speed of light issues.

Comparing a user such as Al Gore on TwitterFE vs. Al Gore on Twitter.com shows a 41% difference in required resources sent over the wire (89 vs. 152 KB). The new site also reduces the total DOM footprint for faster parsing, layout, rendering, and addressing.

Microformats expose unique structured objects within each page such as people, relationships, and feed mappings. Search engines such as Yahoo! tap into microformat content to expose deeper information about a page. I cleaned up microformat support on Twitter pages and added support for Internet Explorer 8 Web Slices.

Expressive markup helps web browsers and search engines better understand the content within your pages. Sites can fully utilize xHTML vocabulary sets independent of default styling to best define the content and the rendered display of each page.

Split the page load

Web pages should respond quickly with progressive enhancement added after the main page content renders on the page. We might, for example, load a page and then apply search field listeners, autocomplete, or menu expansions as a second wave. Splitting our pages into “must have” and “nice to have” segments helps us deliver core content quickly while still providing the on-page interactions and magic sprinkles that thrill our visitors.

Twitter’s profile pages load 36 of the profile owner’s following list onto each page. That’s 36 tiny little 700 byte profile images all waiting in line for a remote connection and display on page. I tripled the total number of displayed member pics but loaded the list asynchronously after the rest of the page finished loading. I can pre-fetch these components into cache on the original profile request and respond very quickly to the async request after page load.

Know your cache settings

How long should browsers and other requesting agents hold on to a piece of content before requesting a fresh copy? A frequently changing profile page might expire its HTML every 5 minutes or so while static assets such as a site logo or icon should be kept in browser cache for a long period of time instead of requested with each page. In some cases Twitter sets image Expires headers 5 minutes into the future, slowing down pages and increasing bandwidth costs for the company and its visitors.

Review access controls

Some sites split pages into public-facing and login-required access models. Twitter places pages such as a following list or a full-sized profile picture behind a login screen while exposing the same data over their APIs without such restrictions. Twitter.com is losing search engine exposure and logged-out user browsing capabilities due to these inconsistencies in implementation, not policy.

Expose site search

Twitter OpenSearch example

The OpenSearch format exposes site search options to web browsers and search engines alike. If your website offers site search you should be lighting up the browser chrome with new search options for the given page. Twitter acquired a search company in July but has not exposed available search hooks in their main website’s front-end. Think about how you might want to scope a search to the currently viewed user account as well as an expanded site-wide option.

Summary

TwitterFE is a read-only clone of Twitter’s front-end that fixes many of my frustrations with the site’s front-end engineering and creates a new platform for future third-party development. Any site could roll these types of improvements back into their core services. Twitter APIs are full-featured enough I can clone the Twitter front-end without creating yet another stand-alone Twitter-like site.

There is a difference between a website or widget rendering in a browser and having the same site perform exceptionally well. Established web teams should revisit their web content to optimize experiences.

TwitterFE.com is the result of one person working part-time for a week to re-write the front end of a website serving millions of monthly visitors. Similar lessons apply throughout the Web world.

I now have a new platform to develop features beyond what’s currently offered on Twitter.com. If you’re an iPhone developer in need of a headless Twitter API proxy for push updates let me know.

What other front-end features do you wish established websites would invest time and effort to improve?

  • Posted
  • Updated at
  • Comments [9]

9 comments

Commentary on "Rewriting Twitter for web best practices":

  1. Terry Jones on wrote:

    Very cool. Love the ads :-)

    • Niall Kennedy on wrote:

      Thanks Terry, The ad is currently not dynamic. Just a 300×250 (medium rectangle) placeholder to float the idea of additional widget or ad content.

  2. Stephen Pierzchala on wrote:

    Sometimes the best way to re-design a site is to walk away from what you are doing and start again. Too much effort is spent making sites try and do what they have always done faster.I hope that the Twitter team takes some of your ideas to heart. Great work.smp

  3. Mack D. Male on wrote:

    This is tremendously cool, well done!

  4. Jillian Sands on wrote:

    As usual, your post was well thought out and very thorough.One question – you mentioned the ranking of the global audience and I was surprised at the dominance of Brazil. Any idea why Brazil is such a heavy user over the other countries?

  5. David Bisset on wrote:

    Nice work. Although my profile page looks a little unfinished in Firefox this is all still very awesome.I like the ad on the side – I don’t think that is a big deal since I use twitter mostly from my iPhone and would never see ads on twitter.com.

  6. Evan Prodromou on wrote:

    Hey, Niall! Great work on this! We’re looking over your changes on Identi.ca to see what we can to to improve our user experience.We’d love your participation in that process. Identi.ca is Open Source software, so you can work on the real code instead of a read-only copy!

  7. HugePierre on wrote:

    Your app seems extremely interesting. Only problems for the newbie (and the dummy too) like me is that I donĀ“t see how to implement it correctly. Maybe a step-by-step explanation for dummies could make some good. In any case thank you for providing your science to the community

  8. Gustavo Franco on wrote:

    Are you going to publish the full or part of the source code? It would be very good for App Engine developers.