Yahoo! Pipes remixes the syndicated web

Yahoo! released Yahoo! Pipes tonight, a visual editing interface for web feed manipulation and reconstruction. The 5-person Pipes team, part of the Yahoo! TechDev incubation group, spent about 5 months developing the product to help people better remix the syndicated content they find online.

Yahoo! Pipes lets any Yahoo! registered user enter a set of data inputs and filter their results. You might splice a feed of your latest bookmarks on with the latest posts from your blog and your latest photographs posted to Flickr. You might automatically translate your favorite news sources to your native language, or only receive the 1 out of 20 news stories from your local paper that reference your town or local schools. A traditional web feed lets you select your news from a set menu, while tools like Yahoo! Pipes let you build your own dish with only the ingredients you care about.

Yahoo Pipes sample edit interface

The editing interface connects pre-configured modules and their option, creating a new feed accessible as RSS, Atom, or JSON. Anyone can share their modules, or clone the work of others to tinker a few things and enable their own customizations.

Yahoo! Pipes opens up some interesting possibility for feed aggregators, letting users filter out unwanted content affecting their experience. Pipes opens up a few feeds that were not practical for a human to read in the past, either due to a high volume or possibly a foreign language. My favorite operator is the location extractor which analyzes an item’s text attempting to identify addresses, locations, or the URLs of popular mapping services.

Publisher Concerns

Yahoo! Pipes has implications for web publishers, changing the reliability of delivered content, the relationship with the end user, and the polling frequency of a mashup that may or may not be actively utilized.

Yahoo! Pipes makes it easy to remove advertising from feeds or otherwise reformat your content. I already know a few publishers who hold back the publishing the full content of their posts for fear of easy resyndication and brand dilution, and if Pipes becomes popular publishers might hold back a bit further or ban Yahoo! Pipes outright. A Yahoo! Mail user searching for a new feed subscription will likely choose an identical feed labeled “No Ads!!!” associated with their favorite brands.

One of Yahoo!’s sample pipes, Aggregated News Alerts, uses the Technorati search API and republishes a key issued to an individual user. A site such as Technorati can increase that user’s allowed queries per day, but they lose control over the issued unique key and its use.

The Pipes troubleshooting section lists three ways of blocking the tool from using your feeds: modify your Apache settings to block User-Agent “Yahoo Pipes”, add a new element to your feed, or send Yahoo! an e-mail asking them to manually add your URLs to a blocked list and verify your authority to make such a request. The suggested meta element added to your XML creates invalid feed markup and might cause your feed to stop appearing in some strict renderers.


Overall I really like Yahoo! Pipes, it’s intuitive interface, and its “View Source” approach to building your own web services. I think a lot of people will build interesting new things using the service, and it ties in nicely with services such as Yahoo! Alerts. It’s a pretty solid product from the Advanced Products group, leveraging web feeds as a simple web service.

  • Posted
  • Updated at
  • Comments [6]


Commentary on "Yahoo! Pipes remixes the syndicated web":

  1. Pasha Sadri on wrote:

    Hi Niall,

    This is Pasha, the creator of Pipes. Thanks for covering pipes. I wanted to comment on your concerns about feed publishers (This is my take, and not an official yahoo policy on the matter).

    I believe that feeds are a mechanism for content/information distribution. Pipes (or similar services) allow consumers of feeds to get more refined (ie: high quality) information out of them. This means that the publishers in return will get more qualified/interested readers of their content. So, I believe that publishers in general will benefit from having another mode of distribution for their content.

  2. Markus on wrote:

    Really interesting service. Could give us more info about the technology behind the scenes? Java, Python, Ruby? Web frameworks and other tools?

    • Niall Kennedy on wrote:


      The site uses Perl and the Yahoo! UI JavaScript library.

  3. Prakash on wrote:

    I actually don’t agree with Pasha “this means that the publishers in return will get more qualified/interested readers of their content. ” For a publisher (online magazine) a reader is only considered a qualified lead or reader when he/she registers with his site. I think it’s best tool for consumers.

  4. Jonathan Trevor on wrote:

    Small correction: Pipes was developed in Bradley Horowitz’s Advanced Development Division, which does indeed contain an Advanced Products Group, but we (the pipes developers) are in a different group (TechDev).

    Pipes will be back soon…

  5. Lance Robinson on wrote:

    This past September you blogged about wanting feeds for data. IMO, pipes is a big awareness step in this direction. Back then I commented about the fact that RSSBus was piping data and feeds – this works really well with Pipes. I’m interested to see how quickly more “rich” feeds become commonplace.