Answering Scott Gatz

Scott Gatz of Yahoo! poses a few questions on his blog that came up during a private dinner Monday night. I think some of the questions already have answers so I’ll post them here. It’s a bit feed geeky and may have an intended audience of about 20 people.

No index flag. We need a way to mark an RSS feed as “OK to aggregate, but don’t show in search results.” For publishers who output a different feed per user, you don’t want to see 100 different feeds that are basically the same thing.

I think the robots meta tag “nofollow” directive already communicates this preference for link alternates listed in the page head and a rel value of “nofollow” communicates the preference at the link level. Search engines should pay attention to these user preferences when discovering new feeds.

The easiest path to personalized feeds is to ask for a username and password over HTTP Authentication. You would also have exact tracking methods for that user since you are basing unique readers on a username instead of something like a unique IP address and User-Agent combination.

Tracking statistics at the item level remains elusive since most feed aggregators load every feed item at once. One possible solution is to track “focus events” such as a mouse pointer hovered over the post area or an item marked as unread. Aggregators would have to gather some of this data and send it to a central location for marketers and feed publishers but there are currently no incentive models to drive such a flow of information.

Update: Greg Reinacker of NewsGator added his own thoughts and prefers a namespaced additional element in the feed.

  • Posted
  • Updated at
  • Comments [3]


Commentary on "Answering Scott Gatz":

  1. Greg Reinacker on wrote:

    Agree that nofollow may work for the HTML page itself…but in the case where we’re looking at a feed, we need a way to know that the feed and its contents shouldn’t be indexed.

    On the authentication – I agree with you completely, and I’ve written about this several times. However, despite that, people are still going to build personalized feeds using unique URL’s, and we need to have a way to deal with them in a way that doesn’t “gunk up the system”. :-)

  2. /pd on wrote:

    Hey not sure, but is this the same qustion that i raised earlier kinda thing or not !!
    The Question ; and some thoughts here with flags
    ALLFEED = OK To Aggreate(TRUE)

    What I am trying to do is understand methods that sustain unique feeds only to a authorized party or community.

    Think of a finanical transaction hitting your bank account and that information is feed via rss to Myself and other account holders.. or in another scceanrio of an imports proces, where each hand off /trade off point is piped via rss…but only to certain users…

    Any thoughts ??

  3. Matt Terenzio on wrote:

    Although SSE has syncing in mind, I think it has the the properties needed to “version” out an unlimited number of personalized or trackable items but also clueing in aggregators and search engines that the item is indeed the same content.

    I have a little project at that uses SSE to create threaded discussions.

    The system does not do the optional conflict resolution but maintains. . . well a thread.

    Couple that with OPML sync and you could have distributed systems contributing in a fluid manner.

    Much better than having to tag an item uniquely for Technorati to index. (No offense. I mean any central index.)