Personal tag cloud

Technorati now displays a personal tag cloud for each member profile! You can now glance at a blogger's profile and get a pretty good idea about his or her most blogged about topics. Tim Appnel's Tags.App plugin for Movable Type displays some similar tag visualizations but now anyone on any blog platform can visualize their topical focus.

Yes, I want this component to be one of the options bloggers can select as part of their Technorati JavaScript embed but that takes just a little more time and it's best to get the goodies out the door and iterate. The top tags per blog has been available via the Technorati API for months but this is the first time it's been exposed on the main site.

New search results format

Search result excerpts are now a bit longer and meant to welcome the newbie by hiding away some of the more advanced options. Above I hovered over the magnifying glass to reveal a scoped search option for the word "soccer" from a mommy blogger. The talk bubble icon displays the number of inbound links to that blog.

I miss not being able to glance at a person's link count on the search results page but I think there may be better ways to represent the same data to quickly communicate worthwhile information. I like the eBay stars program used to communicate a feedback rating and perhaps some similar tiny graphics could be applied to Technorati link counts but with more meaningful colors.

Keyword search charts

It's now easy to visualize keyword use trends over the past 30 days with keyword search charts. The chart above shows the use of the word "Santa" in blog posts over the last 30 days. I've been using these charts to follow spikes around product announcements around the industry to see if the excitement and interest in a product remains or quickly fades. Right now I am watching the search result for "Ronaldinho," a Brazilian who was just named best soccer player in the world for the second year in a row.

Advanced charts

You can easily construct your own charts in various sizes using a standard URL structure:

http://charts.technorati.com/chart/ + [keyword] + ?size= + [size]

Charts are outputted in PNG format. The size of the chart varies from a default of s (small, or 180x150 pixels) to xl (extra large, or 700x500 pixels). Publishing tools could easily add a Technorati chart to graph trends over time.

Enjoy! We've promised the Technorati ops team not to release any new features until after Christmas to keep our servers running a bit more predictably than normal.

I almost never use the keywords field, yet it is already built-in as a storage option for every post. Perfect! No plugins necessary. Here's how you do it.

1. Add your podcast's duration ( mm : ss ) as your entry keywords.
2. Add a line to your RSS 2.0 feed:
   <itunes:duration><$MTEntryKeywords$></itunes:duration>

The time column will now be populated in iTunes for all of your podcasts.

Tags: itunes, movabletype

This week's session is 26 minutes and 28 seconds in length and a 12.2 MB download.

Topics

Amazon introduced its Alexa web search platform on Monday. The new services give developers the ability to use content from Alexa's crawler data and other web services in a hosted environment. Amazon is letting developers upload their own data, process the data on their machines, host the results, and even create new web services based on the final results. Amazon's move caused a reexamination of developer programs in Silicon Valley as Amazon is charging for premium features not currently offered by eBay or Yahoo!.

On Tuesday Google introduced the Google homepage API , opening up its personalized homepage to outside content. The new modules are similar to desktop widgets such as Konfabulator but exist within the browser page and are intended to be the first thing a user sees when he or she launches a browser window. I wrote my own Google homepage module earlier this week.

FeedBurner launched FeedFlare on Tuesday, allowing feed publishers to easily add content to their posts from services such as del.icio.us or Technorati. I think the new service changes how publishers view their feed as part of a larger ecosystem. I wrote about FeedFlare earlier this week.

Yahoo! released a new output format, JavaScript Object Notation ( JSON ), for its search and maps API s on Thursday. This new format helps developers potentially skip a step when interacting with Yahoo! data, making dynamic applications respond faster and easier to develop. JSON also helps Yahoo! get around the JavaScript XMLHttpRequest function's cross domain security policy

Google seeks to extend its reach into the VoIP and video IM space with its introduction of jingle, a new API for Google's GTalk client. Google proposed two new extensions to the XMPP standard used in its clients and many others throughout the world. The new libraries and API s should extend the reach of Google's instant communication services throughout the Web.

Tags: alexa, amazon, feedburner, google, jingle, json, yahoo

Popular blog hosting provider TypePad.com was unavailable to its members for 18 hours today as a result of a failed storage upgrade during scheduled maintenance late last night. I visited Six Apart's headquarters in San Francisco earlier this evening to talk with Six Apart employees about the events that transpired and the current state of TypePad's blogging platform.

I sat down for an interview with Anil Dash, Six Apart's Vice President of Professional Products, about an hour after TypePad's servers came back online. We discussed TypePad's reliability issues over the past two months, what went wrong today, and how the company plans to prevent future outages and create a reliable service.

Anil and I had a candid conversation about the difficulties of building a large-scale hosted service, the uniqueness of TypePad's members, future plans for the TypePad product including Project Comet and a business-level offering, and how the company approaches scalability and building dependable applications that people love to rely on as a utility.

The entire 23-minute interview with Anil Dash on TypePad service issues is available as a 10.4 MB MP3 audio download. Any rumblings you may hear is the sound of TypePad engineers running back and forth nearby and literally shaking the entire room.

Each paragraph in the interview transcript below is uniquely identified in the page markup. To link directly to a paragraph just use add "#" and the question and paragraph id found in the source.

Interview Transcript

Niall:

Hi I'm Niall Kennedy and I'm here with Anil Dash of Six Apart. Anil is VP of Professional Products and we're going to talk about TypePad and some of the issues that happened today, Friday, and last night. Hi Anil.

Anil:

Hello.

Niall:

Can you tell us a bit about what happened last night to cause a loss of service to TypePad?

Anil:

Yeah. I think the first thing I would start with is no data was lost and the service is back up and running now. A couple weeks ago, as you well know, we had some problems with capacity and scalability as TypePad has been getting more popular. We made a bunch of commitments in terms of improving bandwidth, improving power, and things like that. Among those changes was improving our storage infrastructure and one part of that was increasing redundancy. We actually have accomplished most of the things we promised we would do on that list and one of the last things to do was to improve redundancy on storage.

We were bringing online another storage system and it's one of those things where you have a moment of vulnerability while you're improving the system and we had a failure during that point. So we had what we thought would be a fairly routine scheduled maintenance last night and we had posted an alert about that so people knew ahead of time "oh, OK, this is why this blip is happening" and right at the end of that window of maintenance we had a failure of the new storage system.

The process at that point was understanding what had happened and whether there were any issues of data loss, which fortunately there were not. The first step that was decided -- and this is something I am sure will be avidly debated -- is that we would go to the last full snapshot. We have incremental backups and we have full snapshots and most people I think that are listening to this probably know the difference. The full snapshot was anywhere from a few hours to a few days old, depending on people's blogs. We then did a verify on the data that we were restoring.

As we brought the disks back up -- the process is like the olden days of Windows 98 when you would get that reboot into the blue screen saying "I am running checkdisk and please wait forever while I check your hard drive." It was doing that process and this is what we started doing last night and all through the night and in the interim we had those old snapshots of pages up. I think a lot of the people who were looking at their blogs today or posting had that alarm of "oh my gosh, old data" and "does this mean old posts are safe and my new posts are not?" It was making sure that was not the case that took a lot of time today. And then, of course, eliminating single points of failure, identifying all the other problems. Between all those steps together it took until this afternoon Pacific Time before we were able to switch the app back on.

Niall:

You guys run a variety of sites from TypePad.com, blogs.com, Major League Baseball blogs. What sites and partners were affected by the outage?

Anil:

All the TypePad-powered sites in the U.S.: TypePad.com, blogs.com, Friendster's blog service, as well as Major League Baseball's service to various degrees. I don't actually know offhand the details about which blogs were affected on those services or what percentage it is. That's something I know our operational team definitely knows but my job today interfere with them. My estimation is the majority of TypePad blogs were affected.

Niall:

How did you notify TypePad members of the outage and the unavailability of their recent posts?

Anil:

We had a couple of different steps. The first is our status blog which is our first line of defense and the first place you should check if you ever have a question as a TypePad user. It honestly is probably not nearly prominent enough to find and that's something we are going to be remedying. It's one of those things you find out in these situations is that people don't know that channel exists. And then we'll be blogging about it. I think one of the things we've shown a pretty good track record of is having a conversation about these things and really engaging in a dialogue with our members and I think you'll see our usual breadth of posts which is here's a technical explanation for people who care about that, here's what we're going to do to make it right for people -- I think for most people that's their priority -- and then a little about what we've learned from the past is something we will go into. You work at Technorati. I think the folks at Technorati have done a really good job about being open in communication on that. I think we've got a pretty good reputation there and I'd like to be sure we stay in touch with everyone.

Niall:

Did you consider sending out e-mails to every TypePad member to let them know that they wouldn't be able to post and their latest posts might not be available?

Anil:

Yeah, I think we talked about a variety of different channels and the thing you want to do is make sure you have all the right information getting to the right people at the right time. Part of it is until you know that everything is up and running and all the data is OK you don't want to send a "we don't know" e-mail because it tends to make things worse before it makes things better. That's something we've gotten by talking to our professional users and people who are really invested in their blog. They've said "just let me know you are aware there is a problem and then let me know when you've fixed it and in between if you're going to tell me about the fsck you're running on your hard drives I don't care." I think that's probably true for most TypePad users. "Just let me know when it works." That's I think what we're really going to focus on is that blogging experience.

There are many different channels. I spent all day commenting on blogs today. Other folks here at the company have. Jay Allen I think a lot of people will see out there. A lot of people have been on IM and on Skype and on phone calls all day too. We talked about if we should do a big Skype call with a bunch of people but finding something that we can setup in a short amount of time that scales to millions of users or millions of listeners is pretty hard. I think in the future we'll see what we can do about that too.

Niall:

How are today's problems related to the problems you had at the end of October? Are these similar issues? Are they going to be recurring?

Anil:

No, it is actually not about capacity at all. I saw somebody describe it as "a perfect storm" today or it was like a "lightning strike." I felt a little bit like Charlie Brown when Lucy pulls the football out. It was absolutely one of those situations where we had tackled the problems that caused application performance issues in the past and we wanted to make sure we were bulletproof. At the point where we are kind of putting up another line of defense we kind of hit ourselves on the head and got knocked out. The timing was terrible and it just one of those things where Murphy's Law will just always in.

Niall:

Are LiveJournal users affected at all or are they on separate hardware?

Anil:

Not at all. They are on separate hardware. The best thing about having LiveJournal in-house is they have really taught us, and the whole web in general really, about scaling and reliability in large-scale operations. They've got 9 million journals and they recently did a datacenter move to the same datacenter we've got TypePad in. They have hundreds of servers, millions of users, millions and millions of posts and they did it with some glitches around e-mail and things like that but no downtime. The open-source technologies they have created like MogileFS, a caching file system and Perbal, a load balancing system, run sites like Wikipedia and Slashdot and Friendster: these gigantic and dynamic sites. They've shown that the technology exists. People on our team build these services capable of making large services reliable. I think you are going to see a lot of knowledge transfer and best practices about how to make this stuff bulletproof and reliable.

One of the examples I think Steve Rubel talked about on his blog is eBay back in 1999 when people were just starting to have their livelihood depend on the auctions closing on time and the service would go down. People solved these things then and I think we are going to see the same thing for us.

The thing that gives me a lot of faith is we have these TypePad users that are rooting for us to do it even though we have absolutely not met their expectations, especially today. We had somebody send us pizzas for the whole TypePad team for lunch today saying "I love TypePad, I know you guys are going to do right by us, I trust you to get it right, and we are thinking of you." That's an astounding thing for a community to do. That's really special, that's why you do it. That's why you keep going we dropped the ball today.

Niall:

Can you talk about the backup solutions that are in place to make sure people don't lose data on TypePad?

Anil:

I'll speak as much as I know. I don't want to overstep what I am knowledgeable about because I am not on the operations or infrastructure teams that we have. There are, as I understand it, the now redundant storage system. One of the interesting things about our architecture is we have a database that stores all your posts but it generates and a machine that generates the pages that your posts appear on. They are separate. The storage failure today was on the system that stores your web pages that you can view and that's why we saw an older cached copy when they went back live. The entries in the database are on a separate database server and that is already redundant and somewhere we knew we didn't lose any data so we felt a lot better about that. We were able to verify that pretty quickly, it was just the disks that took some time to come back online. In terms of architecture I don't know exactly how the different tiers split up but the database is separate from the web server pages and now we have redundancy on both.

Niall:

Last February in an interview Ben and Mena mentioned that TypePad was originally intended for 3000 users and then it was going to switch over to an invite-only system. Is this a scalability issue and is TypePad up to the challenge of hosting hundreds of thousands of blogs?

Anil:

I think what they were talking about there was not so much the architecture of TypePad but the concept of how many people would want to be blogging, how many people would want to be on a hosted service, and how many people would pay for a hosted service. It really spoke more to whether, I think this is something Om Malik was talking about with Web 2.0 scalability in general, are you designing for a community of people who you are personal friends with and can talk to one-on-one or are you trying to bring something to the masses? I think that's the biggest distinction that Ben and Mena were talking about the chang

Today's outage, coupled with the performance issues of the past month, has shaken the confidence of even the most dedicated users, and many members are seriously considering taking their business elsewhere. I spoke with one TypePad member who has used the service since it was first introduced in beta form back in July 2003. He told me "this sort of outage leaves me wondering if my blog is really okay in their hands, or if I should take it back into my own." An advertisement for the term "TypePad" is currently 78 cents on Google and I expect the price will rise this weekend as web hosts capitalize on the opportunity.

Six Apart agreed to do a podcast with me tonight when they find some time. I cancelled my plans for the night and like any good disaster reporter I am going to plop myself down across the street from the Six Apart offices to help get the full story out to the world.

Please leave comments and questions below that you would like me to address with the staff at Six Apart tonight once I am able to get some face time. You may also contact me privately with questions and concerns.

Update 7 p.m. : Comments are closed. I recorded a podcast and I am currently editing audio.

Tags: TypePad

I've known Scott for a few years. Scott has split his time over the past year between Feedster's headquarters in San Francisco and his wife, son, and home in Indiana. I think once some dagger wounds heal Scott will be much happier on his next project.

What's next? Scott said if he'd love to start a company to sell pre-configured MythTV boxes but does not want to put his family through the legal troubles involved with shaking up a large industry. He's very interested in making complex technical problems easy for end users and I am sure his next project will involve simplifying technology in some form.

Good luck Scott!

I may have a Technorati version of the Blogger Web Comments Firefox extension running on my machine right now. Google directs users through a terms of service for the extension that asks users to agree to not modify the code, so I can can't confirm my hacking...

You can download the Blogger Web Comments Firefox extension directly without passing through the terms of service if you would like.

Under part 4 of Google's Firefox tools terms of service, proprietary rights, they claim full ownership over all released plugins and prohibit all modifications and derivative works.

Except as expressly authorized by Google, you agree not to modify, rent, lease, loan, sell, distribute or create derivative works based on the Software, in whole or in part.

That sucks. When I use an open-source product I expect to be able to tweak, modify, hack, and break a few things. Slapping protections and restrictions on an extension of a Mozilla Public Licensed product seems a bit evil to me.

If anyone would like a copy of the plugin with green or pink styling, or with results from Technorati or another service, or with full support for the MetaWeblog API , I can tell you that definitely possible but Google's lawyers might come after me if I admit to modifying the extension.

Tags: firefox, mozilla, opensource

Google just introduced an extension for Firefox that allows you to see the latest blog post linking to any page as it's viewed. The extension brings up a small window in the bottom right-hand corner of the browser window with the most recent reference to the page. You can click on the box to extend it to the last 6 posts or view the entire result set on Google Blog Search.

Users can also add their own comment about the page by clicking a special "Add Comment" button linked to their Blogger account. The Google FAQ mentions they hope to add support for blogging services other than Blogger in the future.

The plugin only works with Firefox 1.5 or above. It's a very cool feature and the smooth execution does not surprise me as Google employs many Firefox core developers. Nice work Glen! The plugin requests an RSS file from Google blog search for the URL and displays the result inside a special chrome container.

Tags: blogsearch, firefox

Profiting from consumer generated media is not a new thing. Shopping sites such as Amazon.com or PriceGrabber have been doing it for years, asking an author to turn over the rights to their content in exchange for the author's work being featured alongside a product. At PriceGrabber we paid anywhere from $5 to $25 for many of these reviews realizing that more information about a product helped drive clicks and completed the buying decision. I think bloggers can be similarly paid for syndicating their content to larger sites to assist in the buying process as one small example.

Blogger merchant relationship

Most people are familiar with the Amazon Associates program where registered affiliates receive a percentage of the final purchase basket for each sale initiated from another site. Review a DVD, provide a special link, and receive a few dollars for your lead generation. Yahoo! plans to pay individuals for generating successful leads through its Shoposphere Pick Lists. In existing cases such as Amazon or Yahoo! if you feature some content on your blog you receive a small percentage of the purchase and both sides are happy.

Blogger toolmaker relationship

What if there were tools available that made it a lot easier to post reviews to your site in a well-styled format with extra features such as an updated price or how other people have bought the same product after reading about it on your blog. Should the toolmaker receive a small percentage of that revenue?

Browsing tools such as Mozilla Firefox or Opera currently receive tens of millions of dollars a year from web searches generated using their tools. The tools are offered for free and people enjoy the experience.

The same partner agreements should exist for specialized blogging tools that help generate a sale or a profitable benefit for another company. If Byrne Reese makes it really easy to blog product reviews with Media Manager shouldn't he be rewarded a few cents for every sale? I think so, and some of the details for paying toolmakers still need to be worked out by affiliate sites.

Distributed content network

What if instead of linking only to Amazon, your content was available for republication to any aggregation sites willing to pay a bounty? Shopping comparison sites, individual merchants, and buying guides could supplement existing information with a network of reviews and first-person accounts. There may some matching and distribution service in the middle connecting the content creators with multiple sites such as Amazon, Buy.com, Overstock, eBay, etc. The matching and distribution service helps the individual blogger make more money and reach a broader audience than they had before.

Reviews are currently rated on sites such as Amazon for their level of helpfulness. These rankings provide a way to rank multiple reviews and reward authors that positively contribute to a purchase even when the research and purchase decision takes place on a site such as Amazon.

Submitting your content to a distributed network or aggregator requires some standardized data format to express key elements of your review for redistribution to interested consumers. The format should be standardized to lower a publisher's switching cost and allow for the highest possible number of consuming services.

Conclusion

I believe some of this ecosystem is starting to come together. E-commerce sites are establishing partner and developer relationships as well as affiliate programs. Toolmakers such as FeedBurner and Six Apart are establishing relationships with advertisers on behalf of users. Formats such as hReview bring together a group of coauthors representing hundreds of millions of users across web properties such as AOL, Microsoft, and Yahoo!.

These are developing thoughts, business models, and code, but there may be an ecosystem forming around improved online content for everyone.

Tags: reviews, shopping

The online audio player uses the Windows Audio format and redirects the stream through Yahoo!'s servers allowing for some additional services as the middleman. I am currently on a high bandwidth connection and Yahoo! is offering me podcasts at 56 Kbps .

Yahoo! passes the audio file location to a JavaScript file to create a new audio stream. Here is a sample JavaScript file for my latest podcast. There are a few interesting variables such as advertisements, beacons, and referrer tracking currently in the JavaScript. As the middleman Yahoo! could choose to insert audio advertisements into my podcast in addition to the large advertisements it already has planned for the center of the player window.

Update 12/16 : Joe Hiyashi from the Yahoo! Podcast team contacted me with more information about the service. He told me Yahoo! Podcasts does not intercept any streams and the audio files are served directly from the publisher at the fastest possible speeds both sides allow. The JavaScript I discovered is used for Yahoo!-generated content such as advertising and are not used for podcast content according to Yahoo!.

Tags: podcasts, podcasttech