Exclusive: TypePad outage update and details

Six Apart executive status report

Popular blog hosting provider TypePad.com was unavailable to its members for 18 hours today as a result of a failed storage upgrade during scheduled maintenance late last night. I visited Six Apart’s headquarters in San Francisco earlier this evening to talk with Six Apart employees about the events that transpired and the current state of TypePad’s blogging platform.

I sat down for an interview with Anil Dash, Six Apart’s Vice President of Professional Products, about an hour after TypePad’s servers came back online. We discussed TypePad’s reliability issues over the past two months, what went wrong today, and how the company plans to prevent future outages and create a reliable service.

Anil and I had a candid conversation about the difficulties of building a large-scale hosted service, the uniqueness of TypePad’s members, future plans for the TypePad product including Project Comet and a business-level offering, and how the company approaches scalability and building dependable applications that people love to rely on as a utility.

The entire 23-minute interview with Anil Dash on TypePad service issues is available as a 10.4 MB MP3 audio download. Any rumblings you may hear is the sound of TypePad engineers running back and forth nearby and literally shaking the entire room.

Each paragraph in the interview transcript below is uniquely identified in the page markup. To link directly to a paragraph just use add “#” and the question and paragraph id found in the source.

Interview Transcript

Niall:

Hi I’m Niall Kennedy and I’m here with Anil Dash of Six Apart. Anil is VP of Professional Products and we’re going to talk about TypePad and some of the issues that happened today, Friday, and last night. Hi Anil.

Anil:

Hello.

Niall:

Can you tell us a bit about what happened last night to cause a loss of service to TypePad?

Anil:

Yeah. I think the first thing I would start with is no data was lost and the service is back up and running now. A couple weeks ago, as you well know, we had some problems with capacity and scalability as TypePad has been getting more popular. We made a bunch of commitments in terms of improving bandwidth, improving power, and things like that. Among those changes was improving our storage infrastructure and one part of that was increasing redundancy. We actually have accomplished most of the things we promised we would do on that list and one of the last things to do was to improve redundancy on storage.

We were bringing online another storage system and it’s one of those things where you have a moment of vulnerability while you’re improving the system and we had a failure during that point. So we had what we thought would be a fairly routine scheduled maintenance last night and we had posted an alert about that so people knew ahead of time “oh, OK, this is why this blip is happening” and right at the end of that window of maintenance we had a failure of the new storage system.

The process at that point was understanding what had happened and whether there were any issues of data loss, which fortunately there were not. The first step that was decided — and this is something I am sure will be avidly debated — is that we would go to the last full snapshot. We have incremental backups and we have full snapshots and most people I think that are listening to this probably know the difference. The full snapshot was anywhere from a few hours to a few days old, depending on people’s blogs. We then did a verify on the data that we were restoring.

As we brought the disks back up — the process is like the olden days of Windows 98 when you would get that reboot into the blue screen saying “I am running checkdisk and please wait forever while I check your hard drive.” It was doing that process and this is what we started doing last night and all through the night and in the interim we had those old snapshots of pages up. I think a lot of the people who were looking at their blogs today or posting had that alarm of “oh my gosh, old data” and “does this mean old posts are safe and my new posts are not?” It was making sure that was not the case that took a lot of time today. And then, of course, eliminating single points of failure, identifying all the other problems. Between all those steps together it took until this afternoon Pacific Time before we were able to switch the app back on.

Niall:

You guys run a variety of sites from TypePad.com, blogs.com, Major League Baseball blogs. What sites and partners were affected by the outage?

Anil:

All the TypePad-powered sites in the U.S.: TypePad.com, blogs.com, Friendster’s blog service, as well as Major League Baseball’s service to various degrees. I don’t actually know offhand the details about which blogs were affected on those services or what percentage it is. That’s something I know our operational team definitely knows but my job today interfere with them. My estimation is the majority of TypePad blogs were affected.

Niall:

How did you notify TypePad members of the outage and the unavailability of their recent posts?

Anil:

We had a couple of different steps. The first is our status blog which is our first line of defense and the first place you should check if you ever have a question as a TypePad user. It honestly is probably not nearly prominent enough to find and that’s something we are going to be remedying. It’s one of those things you find out in these situations is that people don’t know that channel exists. And then we’ll be blogging about it. I think one of the things we’ve shown a pretty good track record of is having a conversation about these things and really engaging in a dialogue with our members and I think you’ll see our usual breadth of posts which is here’s a technical explanation for people who care about that, here’s what we’re going to do to make it right for people — I think for most people that’s their priority — and then a little about what we’ve learned from the past is something we will go into. You work at Technorati. I think the folks at Technorati have done a really good job about being open in communication on that. I think we’ve got a pretty good reputation there and I’d like to be sure we stay in touch with everyone.

Niall:

Did you consider sending out e-mails to every TypePad member to let them know that they wouldn’t be able to post and their latest posts might not be available?

Anil:

Yeah, I think we talked about a variety of different channels and the thing you want to do is make sure you have all the right information getting to the right people at the right time. Part of it is until you know that everything is up and running and all the data is OK you don’t want to send a “we don’t know” e-mail because it tends to make things worse before it makes things better. That’s something we’ve gotten by talking to our professional users and people who are really invested in their blog. They’ve said “just let me know you are aware there is a problem and then let me know when you’ve fixed it and in between if you’re going to tell me about the fsck you’re running on your hard drives I don’t care.” I think that’s probably true for most TypePad users. “Just let me know when it works.” That’s I think what we’re really going to focus on is that blogging experience.

There are many different channels. I spent all day commenting on blogs today. Other folks here at the company have. Jay Allen I think a lot of people will see out there. A lot of people have been on IM and on Skype and on phone calls all day too. We talked about if we should do a big Skype call with a bunch of people but finding something that we can setup in a short amount of time that scales to millions of users or millions of listeners is pretty hard. I think in the future we’ll see what we can do about that too.

Niall:

How are today’s problems related to the problems you had at the end of October? Are these similar issues? Are they going to be recurring?

Anil:

No, it is actually not about capacity at all. I saw somebody describe it as “a perfect storm” today or it was like a “lightning strike.” I felt a little bit like Charlie Brown when Lucy pulls the football out. It was absolutely one of those situations where we had tackled the problems that caused application performance issues in the past and we wanted to make sure we were bulletproof. At the point where we are kind of putting up another line of defense we kind of hit ourselves on the head and got knocked out. The timing was terrible and it just one of those things where Murphy’s Law will just always in.

Lucy pulls back the football

Niall:

Are LiveJournal users affected at all or are they on separate hardware?

Anil:

Not at all. They are on separate hardware. The best thing about having LiveJournal in-house is they have really taught us, and the whole web in general really, about scaling and reliability in large-scale operations. They’ve got 9 million journals and they recently did a datacenter move to the same datacenter we’ve got TypePad in. They have hundreds of servers, millions of users, millions and millions of posts and they did it with some glitches around e-mail and things like that but no downtime. The open-source technologies they have created like MogileFS, a caching file system and Perbal, a load balancing system, run sites like Wikipedia and Slashdot and Friendster: these gigantic and dynamic sites. They’ve shown that the technology exists. People on our team build these services capable of making large services reliable. I think you are going to see a lot of knowledge transfer and best practices about how to make this stuff bulletproof and reliable.

One of the examples I think Steve Rubel talked about on his blog is eBay back in 1999 when people were just starting to have their livelihood depend on the auctions closing on time and the service would go down. People solved these things then and I think we are going to see the same thing for us.

The thing that gives me a lot of faith is we have these TypePad users that are rooting for us to do it even though we have absolutely not met their expectations, especially today. We had somebody send us pizzas for the whole TypePad team for lunch today saying “I love TypePad, I know you guys are going to do right by us, I trust you to get it right, and we are thinking of you.” That’s an astounding thing for a community to do. That’s really special, that’s why you do it. That’s why you keep going we dropped the ball today.

Niall:

Can you talk about the backup solutions that are in place to make sure people don’t lose data on TypePad?

Anil:

I’ll speak as much as I know. I don’t want to overstep what I am knowledgeable about because I am not on the operations or infrastructure teams that we have. There are, as I understand it, the now redundant storage system. One of the interesting things about our architecture is we have a database that stores all your posts but it generates and a machine that generates the pages that your posts appear on. They are separate. The storage failure today was on the system that stores your web pages that you can view and that’s why we saw an older cached copy when they went back live. The entries in the database are on a separate database server and that is already redundant and somewhere we knew we didn’t lose any data so we felt a lot better about that. We were able to verify that pretty quickly, it was just the disks that took some time to come back online. In terms of architecture I don’t know exactly how the different tiers split up but the database is separate from the web server pages and now we have redundancy on both.

Niall:

Last February in an interview Ben and Mena mentioned that TypePad was originally intended for 3000 users and then it was going to switch over to an invite-only system. Is this a scalability issue and is TypePad up to the challenge of hosting hundreds of thousands of blogs?

Anil:

I think what they were talking about there was not so much the architecture of TypePad but the concept of how many people would want to be blogging, how many people would want to be on a hosted service, and how many people would pay for a hosted service. It really spoke more to whether, I think this is something Om Malik was talking about with Web 2.0 scalability in general, are you designing for a community of people who you are personal friends with and can talk to one-on-one or are you trying to bring something to the masses? I think that’s the biggest distinction that Ben and Mena were talking about the change for.

As far as the application architecture actually I feel more confident than I ever have that we can scale out to support the number of users TypePad is hosting. We have got millions of people on this platform. TypePad and TypeKey are related and I think between the two of them you are certainly into the millions of users. That’s something where I think the learnings from LiveJournal in particular are really really educational for us.

There are a couple of different places you can fail. You can fail at app, you can fail at hardware, you can fail at connectivity. We’ve probably had each of those problems. I think that we have stated that it’s a goal for us and that it is a requirement for us to be business class. It’s really really important to be professional-grade.

I think that is a distinction from … I have this debate all the time with Jason Fried from 37signals where his team and David Heinemeier Hansson have made Ruby on Rails into this web development phenomenon but nobody’s ever done a really large-scale service, certainly not the scale of TypePad let alone LiveJournal, on that platform. David Heinemeier Hansson has said “we don’t want to invest in scale.” I think he is overstating the point for rhetorical effect, but it’s understated whether it is a priority of yours or not and whether you want to have impact with a small influential crowd or whether you want to bring this technology to the masses. Those are things that we are really clear on. I think it’s really important to us that we are a company of bloggers. I think, you know, we’re all bloggers. I’ve had a blog for 6 years now I think and there are a number of people in this company that is true for. We’ve been doing it a long time and we want it to be something that everybody can do. It’s a different goal and it matters to us that people that are of blogging and from blogging that bring blogs to a regular user as opposed to being Bill Gates saying “this is a high growth area that I think we should maximize revenue potential in.” We want people to have a voice and we think we can help them do that.

I think TypePad users today are sharing their passion for their blogs and they care that much about their sites. We are absolutely lucky to have that kind of passion from our users. One of the kindest things today was a friend of mine on the Blogger team IM’s me and said “hang in there, we had a 7-hour outage one time” and everyone has had different outages but to be really supportive and also to say we’re kind of kindred spirits and we’re a team of people that are bloggers too. It’s a nice thing that there is still that sense of community and that people notice.

Every major blogging service except you guys has had downtime this week. Del.icio.us got unplugged for a while, and I think Bloglines said “they are sucking eggs” was their quote. That’s nothing negative against those guys, I have huge amounts of respect for both those teams, and we probably share a huge overlap in users too so it’s the same people we are talking about. It’s a hard problem and it has to be something that either that’s your priority or it’s not. You only get the problems that you are uniquely qualified to suffer for. TypePad is the thing we care about and so this is the burden we take on.

Niall:

I know you guys are planning some changes to TypePad’s infrastructure including some behind-the-scenes stuff you are doing as part of Project Comet and some other initiatives. Can you talk about that and what’s being done?

Anil:

I don’t have a lot of specifics there because I think a lot of that stuff is always evolving based on what feedback people have. Frankly we redoubled our efforts on TypePad’s infrastructure sometime before the problems we had in October. It was something we recognized and we wanted to focus on and the first step in that was moving our server location. We didn’t get done entirely in time for that but it did happen and pretty smoothly once the performance issues were addressed.

I think we’ve got our best team, some of the people who have been looking at TypePad the longest, really looking at all of the different ways TypePad can scale. Something you and I, Niall, talked about that we haven’t really announced but the intention is to do a business-level TypePad. We have the kind of flagship customers that some people were blogging about today like MSNBC or ABC News or USA Today are all running their blogs on TypePad. That’s an incredible responsibility. It’s just as serious to me as people who have their baby pictures and talk to their friends and family on there. That’s an incredible amount of trust to put in us and I am at least heartened that it made the news that we let them down today. You want it to be newsworthy when you fail, not when you don’t.

That much is good, but we do have a bit of a hole to dig out of as far as people trusting us and I don’t know what the ratio is. It’s maybe for every day you screw up you have to have a month where you’re perfect or a week, or whatever it is. That’s something I have a lot of confidence people are going to give us the chance to get right because we’ve had that dialog with them because they know that we are out there listening and they know they can call me or Skype me or IM me or TrackBack me or whatever and we’ll all reply.

Niall:

Talk about that “business class.” How does that differ from what there is now? How do you define business class reliability at Six Apart?

Anil:

We haven’t gone into specifics on it and I think obviously this is something we are going to redefine based on today to some degree. It really is about saying that it matters to us to be a professional tool people can trust. We understand that the constraint for a lot of companies with blogging is they don’t want to deal with their IT department. You’d rather sign up and just use something you can expense on your credit card and ask forgiveness rather than permission, although we’ll probably say it nicer than that on the website. People also want an option for priority support, people want to be able to have that relationship with us, people want to optionally purchase a service-level agreement.

I think that’s absolutely a reasonable option for people to have if they think that it’s worth their investment. Those are things where we’ve had people say “I do this for fun” or “I like a service to be free” or “I don’t like to deal with SLAs or boring legal stuff,” bring on the boring legal stuff. If we can do things the way we want to for the business audience blogging should be deadly, dull, boring. Something that you just don’t even have to think about, it just runs like an e-mail server or a web server or a search server where it’s just there as a utility. I think you guys have stated probably different terms but similar aspirations for Technorati where you just count on it, you know it’s comprehensive, and you know it’s there. That’s definitely a parallel to what we want to do.

Niall:

Are you guys going to take any time off this Christmas? Is your team going to get a break at all?

Anil:

I work most closely day-to-day these days with the Movable Type team and they have had a great week with the Yahoo! partnership we announced so they are going to have some well-deserved time off.

I personally am going to be traveling and dealing with some of our partnerships overseas. The nice thing about having a break here in the U.S. is it’s not always taken in other countries so we get some time, some downtime, to really connect with our partnerships in Japan and things like that.

There’s no question to me that everybody who is in operations is going to be 24/7 for the foreseeable future on their beepers and on their phones and on-call. They have done that before and it wouldn’t be the first time that a hard drive was the grinch that stole Christmas. That’s what they signed on for. They really do have an amazing amount of dedication to making sure people can connect with the people they care about so that goes a long way.

Niall:

Is Six Apart hiring operations personnel to take care of some of this stuff?

Anil:

We’ve brought on a new director of ops, we do have a bunch on new people on the ops team. We’ve been building up that team for some time and like I said, it’s funny I was looking at one of the performance monitoring systems that monitors TypePad and we had these old cached copies of the pages coming up but they were coming up really fast! It was one of those things where that was an improvement that had been made, a performance improvement and a reliability, and even when you have this hardware failure there are these other things that are good news.

They are a really dedicated team. This is not something that was any human failure or error. I have a tremendous amount of respect for them doing hard work day in and day out.

Niall:

If there are aspiring sysadmins out there should they be sending in their resumes?

Anil:

Yeah. I think anybody that’s got talent and cares about blogging we want to have on our team and have them contribute. So yes, please.

Niall:

Where is the best place for TypePad users to send in comments or feedback?

Anil:

There are a couple of different ways. We have our help ticket support system built-in to the app. People who use TypePad know that support team is pretty legendary for their dedication. Feel free to get me directly. I’m Anil [at] sixapart.com so I am pretty easy to get ahold of for most people. I know Barak our CEO is also willing to get feedback. He’s BarakB [at] sixapart.com.

Those are great ways to get us. We are going to have TrackBacks on the posts that we will have up about this so feel free to link. Blog about it, we do our Technorati searches and find everybody talking about what we’re doing. We probably can justify the ego surfing more than most people so give us a mention and talk about it. We’ll try to be out there. We don’t guarantee we’ll read every single post but we try to.

Niall:

Thanks Anil. I hope you get some downtime this Christmas season.

Anil:

Bad word!

Niall:

I hope you get some relaxation time this holiday season and I hope everyone out there using TypePad keeps on blogging.

Anil:

I appreciate it and thanks for giving us a chance to talk to everybody.

Tags:

18 comments

Commentary on "Exclusive: TypePad outage update and details":

  1. Russell Buckley on wrote:

    For me, there’s five letters missing from this. S O R R Y.

    I also don’t think that Anil realises that the wind has changed for 6A. There’s very little evidence of tolerence and support anymore today and an awful lot of anger.

    We’ll certainly be moving platforms and I think most of their core, influential and hitherto loyal bloggers will too.

    Nothing personal, guys, but you’ve blown it.

    Russell

  2. Anil Dash on wrote:

    Russell, we’re very sorry we’ve let the TypePad community down.

    The tone and tenor of the conversation I had with Niall is reflective of what it was, a conversation with a friend in my office. I definitely understand the severity of the issue (my own blog was one of the ones that was down) and I hope you’ll be able to see how much it matters to us to make things right over time. I also hope you’ll give us the chance to get it right.

    I think you and Carlo will both find over time that TypePad helps you do the most professional and, yes, most *reliable* job of communicating with your core audience. I’m eager to show you that this is the case, and I know our whole team will work to make sure that’s true.

    By all means if you’re concerned or (rightfully) skeptical, feel free to get in touch and I’ll be glad to talk anytime. You can usually reach me on IM or Skype as anildash, and my email’s in the interview above.

  3. jeremyet on wrote:

    A couple of quick comments.

    Our site was down all yesterday (terrible timing as the BBC had linked to our podcast and we were getting huge traffic) and when service was restored, our recently submitted files had been wiped from the typepad servers, leading to ‘file not found’ errors. This might have happened to other users.

    Also, communication was not great yesterday. We’re based in the UK, so 8 hours ahead of you guys, and perhaps because of the time difference, the typepad status blog was not really keeping us informed.

    I’ve always been impressed with typepad in terms of flexibility and customer service. I really hope Anil is right and that yesterday’s hiccup was a one-off.

    jeremy

  4. Dan on wrote:

    Putting aside what may, or may not have been avoidable from a purely technical perspective today, as a user, I had absolutely no pro-active communication from 6A to tell me I had an issue.

    Furthermore, it fell to me to be an adhoc help desk for far too many users who emailed seeking answers. Eventually, I was able to direct them to the status updates which, as stated, are far from readily available.

    I understand this is a complex issue even in its resolution. More or better staff costs money, that impacts pricing, so on, and so forth.

    Given the way things have been handled over the past month, on top of reading some bad press from a recent conference which made the management team look, at best, immature, trusting 6A to make good complex decisions going forward is going to take more than a month or two of good service.

  5. Robin Capper on wrote:

    Hi,
    One thing you could do is make it possible, easy, for a blogger to create a full backup of their site independent of TypePad severs.
    I can cope with an occasional hiccup with service but having just uploaded, added titles and comments on 130 images the day before the outage was somewhat distressed to see them gone today. Yes, they will be back – hopefully… but what if your backup system has a total failure. There is a lot of time invested in content but users have no easy way to archive it. I’d happily devote a few mb of my harddrive to maintain an off-line backup.
    You can export post content but not the site structure, templates, photo albums etc as a whole. Perhaps something to consider?

  6. Frank Gruber on wrote:

    Great job on getting the scoop Niall! I wanted to respond to the comments left by Russell Buckley in saying I also spoke with Anil Dash over IM yesterday and he was very apologetic. So the five letters are no longer missing. Six Apart is sorry.

    As I mentioned in my recent post on Somewhat Frank, I am sticking with TypePad but I am going to start backing up my blog content locally in case this sort of thing happens again. I would urge others to do the same. I am also considering hosting all my blog images on Flickr instead of TypePad since I would rather not have a single point of failure. It appears that you do that Niall, how has your experience been in doing that?

  7. Niall Kennedy on wrote:

    Hi Frank,
    I host my images on Flickr for use in my blog posts for virality and for broader distribution. All of my images are licensed under Creative Commons and their availability on Flickr allows other bloggers to reuse the image with attribution at whatever size works best for them on their post.

    I recommend Flickr for blog images.

  8. Randy on wrote:

    I guess I don’t understand the whole “hosted” blog thing for anyone serious. It’s not expensive and fairly easy to install your own blogging software under a LAMP setup so you have your own url and are in control of your own destiny.

    I happen to use the commercial version of Expression Engine, they have a free version called EE-Core, and am happy but there are others. It’s not hard and you probably have far more features if you want.

  9. Scottitude . on wrote:

    Great interview. Unfortately, some users appear to have unreasonable expectations. This is true in any industry whether it’s manufactured goods, retail, and apparently blog-hosting.

    Before building my TypePad blog, I used practically every ‘free’ service out there and earlier this year chose what I thought to be the best one but incompetence and an absolute failure of communication from site administration during TWO WEEKS of complete downtime pushed me to research and find a stable alternative. I chose TypePad and haven’t been disappointed.

    Sure, It was unsettling to see last week’s content without anything current, but but one click took me to the support page where I the situation was explained.

    The alternative is to host your own blog on your own domain with your own software or establish a network of two or more blogs to ensure accessability. Even a separate host can have downtime and none are as communicative as TypePad has been about reliability issues.

    I passed the time using my free EFx2 Blog. EFx2 is a reliable, member-focused blogging community, and a great alternative for any blogger to consider. Had I found EFx2 before signing on with TypePad, I may not have felt the need to pay for a host.

    Still, I love my TypePad blog and will continue to pay for it. I appreciate 6 Apart’s professional management team & the high level of communication and support they provide. some people simply can’t be satisfied, and the nay-sayers here are among them.

    Hats off to TypPad for their service and communication.

  10. Rick Cummings on wrote:

    Great interview Niall and my hat’s off to Anil and the Typepad crew. I appreciate and understand that Typepad’s success is creating the growing pains that we are seeing right now. I much prefer that Typepad grows and implements for the long haul. They are making an effort to handle the problems they are faced with and communicating that the best they can. You will always get a group of people that scream like stuck pigs when something is not perfect. For them, it is the controversy that brings them joy. I appreciate that during the service problems last month Typepad went above and beyond to make it right with their users. The fact that they offered a service extension to their users proves they understand “Do the Right Thing”. I have been a Typepad customer since Nov 2003 and I am a happy customer. Is the service outage an inconvenience? Yes. Is it a show stopper? No. Life goes on and they will fix this only to have it break again down the road. That’s what a growing business does. Grow, Break, Fix, Repeat. It’s the never ending challenge. You just hope to make good decisions and keep the majority of your customers happy. Anil, keep up the good work and remember that there are no problems, only opportunities to prove why we chose Typepad. It is how you handle that opportunity that you prove yourself and earn the loyalty of your customers.

  11. Eric/Fyre on wrote:

    LiveJournal not having any downtime during its datacenter move is not correct. LiveJournal did experience some downtime, about an hour or two, as a result of DNS propogation issues. Their image hosting service (pics.livejournal.com) encountered about three to six hours (it went up and down a few times, so I’m uncertain of the actual amount) of downtime. Posting by telephone encountered a lot of downtime.

    Overall, the site was accessible, save for maybe two hours total. Certain functions/portions of the site had extended downtime, but nothing that qualifies as excessive.

  12. Niall Kennedy on wrote:

    Eric,
    My question about LiveJournal was related to yesterday’s TypePad outage and how it might affect other services offered by Six Apart.

    LiveJournal’s datacenter move last month may have encountered a few glitches but that’s to be expected when you move data from one city to another.

  13. Sarah on wrote:

    Nice scoop!

    Linking to you now…

  14. paul on wrote:

    Six Apart has received millions of dollars in funding but has chosen to spend more generously on PR then technology.

  15. Russell Buckley on wrote:

    I just wanted to respond to Rick’s post about people supposedly getting some kind of kick out of this situation.

    Over at MobHappy, nothing could be further from the truth. We’re building a business out of blogging and our software/hosting is therefore mission critical. We’re not dabbling in vanity blogging and if we were, I can see how this is all a bit of a storm in a teacup.

    Furthermore, I was very supportive on the blog over the last outrage and wrote about 6A in contrast to Sony’s DRM fiasco.

    I also get no pleasure in watching 6A, a company I’m sure, full of committed people go through such a terrible experience. There but for the grace of God, go I.

    But my original point remains. Yes, it’s very nice of Anil to apologise in the comments here and I’m sure they’re truly heartfelt.

    But 48 hours after the outrage, I still haven’t heard anything officially from 6A. At the very least, I’d have expected everyone to get an email by now saying “Shucks, sorry, we screwed up again and we know it. We’re working out what to do and will get back to you shortly.”

    It might cool some of the anger that’s definitely out there. We don’t feel anger, by the way, just mildly pissed off that we have to go through the hassle of changing, when we’d rather be blogging.

    We NEED reliability, otherwise we’d hang in there and let them go through their growing pains.

    Cheers

    Russell

  16. Anil Dash on wrote:

    Russell, thanks for the feedback… I think there’s a tough balance to strike here, where I’m glad to apologize to you personally but you also want “official” word. We hear from other members of the community that they’d prefer to talk to us directly as individuals, rather than get a general blog post that anyone can read. So I think we’ll be doing both, but it may just be that different people prefer different methods of communication, and we’ll work that out over time. I do expect you’ll get formal communication from Six Apart as a whole on Monday, with the goal there being to make sure we’ve got a full understanding of everything when we communicate formally.

    “Six Apart has received millions of dollars in funding but has chosen to spend more generously on PR then technology.”

    I don’t usually respond to these kinds of ad-hominems, but this is just demonstrably false and makes you look silly, Paul. Your blog’s not on TypePad, I’m not certain that you’ve ever contributed to the TypePad community (or to any other community around Six Apart platforms), and so it seems almost impossible to me that you could fairly judge how resources are being devoted. Given that you’re on a platform which lacks literally dozens of features which TypePad supports, it seems likely to me that your ignorance of our technology informs your statement.

    If you’d like to know exactly how we’ve focused on technology for the future, feel free to get in touch and I’ll be glad to have a dialogue with you. If you’re just interested in not being constructive, there may be other places to focus that effort.

  17. Jason Corsello on wrote:

    As a great philospher once said…”Screw me once, shame on you…screw me twice, shame on me!”

  18. paul on wrote:

    I re-read this just now (after a couple of days) and am still struck by the comments that Anil and Jay Allen spent their time commenting on other blogs when they should have been communicating with their user community. Is there no mailing list for typepad subscribers? Not that the my-users list was ever used when I was a MovableType user . . .

    I left the MT community with the launch of 3.0 after too many performance issues that 3.0 wasn’t going to address, and one of my chief gripes was the 6Apart style of communicating: come read Mena’s Six Log or be in the dark.

    We were all asked when we downloaded the MT install kit if we wanted to join a making list, and I never saw a single email from it. Upgrades, bug fixes, patches, all news of that was delivered via word of mouth, as notes on other blogs.

    I thought 6Apart was about communications and community but I never saw it when I was part of it and it still seems to be missing. These are paying customerts now, not just eager hobbyists.

    This doesn’t make me regret my decision to move on.