Comments on Facebook’s photo storage rewrite Niall Kennedy's home on the web 2012-07-02T14:52:24Z http://www.niallkennedy.com/blog/2009/04/facebook-haystack.atom Niall Kennedyhttp://www.niallkennedy.com/blog@niallkennedy.com http://st.niallkennedy.com/icon32.png http://st.niallkennedy.com/niall.jpg Creative Commons Attribution Non-Commercial 3.0 United States By: David David http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2399 2009-04-09T20:11:27Z 2009-04-09T20:11:27Z Where can I find this “Cachr”? I can’t find anything on it online. I want to try it out.Cheers, DT

]]>
By: Paul Lindner Paul Lindner http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2398 2009-04-08T20:08:22Z 2009-04-08T20:08:22Z It would be interesting to see this compared with MogileFS. I’m especially curious to know how they handle replicas between data centers and how they deal with commodity hardware failing at regular intervals.

]]>
By: Terry Terry http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2397 2009-04-07T21:59:46Z 2009-04-07T21:59:46Z What are they using for backup in Haystack?

]]>
By: Anna Merkin Anna Merkin http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2396 2009-04-07T00:48:26Z 2009-04-07T00:48:26Z Are most photo sharing collections long-tailed like this? I wonder what is the distribution of calls to a typical photo collection over a given year. For example, do they get 90% of the calls within the first 2 months of upload?

]]>
By: Erik Giberti Erik Giberti http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2395 2009-04-06T22:59:37Z 2009-04-06T22:59:37Z The idea of this scalable filesystem using commodity hardware is what intrigues me. Small companies can get something like this from Amazon and other cloud storage platforms, but the costs – both fiscal and latency – become unsustainable long term. Furthermore, when S3 goes down, so does your app (and likely your revenue). Not that these solutions work at Facebook scale anyway – but you get the idea.

]]>
By: Phil Strong Phil Strong http://philstrong.wordpress.com/ http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2394 2009-04-06T19:41:35Z 2009-04-06T19:41:35Z Am I the only one thinking it’s about time?  Any one with this much overhead and this many engineers should be jumping at ways to do things themselves.  A project/plan that saves even 1% cost is worth it when dealing with these incredible usage numbers.

]]>
By: Eric Eric http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2393 2009-04-06T17:47:57Z 2009-04-06T17:47:57Z This is a pretty fascinating look at what Facebook has done, they know their business and have optimized for it.  I wonder how much this applies to most other companies, though – most of us are not Facebook.
Facebook made their make or buy decision, choosing make, but most of us would worry about our ability to support that over time.  If we construct our own one-off solutions using our own Jason Sobels, what about when they leave?  I’ve had to abandon perfectly good applications which were core to my company when we could not get sufficient support.
The hacker in me loves this though – this seems like a brilliant hack

]]>
By: Ben Godfrey Ben Godfrey http://aftnn.org/ http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2392 2009-04-06T16:52:06Z 2009-04-06T16:52:06Z It would nice to see Haystack open-sourced. With more tools for high-scale in the open source domain, fewer companies who have great ideas but not the resources to scale will die. That’s a hot topic currently, but even in the good times, anything that lowers the cost of building new stuff levels the playing field.

]]>
By: Scott Johnson Scott Johnson http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2391 2009-04-06T15:26:15Z 2009-04-06T15:26:15Z Hey Niall,Great article.  Thank you for writing this up.  I’d be very curious how they’re setting up those commodity servers if you have any insights on that.

]]>
By: Mark Nottingham Mark Nottingham http://www.mnot.net/ http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html#comment-2390 2009-04-05T23:29:14Z 2009-04-05T23:29:14Z Hmm, seems remarkably similar to Flickr’s architecture, except they use Squid instead of lighty.Using a CDN in front of such a long-tailed collection of resources doesn’t make any sense; I’m sure Akamai et al loved them for the revenue flow, but they weren’t providing much value there, at all.

]]>