blog.noetech.com

I'd say that most people who think it's *so* easy to scale social networking sites haven't thought enough about the problem. Here are some of the issues, described by one of the engineers there:

http://www.sims.berkeley.edu/academics/courses/is255/f03/lectures/lecture22.pdf

Building a computational cluster or 'supercomputer' is not is what is needed, even remotely. The supercomputers mentioned in that link have little or no users, interacting with it in barely predictable ways.

Why is the site slow ? Because 5 million people use the same system as the 500 or so when it launched, and any major changes to the site's architecture greatly affects the rest of the system.

It's just like people complaining about traffic in urban areas...it's always viewed as a "simple" problem to the everyday joe sitting in traffic, but in reality many cities have many smart people who have backgrounds in traffic analysis and can't come up with the answer.

Basically, this problem of scalability is not specific to Friendster. No other social networking site has such a load on it, by a longshot. Orkut, LinkedIn, Tribe....they will all face that same problem.

I never claimed it was easy, just that 13 million is probably enough to do it.

There are plenty of sites with massive loads and heavy database activity that do just fine. Don't try and tell me it can't be done.

13 million sure is a good amount of money to do it, but to do it right will take more than 4 months to get it done. (they got the money in November)

Again, sites with massive loads and heavy database activity are not apples and oranges. You can't compare the use and backend architectures of a social networking site to existing sites. Ebay and Yahoo both use databases and have very heavy load, but what's happening on both situations is very simple, and their architectures reflect that. Social networking is not just web frontends to databases...there is more calculation going on than in say, a Paypal or Yahoo.

Doing the math on how each user is connected (if at all) to another person, in real time, and in which directions, is a very costly operation, from a computational standpoint, and for these sites (friendster, linkedin, etc.) to be usable in any way, this math has to happen, for every user, everytime they log in, and cannot be easily cached.

Either way, I do agree that 13 mil should be enough to do it. My objection is not to whether it could, and will be done. My frustration is that the use and popularity of these sites have presented technical challenges that companies have not seen before, and that point is very difficult for people to realize, since the prevailing thought is what you imply:

that this has been done already, why is it so hard for places like friendster to be fast ?

People have the perception that social networking sites are essentially just another database-driven website. The real fact is, they aren't at all, and will have to undergo a lot of tuning over the next year or so to get it right. (yes, at least a year) The problem with the lesser popular sites (i.e. not friendster) won't be experiencing the same scalability issues until later, because they don't have the same growth as friendster.

So my point is a little orthogonal to yours. Friendster has the money now, and one can assume that they are using it wisely to address the scalability and performance issuse. But it's not gonna happen right away.

You point is well taken.