Many folks will be familiar with the slashdot effect whereby a site gets linked from slashdot and falls over dead under the load as many thousands of Slashdot readers follow the link. A good definition of the pheonomenon is provided in the Jargon File under, oddly enough, the entry for "slashdot effect". The term slashdot effect has taken on a somewhat more generic connotation and doesn’t always refer to Slashdot specifically. This generic term is used in this discussion.
Many site owners would love to be on the receiving end of a slashdotting. But at the same time, the dread it. Who should be responsible for protecting a site from such an effect? Indeed, these questions are raised by a recent twofold slashdotting. On February 3, Google had a special logo commemorating Julia fractals. Clicking on the link yielded a number of search results, a couple of which pointed to one server in Australia. This clobbered the site. Then two days later, Slashdot notices and posts a story about it and the poor site gets clobbered again but this time by actual Slashdot readers. Their own description of the events is here. Interested readers should also follow the link to the followup story.
The page referenced above askes a few hard questions. These include the following:
- Should Google ask permission before doing something like this?
- Should a site directing a lot of traffic in one direction take steps to rate limit it?
- Whose responsibility is it to protect against this sort of scenario?
Obviously, I have a take on these questions or I wouldn’t be writing this. So, without further ado, my discussion of these questions.
First, should Google, or any other site, ask permission before directing traffic in large volumes to a particular site? It seems to me that there are two dimensions to this question. First, is not the very fact that a web page is publicly visible tacit permission for people to look at it? And, by extension, is it not tacit permission for someone else to link to it? After all, isn’t this the point of the web in the first place? But then, one could argue that just putting up a page on the web does not mean that I am willing to handle a massive flood of traffic. Should I be notified when someone makes such a link to my site so I can take steps to deal with it if neceessary? How would they necessarily know that they would be clobbering my site? In a dynamic database like a search engine, how would they know that one particular site is going to get clobbered?
My answer to all of that is this: no site should require permission to link to another site under any circumstances. If the information is not intended for public consumption, it should be protected against the casual browser. In the case of a static link from one site to another, it should be a matter of courtessy to notify destination site of the link, especially if the source site is busy and the destination site is not known to be able to handle the possible click throughs. My reasoning is simply that putting up a site for public consumption is an implicit approval for the public to look at it. And talk about it. And point their friends at it. And so on.
On to the second question. Should a large site take steps to rate limit the traffic that clicks through its link? This is a different animal all together. This depends heavily on the nature of the site. For example, Google, who tries to provide the most relevant results could argue that by juggling the results around, they defeat their own purpose. The opposite could be argued as well, especially in the case where there is no particular real difference in relevance between the top results. A static link from a busy site might make the same arguments, one way or the other. But all this leaves aside the question of how do you partically rate limit click throughs without having your own site extremely dynamic? For the likes of Google, this may not be a problem, but for a busy site with static content, having a link appear and disappear may be an unbearable load on their own server. Various other methods of rate limiting would have their own limits on success.
My take on this question is this: The site posting the link should not be required to take steps to protect the site they are linking to. However, courtessy would dictate that they do everything in their power to deal with a problem that is caused directly or indirectly by such a link. After all, knocking the site you are referring to off the web is not the best way to use that site’s information.
Now for the big one. Whose responsibility is it to protect against this type of scenario? This overlaps to some extent with the discussion of the previous questions. Extending my stated arguments forward, the clear answer is the site owner is responsible for dealing with the consequences of operating a site. That is to say, if I put up a site that becomes popular, I have to deal with the consequences of doing so. It is a risk taken by anyone who puts up a site that has anything of even remote interest. There is another dimension to this, however.
I, as an end site, do not have all the tools ncessary to handle an incoming flood of traffic. I cannot block traffic until after it has come across my bandwidth (or that of my provider). Thus, even if I take all measures available to me, I cannot slow down the inbound torrent. Fortunately, this is usually not the choke point, except on very popular sites on relatively low bandwidth connections.
What I can do as a site operator is make certain I have a version of the site that can be switched on that creates as small a bandwdith and processing footprint as possible. Then, if that doesn’t work, temporarily disabling the site might be a reasonable solution for the short term. This can deal with short term demand that dissipates.
There is little a hobby site operator can do in the face of a massive increase in demand that stays. If the operator cannot afford to continue operating the site, it must, obviously, be removed. This is unfortunate for the internet, but it is a fact of existence on the internet or, indeed, anywhere else. For example, if I cannot afford to maintain an office building, I must sell it or demolish it.
All the above said, since everyone on the internet can be the victim of the slashdot effect, everyone should do their best to mitigate such an effect. This includes linkers who should work with site owners to reduce the impact of linking. It includes internet providers and their upstreams who should cooperate in controlling traffic floods. Web hosting providers should make available the ability to rate limit web sites. And so on. But the web surfer himself has the most power to mitigate such a siutation. Web site operators could ask for mirror sites to be set up if possible, and, in particular, should not discourage mirrors of content when faced with an slashdotting.
I make the following recommendations to all web surfers. If we all followed them, we may reduce the impact of so many slashdot effects.
- When visiting a site which is responding very slowly, go away for a period of time, preferably longer then half an hour. Continuing to stop your browser and reload the site in the hope that it responds only makes the problem worse.
- If you know of a mirror of the site that is not responding, use it. In many cases, there are mirrors of the site which are hardly getting any traffic yet which are clearly marked in the links to the site.
- If there are mulitple links that idenfiy themselves as mirrors, all the content should be the same. Do not always click on the first one in the list. (And site operators should, if at all possible, make the list itself rotate.)
- Do not hammer on your reload button when the site takes longer than three seconds to display. This only slows it down and makes the load on the remote end higher.
Well, that about sums up my ramble on this topic.