Critical Section

the referer mess

Sunday,  01/20/08  10:33 AM

I started blogging a little over five years ago (!), and at that time referer logs were a really cool thing.  (BTW, yes, "referer" is misspelled; it always is, going back to the early days of the 'net when a bad speller defined the HTTP header syntax.)  It was fun to see who was linking to you, and to find interesting new sites in that way.

Quick digression if you don't know what this is about:  Whenever you click a link on a web page, the website which serves the new page gets passed the old "referring" page's address.  By looking at server activity logs, a website owner can find all the websites which link to his.  In my case, I can find other blogs and sites which have linked to my blog, which is fun.

So now, in 2008, referer logs are basically a cesspool of misinformation.  If you are ever able to find anything useful of interesting in there, good luck.  First you have all the spiders; programs which access pages on your site for indexing purposes, their referers are often faked or meaningless.  Next you have access via RSS feeds, feed readers typically don't even pass a referer.  And then you have referer spam, which you might remember from my referer spam wars, where various websites send you fake referrals in the home that you'll click through to them.  (This is a weird and despicable practice, first, the volume from such traffic has to be uselessly low, because only webmasters see the spam, and second, no webmaster is going to care about a slimy site which is advertised in this way.  But I digress.)  And finally you have just plain old garbage; various bots which don't access anything useful and which pass junk as a referer.

This is a classic tragedy of the commons, we see this all over the 'net.  In the bad old days it worked great, but people have subverted the mechanism for other purposes to the point where it barely works at all.  The legitimate stuff is till in there but the signal to noise is decreasing to the point of uselessness.  Kind of like email :(

About Me

Greatest Hits
Correlation vs. Causality
The Tyranny of Email
Unnatural Selection
Aperio's Mission = Automating Pathology
On Blame
Try, or Try Not
Books and Wine
Emergent Properties
God and Beauty
Moving Mount Fuji The Nest Rock 'n Roll
IQ and Populations
Are You a Bright?
Adding Value
The Joy of Craftsmanship
The Emperor's New Code
Toy Story
The Return of the King
Religion vs IQ
In the Wet
the big day
solving bongard problems
visiting Titan
unintelligent design
the nuclear option
estimating in meatspace
second gear
On the Persistence of Bad Design...
Texas chili cookoff
almost famous design and stochastic debugging
may I take your order?
universal healthcare
triple double
New Yorker covers
Death Rider! (da da dum)
how did I get here (Mt.Whitney)?
the Law of Significance
Holiday Inn
Daniel Jacoby's photographs
the first bird
Gödel Escher Bach: Birthday Cantatatata
Father's Day (in pictures)
your cat for my car
Jobsnotes of note
world population map
no joy in Baker
vote smart
exact nonsense
introducing eyesFinder
to space
where are the desktop apps?