Data Logging

I read a BBC News article titled "Web porn: Just how much is there?" - something admittedly I would not have expected the BBC to even cover in the first place.  The article content didn't interest me much as I have read various estimates and guesstimates on this over the years and in my own experience you find porn in the places you would least expect - more than that the definition of porn itself is one that is not that clear, you would likely argue for instance that Youtube does not contain porn but that really comes down to what you define as porn, if we are talking about things you find sexually arousing and what you can masturbate to then for quite a lot of people yes there is porn to be found even on a site that officially does not allow it.

This is getting beside the point though, what did interest me about the article was this rather curious quote:

"According to estimates from Scandinavian research centre Sintef, 90% of all the data the human race has ever produced has been generated in the past two years"

Now if we ignore the debate about when the Human race was born and simply go with the one thing we can probably agree on - the birth of civilisation, which is marked to have been around 10,000 BC, that means that we as a race have lived in some form of civilised society for the past 12,000 years give or take.  To consider the possibility that in the last 2 years alone we have generated more data than humanity did in the preceding 11,998 years is something that at first is quite unbelievable.

However, considering this quote and taking time to reflect on it, I do have to ask, is it really that unbelievable?  I mean we live in a world that is connected like never before.  There is an unprecedented amount of monitoring and logging of data to be processed - both authorised and unauthorised, through consensual agreements like your ISP's right to log every connection you make, to espionage and snooping.  Computers themselves are a remarkable testament to how fast we are progressing now than at any other point in history.  Beyond our habits on a PC though our day to day lives consist of fountains of information generation.  Every single thing we buy is logged in some way or another, anywhere you get a receipt a copy of that transaction is stored in the computers of that retailer.  From food to phone bills.

The question is raised - does this actually represent an increase in production or are we simply living in a time where the data we generate is recorded?  i.e. did people thousands of years ago generate just as much it was just not recorded?  We only have to look at our own society and see things that we had the capacity to record for a long time but never bothered - weather records from hundreds of years or thousands of years ago for example.  It would have been simple enough to record a brief summary of the weather day by day for future generations to reference.  Instead we end up in the situation where a lot of our historical records only mention events that were abnormal. 

That does inspire another question though, maybe we should consider taking a leaf out of our ancestors' books and consider whether we really need to record everything that can be recorded - i.e. consider what is and is not noteworthy and only record that which is and discard the rest.  The only problem I can foresee with that is that future generations would likely end up in the same situation as we are now - they might one day want to draw on records of something we decided wasn't noteworthy; maybe our ancestors thought "Oh no-one is going to care what the weather was like every day, don't bother writing that down"

The question remains, should we really be recording everything we can just because we can?