Not so long ago, a story about the US retail giant Target became popular online. It tells how an irate, middle-American father stormed into a Target store in Minneapolis clutching a batch of Target vouchers for baby clothes, toys and cribs.
"How dare you send this to my daughter?" he demanded. "She's still in high school. Are you trying to encourage her to get pregnant?"
Trouble was, his daughter was already pregnant. Target knew that, even though her father didn't yet. It knew, thanks to the vast amounts of data it collects on customer behaviour, online and off - everything from products bought, to holidays taken, to music listened to, to social networks used - and the algorithms the company uses to make sense of it all. It turns out that Target's pregnancy predictor algorithm is spookily accurate - enough to beat the father of a teenager daughter to a crucial discovery.
The story was part of a February New York Times article on retailers and data. And it's become an emblem of the amazing, frightening reality that is fast emerging around us: one in which the mass of data each of us now creates every day will silently be used to shape the world we live in, around our personal preferences and circumstances. It's being called Big Data. And it's coming your way.
Of course, retailers have always been hungry for information about their consumers. It's the rise of always-on connectivity that has changed everything. It means that every day, each of us scatters a mass of "digital breadcrumbs" that tell a story about our lives - credit card purchases, Facebook posts, Twitter conversations, phone calls, newspapers read - none of it goes unnoticed. What's more, data scientists now have computers powerful enough to take that mass of unstructured information and draw meaningful conclusions from it on who you are and what you like.
Retailers such as Target are buying those conclusions, hence the pregnancy voucher furore. But it's not just them. Google - a key player, of course, in the Big Data revolution - this week showcased the near clairvoyance its mastery of data lends it when it launched the Google Now Android App (www.google.com/landing/now/).
Leaving your house? Google Now is automatically sending you traffic information to help with the journey to work. In a restaurant? Google Now will use your past food preferences to suggest options from the menu. Scared yet?
And it's not only the present that Big Data can get inside. Recorded Future (www.recordedfuture.com) crunches tens of thousands of data sources a day - from Twitter, to mainstream media, to obscure government records - to discern current trends in popular sentiment, political influence and financial markets. In January 2010, the company posted an entry on its blog entitled: "Yemen, heading for disaster in 2010?" The Yemen uprising occurred in January last year.
In the years to come, Big Data will unlock human mysteries - patterns of social action and individual thought - hitherto unimagined.
One thing's for sure, you cannot avoid being Big Datarised.So why not join in? At The Human Face of Big Data (www.thehumanfaceofbigdata.com) you can download a smartphone app that will record metrics about your life - from sleep, to travel, to work - and let you compare yourself to others.
After all, if Target is going to know all about you, you may as well know, too.
David Mattin is a senior analyst at trendwatching.com
