Have you ever phoned a large company, navigated your way through their phone menu system, and then had to explain the same thing to multiple people within that organisation?
In this day and age, you would think that your account details, and perhaps a comment log describing your previous conversations, could “follow” your call across the company so that you didn't keep having to repeating yourself. It seems this quite a hard thing to do though.
The 'Single Customer View'
For years, companies have been trying to effectively bring together all the data they have on a single person. The so called 'Single Customer View' or '360 degree customer view' is nothing new. The thing is, Big Data doesn't really make this any easier. If anything it makes it harder, because technology is now capturing more data, from more places and in more formats. You can already track every web page that everyone ever viewed, how long they spent there and where mouse went. I expect soon we'll be able to track how people are moving around a bricks and mortar store too; how long they spent standing near the Bananas, which products they picked up to look at, which sites they browsed to on their phones using the free in-store WiFi and how often they picked their nose while waiting at the checkout.
NoSQL technology lets you store all that data cheaply, without having to figure out up front how its going to be used. But if you want to use it, someone somewhere is going to have to link everything together. That's the hard bit.
If my energy supplier can't figure out how to seamless link my electricity account to my gas account, maybe I don't have to worry. They're probably not going to be emailing adverts for extra insulation every time I turn the thermostat up any time soon.
There are two particular approaches that might be cause for concern though, pattern recognition by automated algorithms, and deep dive analysis that focuses on particular customers.
Machine Learning and Pattern Recognition
When you have more data than one person can reasonably make sense of, one approach is to point a machine learning algorithm at it and wait for it to “learn” where the patterns are. Machine learning is currently keeping my inbox free from Viagra and Louis Viton Handbag offers because certain worlds, or certain series of words, or the servers that the emails originate from, have become associated with spam in the minds of these algorithms.
Machine learning is on the rise and is giving people all sorts of insights that previously didn't have. There are some interesting examples of machine learning being used to predict crime, we're not quite at the level of Minority Report yet though.
While algorithms making predictions about my future behaviour might be a little unnerving, a prediction is not in itself an oppressive thing. The extent to which those predictions affect how other human beings treat me is hard to predict. Should we be worried? Maybe.
The Deep Dive
Another approach to handling large volumes of data is to focus in on small subsets of it and really go to town on how specific individuals are behaving. One imagines this is what the NSA and GCHQ are doing with all the data they're gathering. Once they've identified a person of interest, almost everything that person has done can be retrieved, and a team of analysts can debate the significance or meaning of each action.
The idea that a team of people might spend a morning poring over my selection of flat screen TV certainly seems more unnerving than, say, a machine learning algorithm spotting that people will pay more for TVs that are showing Pixar films in the store. But again, where this really makes a difference is in how people's behaviour is effected by the conclusions of that analysis.
Make Something People Want
In some ways, I'm an ardent capitalist. I think the point of a company is to make something people want. In that sense, the message of capitalism seems to be that if you want to get rich, you have to first make other people's lives better in some measurable way. It doesn't always work. You get monopolies, corruption, false advertising, the tragedy of the commons, and some benefits that just cannot be monetised. But if the only outcome of Big Data is that companies get much better at understanding what people want, then we're in for a bright future.
Insights Don't Shoot People, People Do
Its easy to see how the insights that Big Data will bring could be abused. What if companies use it to manipulate me into buying things I don't want? What if law enforcement decide I'm about to do something bad and pre-emptively detain me? What if a company gives me a bad service because it's decided I'm unlikely to leave?
Ultimately though, it's not insight that is evil, its the action that people choose to take as a result of the insight that matters. The big data genie is out of the bottle now, there is no putting it back. But over the coming decades I predict that ownership, control and security of data will become a hot topic.
I think we need to start at the top with a strong legislative framework for data, and back that up with a culture of professional ethics among Business Intelligence professionals. The practice of casually sharing data on spreadsheets or giving database access to anyone within an organisation will become a thing of the past. It may take several cases of highly publicised abuse to make it happen, but over time I think it's the only viable option.
Is Big Brother inevitable? I don't think so. But it's going to be a bumpy ride.