The war has raged for as long as anyone can remember. Through countless cycles of off-shoring, acquisitions, out-sourcing, mergers and technological change; the battle between the standardisers and the disruptors has continued within companies all over the world.
The Standardisers
Brought up on a strict dogma of “don’t repeat yourself (DRY)” and “one version of the truth”, they pride themselves on efficiency and purity. There will only be one way to do anything, and each thing will be done by only one team. The result will be only one set of data regarding what has been done. Audits will be simple, metrics will be easy to calculate, the numbers will always add up and handing change will be — “I’m sorry, handling cha…what?”
The Disruptors
The disruptors, do not directly disagree with the standardisers, after all, the standardisers’ logic is hard to argue with. However, the disruptors were brought up on different doctrines. They prefer to “Keep it Simple, Stupid (KISS)” and “ask for forgiveness, not permission”. They prefer to make money, rather than to save money.
The disruptors want to extend the company’s frontier rather then engage in direct conflict with the standardisers. The standardisers are often to be found chasing disruptors across newly gained territory, hoping one day to catch up. Occasionally though, a band of intrepid disruptors run into a native standardiser settlement. When that happens, things can get messy.
And so the war rumbles on, across open fields of Sales, canyons of new Product Development, oceans of IT and the dark and ominous forests of Human Resources and Legal.
The Business Intelligence Tribe
Traditionally, the Business Intelligence (BI) tribe lean more towards the Standardisers than the Disruptors. They would insist that reliable metrics are only possible once standardisation has been established. Standard processes, lead to standard definitions of terms, and only then can meaningful metrics be produced.
More recently though, some more disruption minded BIs have emerged. They call themselves “Agile” or “Lean” and “If the processes change,” they say, “we’ll just have to deal with that in the reports”.
The Oncoming Storm of Data
The landscape of business does not remain static, however. Forces outside the company do play a part in deciding the outcome of these battles. Government regulation tends to give Standardisers the upper hand. Whereas turbulent economies and new technology usually benefit the Disruptors.
It should be no surprise then, that the oncoming wave of data threatens to upset the balance once more. At first, the web came along, and suddenly you could watch people responding to your ads and your products in almost real time. You could track impressions, clicks and signups. You could measure demographics and buying patterns in a way that TV advertisers and physical store operators could only dream of.
Then mobile happened. Now every one of your customers and employees has location tracking and a video camera with them at all times. Companies with a distributed infrastructure (like a fleet of vehicles or a chain of high street stores) are getting realtime data in from employees and customers on what’s happening, where it is, and what it looks like. If your passenger jet is grounded, somebody somewhere is tweeting a video of an overcrowded airport at you.
But it will not stop there; Next will come the wearable tech wave, which will add physiological data, and even more pervasive photo/video capturing.
Then coming down the line we can see “the Internet of Things”. Meaning almost all electrical devices will be capturing data. Your in-store air-con units, tills, automatic doors, water valves, refrigeration units, and motion sensitive lighting will all be gathering data. Even the damn coffee cups will be beaming data on how caffeinated your customers are back to… well, somewhere.
And the Standardisers are hopelessly unprepared for this onslaught. We will see new forms of data, from new sources, in new formats, and all the while the volumes will increase at an exponential rate. Keeping a centralised data-mart up to date with all this change will become virtually impossible. Like the Soviet economy planners trying to control how much of everything the nation produced, they don’t stand a chance. The result will be a chronic undersupply of meaningful reports, and a thriving black market in questionable metrics. BI analysts only hope is to abandon the idea of a top down, centrally planned, single version of the truth.
But there is a model they can follow: Science.
The scientific community does not try to standardise the world, they simply observe it. And rather than forcing every researcher to work in a particular way, the emphasis is on transparency of process, peer review, and independent verification of results by others. Scientists build their reputation by being very careful about releasing conclusions, and when they do, they usually report a margin of error and a level of confidence in their numbers.
The Data Scientist
And so we are already starting to see a new bread of BI practitioner, the Data Scientist.
In many ways, “data scientists” are not new. They combine the analytical ability of a quant, with the undirected research/discovery process of a scientist, and the technical abilities of a software developer. It’s always been rare to find all three qualities in one person. The fact that Google and Netflix are desperate to hire such people has increased their profile but not their numbers. And, I suspect, tempted a lot of people to claim that they are “data scientists” when really all they’re doing is traditional BI.
But the need for people with coding skills as well as scientific mindset and advanced mathematical abilities is a temporary state of affairs I think. Once big data stores are the norm, technologies will evolve to make them easier to interact with them. No doubt one day I’ll add HBase or Cassandra support to QueryTree, and you won’t need to write code to build complex queries on your tabular data any more.
When that happens, what’s left will be a more conventional scientist type role, but embedded within a corporate enterprise. We’ve already seen this happen, in June 2012 prominent economist Yanis Varoufakis announced that he was joining the video games maker Valve to study the in-game economies that existed in their massively multiplayer online games.
These corporate scientists will study the vast oceans of data within the company’s data stores and engage in research towards a better understanding of customers and systems. Papers will be written, peer reviewed and published internally. With C-suite executives having to take their lead from the editors of internal scientific journals rather than traditional heads of Business Intelligence.
And What of the Standardisers?
Having lost the territory of BI, perhaps for good, I imagine they will retreat to their natural base: IT, Legal and Accounts. Departments that tend to get punished when unexpected things happen and are rarely incentivised to bring about change. There will always be a place for standardisers, but the BI team of the future is not, in my view, one of them.