Darth Vader, Big Data, and Predictive Analytics
Oct
2
Written by:
10/2/2012 6:15 AM
“He’s more machine now than man. His mind is twisted and evil.”
Those were the words of Jedi Master Obi-Wan Kenobi, describing his old friend Anakin Skywalker who, after being seduced by the Dark Side of the Force, became the cyborg Sith Lord Darth Vader.
I sense a similar sentiment in business leaders who resist the machine learning algorithms that are becoming more of a necessity in the era of big data. Some business leaders fear that the future of big data analytics could twist decision making into becoming more machine than man — if more decisions are made by a computer algorithm, and fewer decisions are made by a human mind.
A common challenge for predictive analytics is that algorithmic predictions often seem counterintuitive to business leaders, whose intuition is rightfully based on their business expertise, which has guided their business success to date. Even those who pride themselves on being data-driven in their decision making naturally resist any counterintuitive insights found in their data.
To paraphrase Obi-Wan, your intuition serves you well, but it could also be your undoing.
Instead, some business leaders are afraid that the Dark Side of the Analytical Force is more likely to be their undoing, citing examples such as the 2010 Flash Crash (the complexity of which included more than just algorithms gone wild), and the 2011 Amazon book price algorithm war that temporarily priced a book at well over a million dollars (the algorithmic-driven price topped out at $23,698,655.93 — plus an additional $3.99 for shipping, of course).
However, despite their profound lack of the tacit understanding provided by human cognition, it’s simply undeniable that computer automation has its advantages.
Computers have much greater memory capacity and much faster processing speed. Computers excel at mathematics and statistics (even more so than humans with PhDs in mathematics and statistics). Computers don’t get tired or hungry. Computers don’t get upset when the local sports team loses or get despondent because the cute fax machine that works on the fifth floor won’t answer any of its calls. And computers don’t get too distracted looking for cat videos on YouTube.
Algorithms make consistent decisions in that when given the same inputs, they always return the same answer. Algorithms also more quickly learn when their previous answer was a mistake. Thanks in part to suffering no embarrassment from making mistakes (and admitting that they made mistakes), algorithms can continually learn without ever assuming (or becoming adamant) that they have nothing left to learn because of their vast experience and proven expertise.
But it’s also undeniable that big data analytics is not a forced choice between human cognition and computer automation — instead, it requires a fusion of the best of both.
“One of the reasons that I am a firm believer in business rules and predictive analytics being used in tandem,” James Taylor explained during a podcast discussion about decision management, “is that I find that you’re less likely to go off the rails in either direction.”
”Business rules are very explicit,” Taylor continued. “These are our policies. These are the regulations. This is our expertise. This is what makes sense. Business rules can be a little bit of a blunt instrument at times because they don’t have the nuances you would get out of data analysis, and they tend to rely on what everybody knows, which is not always true. But business rules are explicit, easy to track, and bounded.”
”Predictive analytics, on the other hand, can sometimes produce somewhat surprising results. Although predictive analytics learns quickly as new data comes in, it isn’t bounded quite as much by the sensibilities of the people who run the business. Therefore, business rules and predictive analytics work much better when they work together.”
Remember that Darth Vader did not intentionally became a cyborg (i.e., a fusion of humanity and technology). You needn’t lose your limbs in lightsaber battles with big data, or get burned in a lava flow of information overload, or literally become a cyborg in order to accept that humans and computers must work together to bring balance to the Analytical Force.
May the Analytical Force be with you. Always.