Big Data, Predictive Analytics, and the Ideal Chronicler
Nov
26
Written by:
11/26/2012 1:41 PM
I recently finished reading the book Everything Is Obvious: How Common Sense Fails Us by Duncan Watts, where I learned about the Ideal Chronicler, a hypothetical being conceived of by the philosopher Arthur Danto. Since the beginning of time, the Ideal Chronicler has observed every single person, object, action, thought, and intention, and has the power to synthesize all of that historical information with real-time data and make predictions about what might happen next.
Sounds a lot like big data and predictive analytics, doesn’t it?
Although the Ideal Chronicler knows everything that is happening now, as well as everything that has led up to now, and can make inferences about how all the events it knows about might fit together, it can’t foresee the future in the sense of understanding the significance of what is happening now — because this requires the perspectives and hindsight from further into the future.
For example, when Isaac Newton published his masterpiece Philosophiæ Naturalis Principia Mathematica in 1687, the Ideal Chronicler might have been able to say it was a major contribution to celestial mechanics, and even predict that it would revolutionize science. But the Ideal Chronicler could not claim that Newton was laying the foundation for what eventually became modern science, or was playing a key role in what eventually was referred to as the Enlightenment.
Danto’s point was that historical explanations do not reproduce the events of the past, but instead explain why they mattered. However, the only way to know what mattered, and why, is to be able to see what happened as a result — information that, by definition, not even the impossibly talented Ideal Chronicler possesses.
Although I am an advocate for the potential of big data and predictive analytics, some discussions about the technology and techniques behind it over-hype the predictive power of real-time analytics. Just because we can analyze massive volumes of historical data and synthesize it with fast moving real-time data in a variety of formats from a multitude of sources, doesn’t change the fact that it often takes more time to determine what the consequences are of what just happened.
“Choices that seem insignificant at the time we make them,” Watts explained, “may one day turn out to be of immense import. And choices that seem incredibly important to us now may later seem to have been of little consequence. We just won’t know until we know. In much of life, the very notion of a well-defined outcome at which point we can evaluate, once and for all, the consequences of an action is a convenient fiction. In reality, the events that we label as outcomes are never really endpoints. Instead, they are artificially imposed milestones, just as the ending of a movie is really an artificial end to what in reality would be an ongoing story. Something always happens afterward, and what happens afterward is liable to change our perception of the current outcome, as well as our perception of the outcomes that we have already explained. It’s actually quite remarkable in a way that we are able to completely rewrite our previous explanations without experiencing any discomfort about the one we are currently articulating.”
Even when predictive analytics enables real-time business decisions that produce a near-term positive result, for example triggering the purchase of shares of a company just before its stock price soars, that outcome is not an endpoint. Over time, the investment could produce a long-term negative result if the company’s stock tanks, at which point, in hindsight, the original decision to invest would seem like an obvious mistake — even though it didn’t seem that way at the time.
Like the Ideal Chronicler, big data and predictive analytics can help us create predictions about the future that are based on data-driven facts, not intuition-driven fictions. However, even the best predictions of data science are a convenient fiction. Until the future becomes history, we will not know if our predictions were true, and more troubling, we may not remember how confident we were in those predictions that time proved false.
7 comment(s) so far...
Re: Big Data, Predictive Analytics, and the Ideal Chronicler
Hey Jim,
I think you found a big hole in the predictive analytics as psychohistory concept.
Unless, given enough data, the ideal chronicler, or the psychohistorian in this case, could predict the larger impact of events and therefore, their effects on later events. That would give the psychohistorian the ability to know ahead of time what would be important and why.
Paige
By Paige Roberts on
11/26/2012 2:25 PM
|
Re: Big Data, Predictive Analytics, and the Ideal Chronicler
Thanks for your comment, Paige.
I think that Asimov's Psychohistory is an excellent spin on what Watts referred to as the central intellectual problem of sociology — the micro-macro problem.
Outcomes that sociologists seek to explain are intrinsically “macro” in nature, meaning that they involve large numbers of people, while at the same time these outcomes are driven by the “micro” actions of individuals. The collective behaviors of massive numbers of humans, as in Asimov's Galactic Empire, are always far easier to understand and make predictions about than are the all-too-often unpredictable behaviors of individual humans.
And this is why many big data analytical applications, such as recommendation engines and sentiment analysis, are more effective when large volumes of individual preferences are aggregated. Although comments from individuals are also available to provide more supporting detail, if you have ever compared the feedback from two people providing the same rating, for example a three-out-of-five star review of a movie, you can get easily flummoxed by comparing and contrasting a "positive" three star review with a "negative" three star review. Therefore, most people, myself include, are more comfortable with the predictive power of millions of people saying four or five stars, while conveniently ignoring all sub-four-star reviews and not reading any of the comments.
Danto's Ideal Chronicler also tried to minimize the unpredictability of the future by aggregating the individual into the whole of society as well as the vastness of centuries of recorded history.
But when predictions about the future are made, a troubling threshold is crossed when the present arrives and the prediction is initially proven true. Although at this point, we would like to relegate the proven prediction to the past and immediately include it in the telling of history, as Watts explained, “history cannot be told while it is happening because what is happening can't be made sense of until its implications have been resolved” and we have no way of knowing how long that will take.
After all, I know that I have watched more than a few movies that received four or five stars only to be so disappointed in how the movie ended that I gave it three or two stars. At least with a movie, the running time lets me know exactly how long I will need to wait before proving the prediction is true — and even then, it remains largely only true or false from my individual perspective :-)
Best Regards,
Jim
By Jim Harris on
11/26/2012 5:58 PM
|
Re: Big Data, Predictive Analytics, and the Ideal Chronicler
Perhaps the issue is that for years, "big data" holds this glorious promise of being predictive, when in fact what it does is allow us to better understand the past, or more quickly bring about the time at which we can accurately analyze an event as history. In truth, what "big data" allows us to do is not predict the future, but rather more accurately ensure we do not fail the same way twice, providing a better chance that we could succeed at future endeavors. Through this perspective, we can increase the number of Chroniclers within an organization with an objective to not necessarily predict a final outcome, but rather to not fail the same way twice because the factor you overlooked will not be overlooked again. So we will not be correct all the time, my goal is to be correct most of the time.
My thinking is that if Ideal Chronicler can make a prediction about an event, there are only two plausible outcomes - they are right, or they are not right. If I were to walk up to my cat and pull her tail, I predict that she will get angry with me. I do not know for sure, but all of my previous experience tells me that I will be correct most of the time. To be successful in the age of realtime Analytics, I only need to be right most of the time, which I can be if I have a good understanding of how a system typically works. And while the Ideal Chronicler may not always be right, they are more likely to be correct more often than I who does not have the same amount of prior perspective as (s)he.
By Will on
11/26/2012 9:07 PM
|
Re: Proceeding comments on Big Data, Predictive Analytics etc.
First, for Jim Harris: You seem to have tapped into the big data Zeitgeist of the moment. And I do mean Zeitgeist (not existential crisis). Last week I read, for the first time, both the Wikipedia article about psychohistory (which remains umm, "localized" as a field of study) and Isaac Asimov's fictional take on psychohistory. From the latter, I arrived at similar conclusions as those that you mentioned.
Next, I've been reading various U.S. Presidential election forecasting post-mortem's, e.g. Gallup v. Nate Silver. A frequent theme was whether statistical predictive methods used by the "quant's" were as remarkably successful as traditional political analysts e.g. George Wills, were so off the mark. The latter was attributed to GOP-inclined analysts' unwillingness to acknowledge even marginally quantitative indicators that were suggestive of a less than favorable outlook for the GOP, rather than lack of analytic ability. I mention that from a completely non-partisan point of view; only that some very fine thinkers such as George Wills are perceiving events through a biased filter, to detrimental effect.
As for Big Data's power, it is likely to be useful for more granular consumer targeting, for marketing purposes, and less prosaic purposes too. That isn't what you are discussing here though. There is a company called "Recorded Future" (I am not affiliated with them in any way whatsoever) that seeks to predict events in the near-future, especially geo-political situations with likely and well-known influences, e.g. oil prices. They use Big Data for their work. Yet Recorded Future also consults traditional subject matter experts who have deep knowledge of regional politics, history, geography, resources and cultural drivers. They don't hide this. The results are presented as interactive visualizations and charts; casual perusal might suggest that there is solely predictive analysis at work.
For Will: With a combination such as that used by Recorded Future (and others, although all guard their methodology and means of integration), I think it should be possible to at least realize the goal you mentioned it in the comment prior to mine, of
"not necessarily predicting a final outcome, but rather to not fail the same way twice because the factor you overlooked will not be overlooked again".
That goal seems so modest, unambitious, at first glance, compared to the predictive wonders suggested by Big Data. In fact, it would be a remarkable and worthwhile achievement, if it were possible on a basis frequent enough to be measurable using a hypothesis test, or other measure of statistical significance.
Thank you, Jim, for this post. I truly enjoyed reading it.
By Ellie Kesselman on
11/28/2012 12:18 AM
|
http://www.lesliehindman.com/index.php?/member/338971/
Blog | Pervasive Big Data - Big Data, Predictive Analytics, and the Ideal Chronicler
# http://www.lesliehindman.com/index.php?/member/338971/
By TrackBack on
11/28/2012 5:09 AM
|
Re: Big Data, Predictive Analytics, and the Ideal Chronicler
Thanks for your comments, Will and Ellie.
@Will — Yes, ensuring that we do not fail the same way twice is why we need to analyze historical data, since as George Santayana cautioned “those who cannot remember the past are condemned to repeat it.” Although the future is not always like the past, incorporating historical analysis into our decision making can allow us, as you said, to be correct more often than we would be without this perspective.
@Ellie — Yes, perceptual filters can have a very detrimental effect on our analysis regardless of the techniques we employ, since even solid statistical models can produce flawed predictions when the data feed into them has been selected based on our confirmation bias. Even though I bought it before the election, I just started reading Nate Silver's book "The Signal and the Noise: Why Most Predictions Fail but Some Don't" and I will no doubt soon join the many bloggers who post something about experts versus algorithms, but I definitely agree with you about the value of their synthesis, which I previously blogged about on this site in my post "Darth Vader, Big Data, and Predictive Analytics."
Best Regards,
Jim
By Jim Harris on
11/28/2012 3:20 PM
|
marketing
Blog | Pervasive Big Data
# marketing
By TrackBack on
12/29/2012 11:32 PM
|