Big Data Blog

Big Data Blog

Big Data, Predictive Analytics, and the Ideal Chronicler

Nov 26

Written by:
11/26/2012 1:41 PM  RssIcon

I recently finished reading the book Everything Is Obvious: How Common Sense Fails Us by Duncan Watts, where I learned about the Ideal Chronicler, a hypothetical being conceived of by the philosopher Arthur Danto.  Since the beginning of time, the Ideal Chronicler has observed every single person, object, action, thought, and intention, and has the power to synthesize all of that historical information with real-time data and make predictions about what might happen next. 

Sounds a lot like big data and predictive analytics, doesn’t it?

Although the Ideal Chronicler knows everything that is happening now, as well as everything that has led up to now, and can make inferences about how all the events it knows about might fit together, it can’t foresee the future in the sense of understanding the significance of what is happening now — because this requires the perspectives and hindsight from further into the future.

For example, when Isaac Newton published his masterpiece Philosophiæ Naturalis Principia Mathematica in 1687, the Ideal Chronicler might have been able to say it was a major contribution to celestial mechanics, and even predict that it would revolutionize science.  But the Ideal Chronicler could not claim that Newton was laying the foundation for what eventually became modern science, or was playing a key role in what eventually was referred to as the Enlightenment.

Danto’s point was that historical explanations do not reproduce the events of the past, but instead explain why they mattered.  However, the only way to know what mattered, and why, is to be able to see what happened as a result — information that, by definition, not even the impossibly talented Ideal Chronicler possesses.

Although I am an advocate for the potential of big data and predictive analytics, some discussions about the technology and techniques behind it over-hype the predictive power of real-time analytics.  Just because we can analyze massive volumes of historical data and synthesize it with fast moving real-time data in a variety of formats from a multitude of sources, doesn’t change the fact that it often takes more time to determine what the consequences are of what just happened.

“Choices that seem insignificant at the time we make them,” Watts explained, “may one day turn out to be of immense import.  And choices that seem incredibly important to us now may later seem to have been of little consequence.  We just won’t know until we know.  In much of life, the very notion of a well-defined outcome at which point we can evaluate, once and for all, the consequences of an action is a convenient fiction.  In reality, the events that we label as outcomes are never really endpoints.  Instead, they are artificially imposed milestones, just as the ending of a movie is really an artificial end to what in reality would be an ongoing story.  Something always happens afterward, and what happens afterward is liable to change our perception of the current outcome, as well as our perception of the outcomes that we have already explained.  It’s actually quite remarkable in a way that we are able to completely rewrite our previous explanations without experiencing any discomfort about the one we are currently articulating.”

Even when predictive analytics enables real-time business decisions that produce a near-term positive result, for example triggering the purchase of shares of a company just before its stock price soars, that outcome is not an endpoint.  Over time, the investment could produce a long-term negative result if the company’s stock tanks, at which point, in hindsight, the original decision to invest would seem like an obvious mistake — even though it didn’t seem that way at the time.

Like the Ideal Chronicler, big data and predictive analytics can help us create predictions about the future that are based on data-driven facts, not intuition-driven fictions.  However, even the best predictions of data science are a convenient fiction.  Until the future becomes history, we will not know if our predictions were true, and more troubling, we may not remember how confident we were in those predictions that time proved false.

30 comment(s) so far...


official source

Its like you read my mind! You seem to know so much about this, like you wrote the book in it or something. I think that you could do with some pics to drive the message home a little bit, but other than that, this is wonderful blog. A fantastic read. I'll certainly be back.
# official source

By TrackBack on   4/26/2014 10:47 AM

click this site

You actually make it seem so easy with your presentation but I find this topic to be really something which I think I would never understand. It seems too complex and very broad for me. I'm looking forward for your next post, I will try to get the hang of it!
# click this site

By TrackBack on   4/27/2014 12:28 PM

discover this

It is the best time to make some plans for the future and it's time to be happy. I have read this post and if I could I want to suggest you some interesting things or suggestions. Perhaps you can write next articles referring to this article. I desire to read more things about it!
# discover this

By TrackBack on   5/2/2014 10:42 AM

find out this here

I loved as much as you will receive carried out right here. The sketch is attractive, your authored material stylish. nonetheless, you command get bought an impatience over that you wish be delivering the following. unwell unquestionably come further formerly again since exactly the same nearly a lot often inside case you shield this increase.
# find out this here

By TrackBack on   5/2/2014 1:12 PM

blog here

Attractive section of content. I just stumbled upon your weblog and in accession capital to assert that I get in fact enjoyed account your blog posts. Any way I will be subscribing to your augment and even I achievement you access consistently quickly.
# blog here

By TrackBack on   5/5/2014 9:38 AM

green coffee bean extract side effects

Unquestionably believe that which you said. Your favorite justification seemed to be on the web the simplest thing to be aware of. I say to you, I certainly get irked while people think about worries that they just don't know about. You managed to hit the nail upon the top and defined out the whole thing without having side effect , people can take a signal. Will probably be back to get more. Thanks
# green coffee bean extract side effects

By TrackBack on   5/5/2014 1:09 PM

green coffee bean extract side effects

This is very interesting, You are a very skilled blogger. I've joined your rss feed and look forward to seeking more of your wonderful post. Also, I have shared your web site in my social networks!
# green coffee bean extract side effects

By TrackBack on   5/5/2014 2:54 PM

green coffee bean extract

Hello there, just became alert to your blog through Google, and found that it is truly informative. I’m going to watch out for brussels. I will be grateful if you continue this in future. A lot of people will be benefited from your writing. Cheers!
# green coffee bean extract

By TrackBack on   5/6/2014 10:27 AM

pure green coffee

I’m not sure where you are getting your info, but good topic. I needs to spend some time learning more or understanding more. Thanks for wonderful information I was looking for this info for my mission.
# pure green coffee

By TrackBack on   5/7/2014 10:36 AM

green coffee bean

Magnificent goods from you, man. I've understand your stuff previous to and you're just extremely fantastic. I actually like what you've acquired here, really like what you're stating and the way in which you say it. You make it entertaining and you still care for to keep it sensible. I can not wait to read much more from you. This is really a terrific site.
# green coffee bean

By TrackBack on   5/7/2014 2:06 PM

find out here

I think this is one of the most significant information for me. And i'm glad reading your article. But should remark on few general things, The site style is ideal, the articles is really excellent : D. Good job, cheers
# find out here

By TrackBack on   5/7/2014 3:00 PM

pure green coffee

I was suggested this blog by my cousin. I am not sure whether this post is written by him as nobody else know such detailed about my problem. You are incredible! Thanks!
# pure green coffee

By TrackBack on   5/8/2014 1:24 PM

home

Hello There. I found your blog using msn. This is an extremely well written article. I’ll be sure to bookmark it and come back to read more of your useful information. Thanks for the post. I will certainly comeback.
# home

By TrackBack on   5/10/2014 4:00 PM

look these up

Attractive section of content. I just stumbled upon your site and in accession capital to assert that I acquire in fact enjoyed account your blog posts. Any way I’ll be subscribing to your feeds and even I achievement you access consistently quickly.
# look these up

By TrackBack on   5/12/2014 3:40 PM

her latest blog

I am really impressed with your writing skills and also with the layout on your blog. Is this a paid theme or did you customize it yourself? Either way keep up the nice quality writing, it is rare to see a great blog like this one today..
# her latest blog

By TrackBack on   5/12/2014 3:50 PM

Click Here

Very nice post. I just stumbled upon your weblog and wanted to say that I've truly enjoyed surfing around your blog posts. After all I will be subscribing to your rss feed and I hope you write again soon!
# Click Here

By TrackBack on   5/12/2014 5:54 PM

designer bags replica

Good transaction, item as described and fast ship.
# designer bags replica

By TrackBack on   5/14/2014 7:24 PM

pop over to this website

I've been surfing online more than three hours today, yet I never found any interesting article like yours. It is pretty worth enough for me. Personally, if all site owners and bloggers made good content as you did, the internet will be a lot more useful than ever before.
# pop over to this website

By TrackBack on   5/15/2014 12:09 PM

click resources

This is really interesting, You're a very skilled blogger. I've joined your feed and look forward to seeking more of your great post. Also, I've shared your site in my social networks!
# click resources

By TrackBack on   5/20/2014 5:30 PM

review

You really make it seem so easy with your presentation but I find this topic to be actually something which I think I would never understand. It seems too complex and very broad for me. I'm looking forward for your next post, I will try to get the hang of it!
# review

By TrackBack on   5/20/2014 5:39 PM

my response

hey there and thank you for your info – I have certainly picked up something new from right here. I did however expertise some technical points using this site, since I experienced to reload the website a lot of times previous to I could get it to load properly. I had been wondering if your web host is OK? Not that I'm complaining, but slow loading instances times will very frequently affect your placement in google and can damage your high quality score if advertising and marketing with Adwords. Anyway I’m adding this RSS to my e-mail and can look out for a lot more of your respective exciting content. Ensure that you update this again very soon..
# my response

By TrackBack on   5/20/2014 8:07 PM

Going Here

Wow, awesome blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your site is excellent, as well as the content!
# Going Here

By TrackBack on   5/23/2014 12:11 PM

check my site

Heya i am for the first time here. I found this board and I find It truly useful & it helped me out much. I hope to give something back and aid others like you helped me.
# check my site

By TrackBack on   5/23/2014 12:29 PM

Recommended Site

It is perfect time to make some plans for the future and it is time to be happy. I have read this post and if I could I want to suggest you some interesting things or advice. Maybe you can write next articles referring to this article. I wish to read even more things about it!
# Recommended Site

By TrackBack on   5/23/2014 12:36 PM

find more

I think this is among the most vital info for me. And i am glad reading your article. But wanna remark on some general things, The web site style is perfect, the articles is really nice : D. Good job, cheers
# find more

By TrackBack on   5/25/2014 1:22 PM

find here

I do not even know how I ended up here, but I thought this post was good. I don't know who you are but definitely you're going to a famous blogger if you aren't already ;) Cheers!
# find here

By TrackBack on   5/25/2014 2:05 PM

Read Full Report

Unquestionably believe that which you said. Your favorite justification seemed to be on the web the easiest thing to be aware of. I say to you, I certainly get annoyed while people consider worries that they just do not know about. You managed to hit the nail upon the top and defined out the whole thing without having side effect , people can take a signal. Will likely be back to get more. Thanks
# Read Full Report

By TrackBack on   5/25/2014 2:37 PM

top article

Hello There. I found your blog using msn. This is an extremely well written article. I’ll make sure to bookmark it and come back to read more of your useful info. Thanks for the post. I will certainly comeback.
# top article

By TrackBack on   5/25/2014 4:49 PM

find here

I was recommended this blog by my cousin. I'm not sure whether this post is written by him as nobody else know such detailed about my difficulty. You are incredible! Thanks!
# find here

By TrackBack on   5/25/2014 5:32 PM

your gay for negative seoing poeple

We have the best porn site aournd on the internet....
# your gay for negative seoing poeple

By TrackBack on   5/27/2014 7:06 PM

Search Big Data Blogs

Tags

Big Data (126)
Analytics (66)
Pervasive (50)
DataRush (33)
Hadoop (31)
Industry trends (22)
predictive analytics (20)
Scalability (20)
Multicore (15)
Data Mining (12)
Parallelism (10)
Java (9)
Jim Harris (9)
KNIME (9)
Cloud (8)
Cyber Security (8)
MapReduce (8)
big data analytics (7)
Data Volumes (7)
Data Warehouse (7)
RushAnalytics (7)
Volumes (7)
Actian (6)
Algorithms (6)
Cost-effective (6)
David Loshin (6)
Decision Support (6)
Julie Hunt (6)
RushAnalyzer (6)
analytics tools (5)
Dataflow (5)
machine learning (5)
Data Science (4)
Forrester (4)
Google (4)
Green IT (4)
Healthcare (4)
Phil Simon (4)
YARN (4)
analytics processes (3)
Big Data Science (3)
BigQuery (3)
Bloor (3)
data centers (3)
data integration (3)
Data Preparation (3)
data tools (3)
data-driven (3)
DataMatcher (3)
machine generated data (3)
Malstone B (3)
Mike Hoskins (3)
Opera Solutions (3)
Retail Analytics (3)
Security (3)
Smart Grid (3)
software (3)
Solutions (3)
telecommunications (3)
transportation analytics (3)
Age of Data (2)
analytics accuracy (2)
architecture (2)
Austin (2)
Bloor Research (2)
Business Intelligence (2)
data management (2)
Data Rush (2)
David Inbar (2)
David Norris (2)
fraud (2)
fraud detection (2)
Gartner (2)
GigaOM (2)
Hadoop Summit (2)
IntegrationWorld (2)
intelligent transportation systems (2)
internet of things (2)
McKinsey (2)
meetup (2)
ParAccel (2)
Pervasive DataRush (2)
Rexer Analytics (2)
smart meters (2)
#FollowFriday (1)
a (1)
Amazon (1)
analytics workflow (1)
Application Development (1)
automation (1)
Benchmarks (1)
best practices (1)
Cloud Analytics Summit (1)
cloud computing (1)
Cloudera (1)
contests (1)
cost (1)
cyber security issues (1)
data flow architecture (1)
Data Integrator - Hadoop Edition (1)
data quality (1)
data visualization (1)
digital marketing (1)
Door64 (1)
easy big data analytics (1)
Ericson (1)
Esri (1)
Facebook (1)
Fuzzy Matching (1)
Goverment (1)
Hadoop User Group (1)
Hadoop World (1)
hardware (1)
HBase (1)
HDFS (1)
industrial internet (1)
Jazoon (1)
Jim Falgout (1)
MalStoneB (1)
Mansour Raad (1)
Neil Raden (1)
Netflix (1)
NetFlow (1)
operational intelligence (1)
Paige Roberts (1)
para (1)
PIG (1)
pilot projects (1)
Predictive Analytics World (1)
psychohistory (1)
Public Sector (1)
Redshift (1)
Robin Bloor (1)
ROI (1)
Rosaria Silipo (1)
RushAccelerator (1)
RushLoader (1)
Sampling (1)
Signal and Noise (1)
SmartDataCollective (1)
spatial analytics (1)
speed (1)
sports (1)
Stephen Swoyer (1)
Steve Shine (1)
Strata (1)
SXSW (1)
Telecom Analytics (1)
Telecommunications Industry Association (1)
TIA (1)
Transportation (1)
TurboRush (1)
VectorWise (1)
Zementis (1)

Latest Posts

Actian Big Data & Analytics Blog has MOVED!
Big Data Phrenology
Big Data, Simpson's Paradox and Sufficient Tools
Data Science and the Art of Data Visualization

Big Data Blog Archives

Archive
<July 2014>
SunMonTueWedThuFriSat
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789
Monthly
Go

Accelerating Big Data 2.0™