Big Data Blog

Big Data Blog

How Predictable Are You?

Dec 10

Written by:
12/10/2012 1:24 PM  RssIcon

Using the science fiction of Asimov’s Psychohistory, Paige Roberts recently blogged about whether or not data science can accurately predict human behavior using machine learning and predictive analytics.

I commented that Psychohistory is an excellent spin on what Duncan Watts refers to as the central intellectual problem of sociology — the micro-macro problem.  Outcomes that sociologists seek to explain are intrinsically macro in nature, meaning that they involve large numbers of people, while at the same time these outcomes are driven by the micro actions of individuals.  The collective behaviors of massive numbers of humans, as in Asimov’s Galactic Empire, are often easier to understand and make predictions about than are the seemingly all-too-often unpredictable behaviors of individual humans.

However, Roberts posited that machine learning and predictive analytics aren’t trying to figure out the rise and fall of human empires, but instead are trying to predict what Psychohistory thought was unpredictable — the behavior of individual humans.  “Given analysis of the actions of millions of humans under certain circumstances over a period of time,” Roberts blogged, “we can now predict which individuals are about to take a certain action, or which action a particular individual will take under certain conditions.”

I think that the reason many people resist the potential of big data analytics is because they innately resist the idea that the complexity of a person — especially themselves — can be encapsulated into a data-driven algorithm that could not only predict their future behavior, but also be able to make better decisions that would guide them toward more positive outcomes.

This reminds me of the 2006 non-fiction book Stumbling on Happiness by Daniel Gilbert, a Professor of Psychology at Harvard University, which was based on the premise that we imagine the future poorly, in particular when in comes to predicting what will make us happy.  At the beginning of the book, Gilbert tells us that he will reveal a way to allow us to make better decisions in the present to increase the likelihood of our future happiness.  But Gilbert also predicts that we will not like his recommendation, which was to leverage the collective outcomes of the past decisions of large groups of people as a guide to help us predict the outcomes of own decisions.  The reason that most of us reject this premise is because it seems to eliminate our uniqueness as an individual.

As Roberts explained in her post, Psychohistory accurately predicted the course of human history in the fictional Galactic Empire for centuries until a single, remarkable individual with extraordinary gifts altered the course of life for billions.  I think that we defy the psychohistorian using machine learning and predictive analytics because we all want to believe that we are that one-in-a-billion remarkable individual.

Although each one of us is as unique as a snowflake, it takes a large volume of snowflakes to accumulate any significant snowfall.  Which is why many big data analytical applications, such as recommendation engines and sentiment analysis, are more effective when large data volumes describing our individual preferences are aggregated.  Although comments from individuals are available to provide more supporting detail, those wonderfully unique snowflakes also produce a flummoxing flurry of eclectic tastes and idiosyncrasies.

If you have ever compared the feedback from two people providing the same rating, for example a three-out-of-five star review of a movie, you can get easily confused by comparing and contrasting a positive three-star review with a negative three-star review.  Therefore, many people, myself included, are more comfortable with the predictive power of millions of people collectively providing an average rating of four stars, while conveniently ignoring all sub-four-star reviews and not reading any of the comments.  Although I have watched a few four-star movies that I would personally rate with three or two stars, more often than not, my enjoyment of a particular movie was accurately predicted by a data-driven algorithm, such as the one used by Netflix.

Of course, it’s not just that data-driven algorithms can predict what movies we watch, but they can also predict how we work, shop, vote, and love, as Stephen Baker wrote about in his illuminating 2008 non-fiction book The Numerati.

I don’t want to minimize the data privacy concerns it raises, but I think that, more than our privacy, it’s our predictability that is being laid bare by big data.  In many cases, I don’t think it’s the predictive ability of data-driven algorithms that scares us, but instead what truly scares is how predictable we might be, how our unique snowflakes are falling toward a common ground, how much like everyone else we truly are.

The existential question underlying big data analytics is: How predictable are you?

The uncomfortable answer is that you are more predictable than you think.  Big data analytics is simply forcing all of us to acknowledge it, not just collectively, but also individually.

25 comment(s) so far...

vivienne westwood

# vivienne westwood

By TrackBack on   7/5/2014 8:22 AM

buy wow gold

These buy wow gold are fantastic nothing can beat buy wow gold. Any vintage in height are amazing becuase these products maintain little feet designer in winter in the event the temperature is on the disadvantages. Comfortable, plus worthy of the money. I exploit the buy wow gold almost all throughout the winter mainly things induce a feet.
# buy wow gold

By TrackBack on   7/11/2014 10:48 PM

Cheap Beats by dre

The son gave me personally it towards moms evening. He knows im maybe not around price tag then again everything comes from all heart. I cannot think it has the favorite color furthermore in fact suits the wrist. I never ever could find out bracelets to fit headphonesthe smaller wrist. Most happy to get recieved your being a gift.
# Cheap Beats by dre

By TrackBack on   7/15/2014 3:45 AM

Michael Kors handbags outlet

Bought the being a gifts to the mom to she liked information technology. awesome grade great amount plus my mom loved it. Additionally delivered very quickly. When you need a gifts quickly and you such as this compared to here is the one you ought to invest!
# Michael Kors handbags outlet

By TrackBack on   7/15/2014 4:10 AM

Coach Factory Outlet Online

That headset had been while spectacular just as in photo. This came quickly. I would suggest getting a towel more than every one coating while you press that it over w / the steam iron. It doesn't vapor out among merely a steamer. On iron is required. It is very delicate, so if you never trust your self aided by the towel plus vapor iron, then take things to a certified. Gorgeous audio.
# Coach Factory Outlet Online

By TrackBack on   7/15/2014 8:27 AM

Coach Outlet Online

Your headset is as breathtaking while in image. That it arrived immediately. I'd recommend getting a towel through every single level while you push information technology outside w / one vapor iron. It doesn't steam outside through merely a steamer. The actual iron ended up being necessary. It can be sensitive, when you don't trust yourself with the towel then steam iron, subsequently accept that to a expert. Gorgeous appear.
# Coach Outlet Online

By TrackBack on   7/15/2014 3:45 PM

Cheap Beats by dr dre Headphones Solo

Close mother's time gift. Then again, that bracelet got too prolonged on show up, therefore came after the vacation. My mom shown which she loved that bracelet, although We have never viewed her put on this still. :-)
# Cheap Beats by dr dre Headphones Solo

By TrackBack on   7/15/2014 5:57 PM

Cartier Love Ring

I had gotten it as being a gift for the my sibling, satisfied mama. She enjoyed things!! I was nervous things is going to be quite limited due to the earlier analysis and just how information technology seemed once I got your top, but it compliment quite nicely. That it looks pretty on top of as well as in their beautiful package it's provided when.
# Cartier Love Ring

By TrackBack on   7/15/2014 10:31 PM

Nike Air Max 2014

Awesome headphonesinvest, well cost and/or exactly what is shown. Stunning present container and poem furthermore enclosed. Great for the mother's evening!
# Nike Air Max 2014

By TrackBack on   7/16/2014 2:58 AM

Coach Outlet

sThe grade that it ideal, but, considering that dimensions operate mini bring that it one particular shape increase.If you're 5f 4inches then size requirements adjusting otherwise exclusively use truly tall sneakers. Besides that loved some sort of gown
# Coach Outlet

By TrackBack on   7/16/2014 7:42 AM

Michael Kors bags

Yet not because fancy in real life provided mothers wrist is actually not 2small don't invest that it cause things get tight .. the cute then again anything yur 5year older makes . Non the particular reduced the a great ok gift towards offer yet not letter wow gifts...cant go wrong should don't come with much to blow on a present subsequently attain information technology
# Michael Kors bags

By TrackBack on   7/16/2014 10:17 AM

Coach Factory Outlet Online

I purchased that to my mom never expecting a lot since this was quite inexpensive. When I gotten it I became delighted! It is beautiful & doesn't look cheap whatsoever! My personal mom enjoyed things!
# Coach Factory Outlet Online

By TrackBack on   7/16/2014 1:23 PM

Coach Outlet

Yet not just as fancy as part of real world should mothers wrist is not 2small don't choose this trigger that it get tight .. their adorable although one thing yur 5year existing make . No that less it is an ok gifts to present yet not the best wow present...cant go wrong provided dont get a lot to blow on a gifts well take that
# Coach Outlet

By TrackBack on   7/16/2014 4:44 PM

%e0%a6%9c%e0%a6%95%e0%a6%bf%e0%a6%97%e0%a6%9e%e0%a7%8d%e0%a6%9c%e0%a7%87 %e0%a6%b6%e0%a6%bf%e0%a6%ac%e0%a6%bf%e0%a6%b0 %e0%a6%95%e0%a6%b0%e0%a7%8d%e0%a6%ae%e0%a7%80 %e0%a6%b9%e0%a7%81%e0%a6%a6

PESCA: Eine Faustregel 眉ber Bonzen, haben sie nicht zu sagen, sie sind gro脽 Sch眉sse.
# %e0%a6%9c%e0%a6%95%e0%a6%bf%e0%a6%97%e0%a6%9e%e0%a7%8d%e0%a6%9c%e0%a7%87 %e0%a6%b6%e0%a6%bf%e0%a6%ac%e0%a6%bf%e0%a6%b0 %e0%a6%95%e0%a6%b0%e0%a7%8d%e0%a6%ae%e0%a7%80 %e0%a6%b9%e0%a7%81%e0%a6%a6

By TrackBack on   7/18/2014 2:23 AM

mulberry effie

Ms. Acierno was at Popular Club, a division of J. Crew, from 1992 to 1996, first as Divisional Merchandise Manager for Women's Ready to Wear and then Vice President/General Merchandise Manager for Women's. Earlier in her career she was a Senior Buyer at Henri Bendel, a division of The Limited, and Director of Merchandising for Women's, Children's, Bridal and Home at Laura Ashley.
# mulberry effie

By TrackBack on   7/18/2014 3:55 AM

Cheap Nike free run 3

Well mother's time gifts. Although, some sort of bracelet got too longer in order to come, therefore came following the holiday. Our mother expressed just that she liked the bracelet, yet I've not spotted her don it however. :-)
# Cheap Nike free run 3

By TrackBack on   7/18/2014 6:49 AM

Cartier Love Bracelet Replica

Good mother's time gifts. However, on bracelet took quite very long in order to appear, and it came following the getaway. My personal mom shown that will she loved the particular bracelet, even though We have not really noticed her don information technology but. :-)
# Cartier Love Bracelet Replica

By TrackBack on   7/19/2014 1:26 PM

toms outlet

Oh so soft, this baby bunting bags will keep baby warm and comfortable. Made from Cashmere, one of the most luxurious fabrics in the world, with a touch of Ralph Lauren style. $175
# toms outlet

By TrackBack on   7/19/2014 11:44 PM

Recieved the product prior to the time provided in a very amazing gift container . The actual image that is displayed for bracelet was exactly what you will recieve. It is most delicate still per striking part of precious jewelry that will endure as iof some sort of toggle clasp that is most sturdy so the bracelet eill not really be removed.

By TrackBack on   7/20/2014 1:10 AM

hollister online shop

芦Mon exp茅rience d茅passe de loin M. Nowak de," dit-elle. "Le fait que j'ai chang茅 de parti est la seule chose qu'il peut accrocher son chapeau."
# hollister online shop

By TrackBack on   7/20/2014 7:36 AM

Longchamp Le Pliage authentique Arbre de Vie Rouge Sacs

Er wurde auf Kaution freigelassen und befahl Fr盲ulein Halliday nicht, Kontakt oder besuchen sie zu Hause. Sie wird erwartet, dass der einzige Zeuge in seinem Prozess sein.
# Longchamp Le Pliage authentique Arbre de Vie Rouge Sacs

By TrackBack on   7/22/2014 9:05 AM

sThe high quality it very good, still, as that the models operated tiny choose that any size upward.If you are 5f 4inches after that length ought modifying or simply put on completely tall shoes. Besides that liked that outfit

By TrackBack on   7/22/2014 1:14 PM

timberland scarpe da barca

While some of the factory shops have received flack over the years for creating clothing specifically for the discount locations, others are known for their treasures. Avid shoppers can find styles from the current season at up to 80% off retail. Tory Burch: Great accessories, shoes, handbags and more from the current season, at a minimum of 30% off retail
# timberland scarpe da barca

By TrackBack on   7/23/2014 12:29 AM

080913_01 10mu sd

In an interesting twist to the story, the home is being marketed by Pulitzer's children, Brown Harris Stevens' Peter McKim Pulitzer and his sister, Liza Pulitzer. Although they are taking advantage of the home's large property in their selling efforts, Liza Pulitzer told the Observer: "We would hate to see the house torn down, but the house is not landmarked."
# 080913_01 10mu sd

By TrackBack on   7/23/2014 10:16 PM

hollister france soldes

Sean John license, which was announced in May 2004. She reports to John"Desiree comes to us with an impressive array of
# hollister france soldes

By TrackBack on   7/24/2014 1:18 AM

Search Big Data Blogs


Big Data (126)
Analytics (66)
Pervasive (50)
DataRush (33)
Hadoop (31)
Industry trends (22)
predictive analytics (20)
Scalability (20)
Multicore (15)
Data Mining (12)
Parallelism (10)
Java (9)
Jim Harris (9)
Cloud (8)
Cyber Security (8)
MapReduce (8)
big data analytics (7)
Data Volumes (7)
Data Warehouse (7)
RushAnalytics (7)
Volumes (7)
Actian (6)
Algorithms (6)
Cost-effective (6)
David Loshin (6)
Decision Support (6)
Julie Hunt (6)
RushAnalyzer (6)
analytics tools (5)
Dataflow (5)
machine learning (5)
Data Science (4)
Forrester (4)
Google (4)
Green IT (4)
Healthcare (4)
Phil Simon (4)
YARN (4)
analytics processes (3)
Big Data Science (3)
BigQuery (3)
Bloor (3)
data centers (3)
data integration (3)
Data Preparation (3)
data tools (3)
data-driven (3)
DataMatcher (3)
machine generated data (3)
Malstone B (3)
Mike Hoskins (3)
Opera Solutions (3)
Retail Analytics (3)
Security (3)
Smart Grid (3)
software (3)
Solutions (3)
telecommunications (3)
transportation analytics (3)
Age of Data (2)
analytics accuracy (2)
architecture (2)
Austin (2)
Bloor Research (2)
Business Intelligence (2)
data management (2)
Data Rush (2)
David Inbar (2)
David Norris (2)
fraud (2)
fraud detection (2)
Gartner (2)
GigaOM (2)
Hadoop Summit (2)
IntegrationWorld (2)
intelligent transportation systems (2)
internet of things (2)
McKinsey (2)
meetup (2)
ParAccel (2)
Pervasive DataRush (2)
Rexer Analytics (2)
smart meters (2)
#FollowFriday (1)
a (1)
Amazon (1)
analytics workflow (1)
Application Development (1)
automation (1)
Benchmarks (1)
best practices (1)
Cloud Analytics Summit (1)
cloud computing (1)
Cloudera (1)
contests (1)
cost (1)
cyber security issues (1)
data flow architecture (1)
Data Integrator - Hadoop Edition (1)
data quality (1)
data visualization (1)
digital marketing (1)
Door64 (1)
easy big data analytics (1)
Ericson (1)
Esri (1)
Facebook (1)
Fuzzy Matching (1)
Goverment (1)
Hadoop User Group (1)
Hadoop World (1)
hardware (1)
HBase (1)
HDFS (1)
industrial internet (1)
Jazoon (1)
Jim Falgout (1)
MalStoneB (1)
Mansour Raad (1)
Neil Raden (1)
Netflix (1)
NetFlow (1)
operational intelligence (1)
Paige Roberts (1)
para (1)
PIG (1)
pilot projects (1)
Predictive Analytics World (1)
psychohistory (1)
Public Sector (1)
Redshift (1)
Robin Bloor (1)
ROI (1)
Rosaria Silipo (1)
RushAccelerator (1)
RushLoader (1)
Sampling (1)
Signal and Noise (1)
SmartDataCollective (1)
spatial analytics (1)
speed (1)
sports (1)
Stephen Swoyer (1)
Steve Shine (1)
Strata (1)
SXSW (1)
Telecom Analytics (1)
Telecommunications Industry Association (1)
TIA (1)
Transportation (1)
TurboRush (1)
VectorWise (1)
Zementis (1)

Latest Posts

Actian Big Data & Analytics Blog has MOVED!
Big Data Phrenology
Big Data, Simpson's Paradox and Sufficient Tools
Data Science and the Art of Data Visualization

Big Data Blog Archives

<July 2014>

Accelerating Big Data 2.0™