Big Data Blog

Big Data Blog

Data-Driven Intuition

Nov 12

Written by:
11/12/2012 6:13 AM  RssIcon

In my blog post Darth Vader, Big Data, and Predictive Analytics, I explained that analytics is not a forced choice between human cognition and computer automation.  Instead, humans, with their intuition-driven tacit understanding, and computers, with their data-driven statistical models, must work together to bring balance to the Analytical Force.

I recently finished reading the book The House Advantage: Playing the Odds to Win Big In Business by Jeffrey Ma.  I highly recommend the book, especially to anyone working in the data management and business intelligence fields where we often oversimplify the business decision-making process by saying it’s either data-driven or intuition-driven — and strongly emphasizing that using data is always better than using intuition.

Although Ma is definitely an advocate for data-driven decision making, toward the end of his book he also acknowledges that there are times when somewhat of a middle ground between data and intuition is called for.

“In the real world, the number of variables are often too numerous and the sample size too small to create perfect models.  But that doesn’t mean you should ignore the data.  It just means that you always strive to find the best data — the most relevant data with which to make the best decisions.  I call this data-driven intuition, and it is a process that many successful people implement when faced with a situation where using a purely data-driven statistical model is impossible.”

The definition of intuition is “a direct perception of truth or fact independent of any reasoning process,” i.e., what is often referred to as “going with your gut.”  In his quest to find a successful individual who truly used their gut and nothing else to make decisions, Ma discovered he needed to redefine intuition as “a direct perception of truth or fact independent of any documented reasoning process” because “no one successful truly makes decisions without some reasoning process.  They may not want to spend the time to explain that reasoning process, or they may not have the sufficient information to document that reasoning process, but it’s clear in all cases I explored that there was a method behind the madness.”

When business leaders talk about relying on their intuition, which is based on their considerable personal experience and professional expertise that has guided their business success to date, most data management and business intelligence professionals lament that the organization’s leadership doesn’t embrace a culture of data-driven decision making.

However, not all data is stored in a database — or even a NoSQL data store.  Some of the most valuable (and most unusually structured) data is stored within the human mind.

The fact is, as Ma explained, “every good decision has some data behind it, as well as a thorough examination of the specific case at hand.  It might not be data that sits in a spreadsheet or an analysis performed by a computer, but it is science much more than art.  Helping people understand what they are doing when they make what they think are intuitive decisions will help them appreciate the true value of using analytics.”

Perhaps human beings have always been data scientists.  Perhaps our intuition has always been more data-driven than we gave it credit for.  Maybe data-driven intuition can bridge the data-intuition divide in analytics.  Or maybe data-driven intuition can remind us that analytics has never been about data or intuition.  Analytics has always been about making better decisions — any way that we can.

Search Big Data Blogs


Big Data (126)
Analytics (66)
Pervasive (50)
DataRush (33)
Hadoop (31)
Industry trends (22)
predictive analytics (20)
Scalability (20)
Multicore (15)
Data Mining (12)
Parallelism (10)
Java (9)
Jim Harris (9)
Cloud (8)
Cyber Security (8)
MapReduce (8)
big data analytics (7)
Data Volumes (7)
Data Warehouse (7)
RushAnalytics (7)
Volumes (7)
Actian (6)
Algorithms (6)
Cost-effective (6)
David Loshin (6)
Decision Support (6)
Julie Hunt (6)
RushAnalyzer (6)
analytics tools (5)
Dataflow (5)
machine learning (5)
Data Science (4)
Forrester (4)
Google (4)
Green IT (4)
Healthcare (4)
Phil Simon (4)
YARN (4)
analytics processes (3)
Big Data Science (3)
BigQuery (3)
Bloor (3)
data centers (3)
data integration (3)
Data Preparation (3)
data tools (3)
data-driven (3)
DataMatcher (3)
machine generated data (3)
Malstone B (3)
Mike Hoskins (3)
Opera Solutions (3)
Retail Analytics (3)
Security (3)
Smart Grid (3)
software (3)
Solutions (3)
telecommunications (3)
transportation analytics (3)
Age of Data (2)
analytics accuracy (2)
architecture (2)
Austin (2)
Bloor Research (2)
Business Intelligence (2)
data management (2)
Data Rush (2)
David Inbar (2)
David Norris (2)
fraud (2)
fraud detection (2)
Gartner (2)
GigaOM (2)
Hadoop Summit (2)
IntegrationWorld (2)
intelligent transportation systems (2)
internet of things (2)
McKinsey (2)
meetup (2)
ParAccel (2)
Pervasive DataRush (2)
Rexer Analytics (2)
smart meters (2)
#FollowFriday (1)
a (1)
Amazon (1)
analytics workflow (1)
Application Development (1)
automation (1)
Benchmarks (1)
best practices (1)
Cloud Analytics Summit (1)
cloud computing (1)
Cloudera (1)
contests (1)
cost (1)
cyber security issues (1)
data flow architecture (1)
Data Integrator - Hadoop Edition (1)
data quality (1)
data visualization (1)
digital marketing (1)
Door64 (1)
easy big data analytics (1)
Ericson (1)
Esri (1)
Facebook (1)
Fuzzy Matching (1)
Goverment (1)
Hadoop User Group (1)
Hadoop World (1)
hardware (1)
HBase (1)
HDFS (1)
industrial internet (1)
Jazoon (1)
Jim Falgout (1)
MalStoneB (1)
Mansour Raad (1)
Neil Raden (1)
Netflix (1)
NetFlow (1)
operational intelligence (1)
Paige Roberts (1)
para (1)
PIG (1)
pilot projects (1)
Predictive Analytics World (1)
psychohistory (1)
Public Sector (1)
Redshift (1)
Robin Bloor (1)
ROI (1)
Rosaria Silipo (1)
RushAccelerator (1)
RushLoader (1)
Sampling (1)
Signal and Noise (1)
SmartDataCollective (1)
spatial analytics (1)
speed (1)
sports (1)
Stephen Swoyer (1)
Steve Shine (1)
Strata (1)
SXSW (1)
Telecom Analytics (1)
Telecommunications Industry Association (1)
TIA (1)
Transportation (1)
TurboRush (1)
VectorWise (1)
Zementis (1)

Latest Posts

Actian Big Data & Analytics Blog has MOVED!
Big Data Phrenology
Big Data, Simpson's Paradox and Sufficient Tools
Data Science and the Art of Data Visualization

Big Data Blog Archives

<April 2014>

Accelerating Big Data 2.0™