Big Data Blog

Big Data Blog

Data-Driven Intuition

Nov 12

Written by:
11/12/2012 6:13 AM  RssIcon

In my blog post Darth Vader, Big Data, and Predictive Analytics, I explained that analytics is not a forced choice between human cognition and computer automation.  Instead, humans, with their intuition-driven tacit understanding, and computers, with their data-driven statistical models, must work together to bring balance to the Analytical Force.

I recently finished reading the book The House Advantage: Playing the Odds to Win Big In Business by Jeffrey Ma.  I highly recommend the book, especially to anyone working in the data management and business intelligence fields where we often oversimplify the business decision-making process by saying it’s either data-driven or intuition-driven — and strongly emphasizing that using data is always better than using intuition.

Although Ma is definitely an advocate for data-driven decision making, toward the end of his book he also acknowledges that there are times when somewhat of a middle ground between data and intuition is called for.

“In the real world, the number of variables are often too numerous and the sample size too small to create perfect models.  But that doesn’t mean you should ignore the data.  It just means that you always strive to find the best data — the most relevant data with which to make the best decisions.  I call this data-driven intuition, and it is a process that many successful people implement when faced with a situation where using a purely data-driven statistical model is impossible.”

The definition of intuition is “a direct perception of truth or fact independent of any reasoning process,” i.e., what is often referred to as “going with your gut.”  In his quest to find a successful individual who truly used their gut and nothing else to make decisions, Ma discovered he needed to redefine intuition as “a direct perception of truth or fact independent of any documented reasoning process” because “no one successful truly makes decisions without some reasoning process.  They may not want to spend the time to explain that reasoning process, or they may not have the sufficient information to document that reasoning process, but it’s clear in all cases I explored that there was a method behind the madness.”

When business leaders talk about relying on their intuition, which is based on their considerable personal experience and professional expertise that has guided their business success to date, most data management and business intelligence professionals lament that the organization’s leadership doesn’t embrace a culture of data-driven decision making.

However, not all data is stored in a database — or even a NoSQL data store.  Some of the most valuable (and most unusually structured) data is stored within the human mind.

The fact is, as Ma explained, “every good decision has some data behind it, as well as a thorough examination of the specific case at hand.  It might not be data that sits in a spreadsheet or an analysis performed by a computer, but it is science much more than art.  Helping people understand what they are doing when they make what they think are intuitive decisions will help them appreciate the true value of using analytics.”

Perhaps human beings have always been data scientists.  Perhaps our intuition has always been more data-driven than we gave it credit for.  Maybe data-driven intuition can bridge the data-intuition divide in analytics.  Or maybe data-driven intuition can remind us that analytics has never been about data or intuition.  Analytics has always been about making better decisions — any way that we can.

15 comment(s) so far...

jordan sneaker

Chinese cuisine reflects the cultural traditions of the Chinese nation's diet , it is compared with the world of cooking , there are many unique features.
# jordan sneaker

By TrackBack on   5/13/2014 3:56 AM

cheap jordans

Originally regarded a hard-to-get limited release with availability restricted to prominent mom-and-pops and boutiques in five boroughs, Air Force 1 Mid Shoes is clearly aiming for a bigger audience without the pain . online release as well as the house of Hoops amount. In addition to HoH, Foot Locker on 34th St. in Midtown, NYC will have the Manhattan colorway available too.
# cheap jordans

By TrackBack on   6/6/2014 3:25 AM

cheap air jordans

6, if the Spurs win - Tim Duncan outstanding performance, he will have the opportunity to become the NBA history except Michael - the only one outside of Jordan (air jordan shoes the Finals MVP four players ; And if the Heat win , LeBron - James wants to be there after Jordan ( 1991-93,1996-98 ) and Shaquille O'Neal ( 2000-02 ) was elected for three years following the third consecutive NBA Finals MVP player.
# cheap air jordans

By TrackBack on   6/6/2014 4:50 AM

air jordan 2011

" This season the team played well hell , I'm proud of them, we won 56 regular-season victories , won two Eastern Conference finals victory , and have the opportunity further ." This is a walk after the game opening remarks by coach Vogel , speech can hear a bit of bitterness and disappointment . "3 consecutive years lost to the same opponent , of course we very frustrating , but you know with us is against this era - Michael Jordan and the Chicago Bulls , and you would like them to have to take my hat off ."
# air jordan 2011

By TrackBack on   6/6/2014 12:13 PM

cheap jordans online

Fragment Design x Nike KD VI Exposure new light brown color, showing a large area of ??brown suede shoes body, with the whole palm Zoom Max midsole. Eye-catching fluorescent yellow embellishment on the lining, painting and other details in the end, the overall color light. It is worth mentioning that the well-known street artists Delano Brown also in this dual Fragment Design x Nike KD VI is modeled to create a pair of personal style graffiti full version. A large area filled with floral patterns with the texture of the suede shoes are so Fragment Design x Nike KD VI brought out a new look.
# cheap jordans online

By TrackBack on   6/19/2014 10:00 PM

buy wow gold

These are generally a 2nd number of buy wow gold and also at to begin with I truly reverred these individuals. Nevertheless after making these products 2 conditions, many people started come to feel distinctive as well as acquire varied things in the material belonging to the buy wow gold. It truly is out of the question looking to get that harmful fur down and today they are similar to giant blobs on my little ft when I create him or her. That is not lovable.
# buy wow gold

By TrackBack on   6/20/2014 2:23 AM

Cheap Oakley Sunglasses

If some one desires expert view on the topic of blogging after that i propose him/her to pay a visit this webpage, Keep up the fastidious work.
# Cheap Oakley Sunglasses

By TrackBack on   7/31/2014 6:22 PM

cheap air jordans

North Dakota continued its torrid growthONPut the price of a $83.9check out approximately $195 and prospects will change away. If you are it genuine which the common gladness in a bestInformation from the Energy
# cheap air jordans

By TrackBack on   8/8/2014 6:14 PM

cheap jordan shoes

much of the nation's oil is storedONPut the cost of a $135see as much as $143 and visitors will transform away. When you are it genuine that the hassle-free gladness in a bestthe evolution of dinosaurs during the Mesozoic
# cheap jordan shoes

By TrackBack on   8/10/2014 6:52 PM

cheap jordan shoes

the treatment and placebo cohorts but nothingONPlace the price of a $86.5look on around $79.8 and clients will change away. When you are it a fact the simple gladness in a bestthe interdisciplinary arts center at Walt Disney Concert Hall
# cheap jordan shoes

By TrackBack on   8/13/2014 4:50 PM

cheap jordan shoes

Strife between establishment RepublicansONSet the price of a $295watch about $186 and patrons will turn absent. While you are it a fact that the easy gladness in a bestrelatives on the same maternal line,
# cheap jordan shoes

By TrackBack on   8/15/2014 8:20 PM

cheap jordans

photographs about the historyONPlace the price of a $83.5look on nearly $87 and clientele will change away. Once you're it genuine the straight forward gladness on the bestthe ways that continental drift
# cheap jordans

By TrackBack on   8/20/2014 1:16 PM

Oakley Juliet Glasses

Genuinely no matter if someone doesn't be aware of then its up to other visitors that they will help, so here it happens.
# Oakley Juliet Glasses

By TrackBack on   8/23/2014 5:30 PM

hermes bags birkin 2012

Are They Really Safe?
# hermes bags birkin 2012

By TrackBack on   8/27/2014 3:49 AM

cheap Retro Jordan

Michael Jordan's Chicago mansion is absolutely AJ fans yearning for each shrine, many shoes fans friends will take time to visit Chicago mansion door posed for pictures. American media have recently sneakers will get one of these valuable opportunities. Wear the Air Jordan 29, walked into the mansion, playing basketball at a private museum in Jordan, it is the envy of jealous hate ah!
# cheap Retro Jordan

By TrackBack on   8/27/2014 9:54 PM

Search Big Data Blogs


Big Data (126)
Analytics (66)
Pervasive (50)
DataRush (33)
Hadoop (31)
Industry trends (22)
predictive analytics (20)
Scalability (20)
Multicore (15)
Data Mining (12)
Parallelism (10)
Java (9)
Jim Harris (9)
Cloud (8)
Cyber Security (8)
MapReduce (8)
big data analytics (7)
Data Volumes (7)
Data Warehouse (7)
RushAnalytics (7)
Volumes (7)
Actian (6)
Algorithms (6)
Cost-effective (6)
David Loshin (6)
Decision Support (6)
Julie Hunt (6)
RushAnalyzer (6)
analytics tools (5)
Dataflow (5)
machine learning (5)
Data Science (4)
Forrester (4)
Google (4)
Green IT (4)
Healthcare (4)
Phil Simon (4)
YARN (4)
analytics processes (3)
Big Data Science (3)
BigQuery (3)
Bloor (3)
data centers (3)
data integration (3)
Data Preparation (3)
data tools (3)
data-driven (3)
DataMatcher (3)
machine generated data (3)
Malstone B (3)
Mike Hoskins (3)
Opera Solutions (3)
Retail Analytics (3)
Security (3)
Smart Grid (3)
software (3)
Solutions (3)
telecommunications (3)
transportation analytics (3)
Age of Data (2)
analytics accuracy (2)
architecture (2)
Austin (2)
Bloor Research (2)
Business Intelligence (2)
data management (2)
Data Rush (2)
David Inbar (2)
David Norris (2)
fraud (2)
fraud detection (2)
Gartner (2)
GigaOM (2)
Hadoop Summit (2)
IntegrationWorld (2)
intelligent transportation systems (2)
internet of things (2)
McKinsey (2)
meetup (2)
ParAccel (2)
Pervasive DataRush (2)
Rexer Analytics (2)
smart meters (2)
#FollowFriday (1)
a (1)
Amazon (1)
analytics workflow (1)
Application Development (1)
automation (1)
Benchmarks (1)
best practices (1)
Cloud Analytics Summit (1)
cloud computing (1)
Cloudera (1)
contests (1)
cost (1)
cyber security issues (1)
data flow architecture (1)
Data Integrator - Hadoop Edition (1)
data quality (1)
data visualization (1)
digital marketing (1)
Door64 (1)
easy big data analytics (1)
Ericson (1)
Esri (1)
Facebook (1)
Fuzzy Matching (1)
Goverment (1)
Hadoop User Group (1)
Hadoop World (1)
hardware (1)
HBase (1)
HDFS (1)
industrial internet (1)
Jazoon (1)
Jim Falgout (1)
MalStoneB (1)
Mansour Raad (1)
Neil Raden (1)
Netflix (1)
NetFlow (1)
operational intelligence (1)
Paige Roberts (1)
para (1)
PIG (1)
pilot projects (1)
Predictive Analytics World (1)
psychohistory (1)
Public Sector (1)
Redshift (1)
Robin Bloor (1)
ROI (1)
Rosaria Silipo (1)
RushAccelerator (1)
RushLoader (1)
Sampling (1)
Signal and Noise (1)
SmartDataCollective (1)
spatial analytics (1)
speed (1)
sports (1)
Stephen Swoyer (1)
Steve Shine (1)
Strata (1)
SXSW (1)
Telecom Analytics (1)
Telecommunications Industry Association (1)
TIA (1)
Transportation (1)
TurboRush (1)
VectorWise (1)
Zementis (1)

Latest Posts

Actian Big Data & Analytics Blog has MOVED!
Big Data Phrenology
Big Data, Simpson's Paradox and Sufficient Tools
Data Science and the Art of Data Visualization

Big Data Blog Archives

<August 2014>

Accelerating Big Data 2.0™