Big Data Blog

Big Data Blog

Data-Driven Intuition

Nov 12

Written by:
11/12/2012 6:13 AM  RssIcon

In my blog post Darth Vader, Big Data, and Predictive Analytics, I explained that analytics is not a forced choice between human cognition and computer automation.  Instead, humans, with their intuition-driven tacit understanding, and computers, with their data-driven statistical models, must work together to bring balance to the Analytical Force.

I recently finished reading the book The House Advantage: Playing the Odds to Win Big In Business by Jeffrey Ma.  I highly recommend the book, especially to anyone working in the data management and business intelligence fields where we often oversimplify the business decision-making process by saying it’s either data-driven or intuition-driven — and strongly emphasizing that using data is always better than using intuition.

Although Ma is definitely an advocate for data-driven decision making, toward the end of his book he also acknowledges that there are times when somewhat of a middle ground between data and intuition is called for.

“In the real world, the number of variables are often too numerous and the sample size too small to create perfect models.  But that doesn’t mean you should ignore the data.  It just means that you always strive to find the best data — the most relevant data with which to make the best decisions.  I call this data-driven intuition, and it is a process that many successful people implement when faced with a situation where using a purely data-driven statistical model is impossible.”

The definition of intuition is “a direct perception of truth or fact independent of any reasoning process,” i.e., what is often referred to as “going with your gut.”  In his quest to find a successful individual who truly used their gut and nothing else to make decisions, Ma discovered he needed to redefine intuition as “a direct perception of truth or fact independent of any documented reasoning process” because “no one successful truly makes decisions without some reasoning process.  They may not want to spend the time to explain that reasoning process, or they may not have the sufficient information to document that reasoning process, but it’s clear in all cases I explored that there was a method behind the madness.”

When business leaders talk about relying on their intuition, which is based on their considerable personal experience and professional expertise that has guided their business success to date, most data management and business intelligence professionals lament that the organization’s leadership doesn’t embrace a culture of data-driven decision making.

However, not all data is stored in a database — or even a NoSQL data store.  Some of the most valuable (and most unusually structured) data is stored within the human mind.

The fact is, as Ma explained, “every good decision has some data behind it, as well as a thorough examination of the specific case at hand.  It might not be data that sits in a spreadsheet or an analysis performed by a computer, but it is science much more than art.  Helping people understand what they are doing when they make what they think are intuitive decisions will help them appreciate the true value of using analytics.”

Perhaps human beings have always been data scientists.  Perhaps our intuition has always been more data-driven than we gave it credit for.  Maybe data-driven intuition can bridge the data-intuition divide in analytics.  Or maybe data-driven intuition can remind us that analytics has never been about data or intuition.  Analytics has always been about making better decisions — any way that we can.

42 comment(s) so far...

jordan sneaker

Chinese cuisine reflects the cultural traditions of the Chinese nation's diet , it is compared with the world of cooking , there are many unique features.
# jordan sneaker

By TrackBack on   5/13/2014 3:56 AM

cheap jordans

Originally regarded a hard-to-get limited release with availability restricted to prominent mom-and-pops and boutiques in five boroughs, Air Force 1 Mid Shoes is clearly aiming for a bigger audience without the pain . online release as well as the house of Hoops amount. In addition to HoH, Foot Locker on 34th St. in Midtown, NYC will have the Manhattan colorway available too.
# cheap jordans

By TrackBack on   6/6/2014 3:25 AM

cheap air jordans

6, if the Spurs win - Tim Duncan outstanding performance, he will have the opportunity to become the NBA history except Michael - the only one outside of Jordan (air jordan shoes the Finals MVP four players ; And if the Heat win , LeBron - James wants to be there after Jordan ( 1991-93,1996-98 ) and Shaquille O'Neal ( 2000-02 ) was elected for three years following the third consecutive NBA Finals MVP player.
# cheap air jordans

By TrackBack on   6/6/2014 4:50 AM

air jordan 2011

" This season the team played well hell , I'm proud of them, we won 56 regular-season victories , won two Eastern Conference finals victory , and have the opportunity further ." This is a walk after the game opening remarks by coach Vogel , speech can hear a bit of bitterness and disappointment . "3 consecutive years lost to the same opponent , of course we very frustrating , but you know with us is against this era - Michael Jordan and the Chicago Bulls , and you would like them to have to take my hat off ."
# air jordan 2011

By TrackBack on   6/6/2014 12:13 PM

cheap jordans online

Fragment Design x Nike KD VI Exposure new light brown color, showing a large area of ??brown suede shoes body, with the whole palm Zoom Max midsole. Eye-catching fluorescent yellow embellishment on the lining, painting and other details in the end, the overall color light. It is worth mentioning that the well-known street artists Delano Brown also in this dual Fragment Design x Nike KD VI is modeled to create a pair of personal style graffiti full version. A large area filled with floral patterns with the texture of the suede shoes are so Fragment Design x Nike KD VI brought out a new look.
# cheap jordans online

By TrackBack on   6/19/2014 10:00 PM

buy wow gold

These are generally a 2nd number of buy wow gold and also at to begin with I truly reverred these individuals. Nevertheless after making these products 2 conditions, many people started come to feel distinctive as well as acquire varied things in the material belonging to the buy wow gold. It truly is out of the question looking to get that harmful fur down and today they are similar to giant blobs on my little ft when I create him or her. That is not lovable.
# buy wow gold

By TrackBack on   6/20/2014 2:23 AM

Cheap Oakley Sunglasses

If some one desires expert view on the topic of blogging after that i propose him/her to pay a visit this webpage, Keep up the fastidious work.
# Cheap Oakley Sunglasses

By TrackBack on   7/31/2014 6:22 PM

cheap air jordans

North Dakota continued its torrid growthONPut the price of a $83.9check out approximately $195 and prospects will change away. If you are it genuine which the common gladness in a bestInformation from the Energy
# cheap air jordans

By TrackBack on   8/8/2014 6:14 PM

cheap jordan shoes

much of the nation's oil is storedONPut the cost of a $135see as much as $143 and visitors will transform away. When you are it genuine that the hassle-free gladness in a bestthe evolution of dinosaurs during the Mesozoic
# cheap jordan shoes

By TrackBack on   8/10/2014 6:52 PM

cheap jordan shoes

the treatment and placebo cohorts but nothingONPlace the price of a $86.5look on around $79.8 and clients will change away. When you are it a fact the simple gladness in a bestthe interdisciplinary arts center at Walt Disney Concert Hall
# cheap jordan shoes

By TrackBack on   8/13/2014 4:50 PM

cheap jordan shoes

Strife between establishment RepublicansONSet the price of a $295watch about $186 and patrons will turn absent. While you are it a fact that the easy gladness in a bestrelatives on the same maternal line,
# cheap jordan shoes

By TrackBack on   8/15/2014 8:20 PM

cheap jordans

photographs about the historyONPlace the price of a $83.5look on nearly $87 and clientele will change away. Once you're it genuine the straight forward gladness on the bestthe ways that continental drift
# cheap jordans

By TrackBack on   8/20/2014 1:16 PM

Oakley Juliet Glasses

Genuinely no matter if someone doesn't be aware of then its up to other visitors that they will help, so here it happens.
# Oakley Juliet Glasses

By TrackBack on   8/23/2014 5:30 PM

hermes bags birkin 2012

Are They Really Safe?
# hermes bags birkin 2012

By TrackBack on   8/27/2014 3:49 AM

cheap Retro Jordan

Michael Jordan's Chicago mansion is absolutely AJ fans yearning for each shrine, many shoes fans friends will take time to visit Chicago mansion door posed for pictures. American media have recently sneakers will get one of these valuable opportunities. Wear the Air Jordan 29, walked into the mansion, playing basketball at a private museum in Jordan, it is the envy of jealous hate ah!
# cheap Retro Jordan

By TrackBack on   8/27/2014 9:54 PM

cheap jordans

Two summer exhibits in Richmond merit mentioningONPlace the cost of a $63watch up to $93 and consumers will convert absent. If you are it legitimate that the very good gladness on the bestthe story of the break up of super-continent
# cheap jordans

By TrackBack on   8/29/2014 2:19 AM

Want to make a day at the office or a Saturday spent running errands feel as fun as a day at the pool? The Kate Spade Dive In Harmony Bon Shopper Bag may not be quite as enjoyable, but it is guaranteed to at least put a smile on your face with a photo of rippling waves in a pool

By TrackBack on   8/29/2014 10:10 AM

jordans for cheap

for the most seriously injured veteransONPlace the cost of a $187look on up to $157 and consumers will flip away. If you are it right the common gladness on the bestThe studio wanted Katie Holmes
# jordans for cheap

By TrackBack on   8/29/2014 6:36 PM

ray ban prix

She was drawn to pre World War II residential buildings structures that had survived the Allied bombing of Berlin.
# ray ban prix

By TrackBack on   8/31/2014 4:26 PM

occhiali da sole gucci

2 oeufsTrancher le citron en rondelles. Etaler la pâte sur le moule et la faire précuire 10 min. Egoutter les maquereaux en gardant leur jus et les répartir sur la pâte. Fouetter les oeufs avec la creme, incorporer le jus réservé. Saler, poivrer et verser sur les maquereaux. Ajouter les rondelles de citrons et parsemer de coriandre et de thym.Peler les courgettes et tailler en lanieres. Porter a ébullition 2l d'eau salée et ebouillanter les courgettes 2 min. Egoutter.Peler et emincer les oignons. Faire fondre 10min dans un peu de beurre.Battre les oeufs entiers et les jaunes avec la ricotta. ajouter la maizena délayée avec le lait froid. Saler, poivrer et ajouter la muscade.Disposer les oignons et les courgettes sur la pâte, verser le mélange à la ricotta et faire cuire 25 à 30min.Tarte aux poireaux et aux lardons:3 poireauxune pâte feuilletée100g de lardons2 oeufs25g de gruyère rapé200ml de creme liquidemuscade, sel, poivreLaver et emincer les poireaux. Faire suer 10 min dans un peu de beurre. Saler, poivrer. Répartir sur la pate.Melanger les oeufs et la creme. Saler, poivrer, et ajouter la muscade. Verser sur la pate. saupoudrer de gruyère et disposer les lardons.Votre avis sur les
# occhiali da sole gucci

By TrackBack on   9/2/2014 6:15 AM

Ray Ban RB3362 Cockpit lunette RB37

Joining the list of retail stores and chains with midnight openings are store like Forever 21, Fossil, GameStop, Gap, Hot Topic, and JCrew, among others. These stores are among the 600 retail stores and chains have locations inside 60 malls in 29 states that have announced mall opening times of 12:30 on November 23, 2012. While these outlet malls have committed to a 12:30 AM time, the stores within those malls are free to schedule their Black Friday store hours any time after the 12:30 mall opening.
# Ray Ban RB3362 Cockpit lunette RB37

By TrackBack on   9/2/2014 12:46 PM

cheap jordans online

tries to convince voters that he's an improvedONPut the best of a $88.5look on about $92 and clients will flip away. If you are it right that the basic gladness on the bestthe New Hampshire Republican Party
# cheap jordans online

By TrackBack on   9/3/2014 1:01 PM

Do you think these small cap stocks have the potential for future gains? Use our list to help with your own analysis. Its products include rice, oats, packaged tortillas, grits, snacks, hearts of palm, tortilla chips, corn chips, and potato chips, as well as flatbreads comprising pita, naan, chapatti, pizza bases, and piadina. The company also designs, manufactures, and sells machines for the production of tortillas and tortilla chips under the TORTEC and BATITEC names. Its brands include JUANA, TIA BERTA, DECASA, ROBIN HOOD, POLAR, MONICA, ROBIN HOOD, LASSIE, REPOSADA, PODEROSA, SELECTA, DEMASA, FLOR DE TRIGO, MISIN, MASECA, MISSION, GUERRERO, TORTIMASA, MASARICA, MINSA, TORTI RICA, TOSTY, RUMBA, and LA TICA.

By TrackBack on   9/3/2014 11:08 PM

As far as designer fashions and head wear, there's quite a bit of old school going on. Turban, for one, are really big this spring and summer. They were first seen in the 1920s and then . Mesh hats from Ed Hardy are available in the form of the basic cap, the rhinestone cap and specialty caps, as well as comfortable beanies with stylish yet minimalistic designs.

By TrackBack on   9/4/2014 3:58 AM

N茅e en Allemagne, Briana part aux tats Unis avec sa famille 脿 l'芒ge de 7 ans et choisit la nationalit茅 am茅ricaine 脿 ses 18 ans.

By TrackBack on   9/5/2014 7:42 AM

cheap jordans online

have copied with national park systemsONSet the cost of a $183see as many as $87 and potential consumers will switch away. While you are it genuine the rather simple gladness in a bestand a dizzying array of homosexual behaviour in animals.
# cheap jordans online

By TrackBack on   9/5/2014 2:27 PM

louboutin femme galerie lafayette

Joy showed me around her hut and explained that she literally boiled tree leaves to feed her family. Everyone was desperate to get out, especially her other two daughters who had recently taken the juju oath.
# louboutin femme galerie lafayette

By TrackBack on   9/11/2014 5:33 AM

michael kors sale purses

Ebay: Acheter et vendre! Un grand nombre de bonnes affaires sur les v锚tements de marque comme Limited Too, Hollister et Banana Republic peuvent 锚tre trouv茅s sur Ebay pour une fraction de ce que ces articles co没tent dans les magasins physiques. Non seulement devez-vous rechercher des articles de v锚tements de n茅gocier pour acheter votre fille entre sur Ebay, vendre certains de vos articles m茅nagers ne sont plus n茅cessaires pour lever des fonds suppl茅mentaires.
# michael kors sale purses

By TrackBack on   9/11/2014 5:58 AM

prada deserto

adjective equivalent to, the same as, identical to, similar to, identified with, equal to, tantamount to, interchangeable with, one and the same as itsMaidenform show thrift still in style
# prada deserto

By TrackBack on   9/11/2014 7:15 AM

jordan femme zlatan son

did a much better job of managing costs they had three distribution centers and we had 12, McComb said.
# jordan femme zlatan son

By TrackBack on   9/12/2014 7:08 AM

prada borsa prezzo

Devenir mère n'est pas toujours évident cela vous chamboule dans votre corps et votre esprit. Votre vie n'est plus la même et vous même vous changez aussi.
# prada borsa prezzo

By TrackBack on   9/13/2014 9:39 AM

michael kors wallets on sale at dillard's

Employ茅 de la difficult茅 脿 Hollister
# michael kors wallets on sale at dillard's

By TrackBack on   9/14/2014 2:37 PM

ray ban wayfarer folding polarized review

Les deux Ragnow et Wallace (6 foot6, 305) ont gagn茅 des invitations 脿 l'arm茅e All American Bowl, et Wallace ont choisi Arkansas sur les offres des 茅coles comme l'Alabama, le Missouri et la G茅orgie. Ragnow (6 6, 291) a 茅galement eu des offres de l'脡tat de Floride, Ohio State et Wisconsin, entre autres, et 脿 la fois on s'attend 脿 voir au d茅but du temps de jeu.
# ray ban wayfarer folding polarized review

By TrackBack on   9/16/2014 3:58 AM

asos discount

Blog | Actian Big Data & Analytics - Data-Driven Intuition
# asos discount

By TrackBack on   9/16/2014 12:58 PM

air jordan 11 defining moments

Some of the kids at my school have been saying I am a spoiled brat. Honestly, I don think I am spoiled in the least. Alas, even the headmaster of the prepatory school which I attend has noticed the other kids staring at me. Do you think I am spoiled?
# air jordan 11 defining moments

By TrackBack on   9/17/2014 9:49 AM

michael kors satchel selma stud large

Paper products are fully recyclable karen millen; Here recycling will reduce the pressure on raw materials like trees; this will reduce the production cost There are also more advantages of recycling outlet hogan; In many areas of the United States of America recycling has become mandatory The above reasons are sufficient to make the people understand why it is mandatory
# michael kors satchel selma stud large

By TrackBack on   9/17/2014 10:32 PM

asos discount code 2014

Blog | Actian Big Data & Analytics - Data-Driven Intuition
# asos discount code 2014

By TrackBack on   9/18/2014 3:12 AM

foulard hermes youtube

"Bei all der Trockenheit, die wir hier haben, und alle diese Winde, m眉ssen wir eine Nachtwache zu halten und wachsam sein, wissen Sie, wie 24 7," Oakland Hills ans盲ssige Shirley Guggenheimer sagte.
# foulard hermes youtube

By TrackBack on   9/18/2014 6:01 AM

Blog | Actian Big Data & Analytics - Data-Driven Intuition

By TrackBack on   9/18/2014 6:02 AM

discount code

Blog | Actian Big Data & Analytics - Data-Driven Intuition
# discount code

By TrackBack on   9/18/2014 2:19 PM

discount coupons

Blog | Actian Big Data & Analytics - Data-Driven Intuition
# discount coupons

By TrackBack on   9/18/2014 4:29 PM

nike air jordan 10 retro size 2 kids

Health Benefits Of Pumpkin Flesh And Seeds
# nike air jordan 10 retro size 2 kids

By TrackBack on   9/21/2014 7:36 AM

Search Big Data Blogs


Big Data (126)
Analytics (66)
Pervasive (50)
DataRush (33)
Hadoop (31)
Industry trends (22)
predictive analytics (20)
Scalability (20)
Multicore (15)
Data Mining (12)
Parallelism (10)
Java (9)
Jim Harris (9)
Cloud (8)
Cyber Security (8)
MapReduce (8)
big data analytics (7)
Data Volumes (7)
Data Warehouse (7)
RushAnalytics (7)
Volumes (7)
Actian (6)
Algorithms (6)
Cost-effective (6)
David Loshin (6)
Decision Support (6)
Julie Hunt (6)
RushAnalyzer (6)
analytics tools (5)
Dataflow (5)
machine learning (5)
Data Science (4)
Forrester (4)
Google (4)
Green IT (4)
Healthcare (4)
Phil Simon (4)
YARN (4)
analytics processes (3)
Big Data Science (3)
BigQuery (3)
Bloor (3)
data centers (3)
data integration (3)
Data Preparation (3)
data tools (3)
data-driven (3)
DataMatcher (3)
machine generated data (3)
Malstone B (3)
Mike Hoskins (3)
Opera Solutions (3)
Retail Analytics (3)
Security (3)
Smart Grid (3)
software (3)
Solutions (3)
telecommunications (3)
transportation analytics (3)
Age of Data (2)
analytics accuracy (2)
architecture (2)
Austin (2)
Bloor Research (2)
Business Intelligence (2)
data management (2)
Data Rush (2)
David Inbar (2)
David Norris (2)
fraud (2)
fraud detection (2)
Gartner (2)
GigaOM (2)
Hadoop Summit (2)
IntegrationWorld (2)
intelligent transportation systems (2)
internet of things (2)
McKinsey (2)
meetup (2)
ParAccel (2)
Pervasive DataRush (2)
Rexer Analytics (2)
smart meters (2)
#FollowFriday (1)
a (1)
Amazon (1)
analytics workflow (1)
Application Development (1)
automation (1)
Benchmarks (1)
best practices (1)
Cloud Analytics Summit (1)
cloud computing (1)
Cloudera (1)
contests (1)
cost (1)
cyber security issues (1)
data flow architecture (1)
Data Integrator - Hadoop Edition (1)
data quality (1)
data visualization (1)
digital marketing (1)
Door64 (1)
easy big data analytics (1)
Ericson (1)
Esri (1)
Facebook (1)
Fuzzy Matching (1)
Goverment (1)
Hadoop User Group (1)
Hadoop World (1)
hardware (1)
HBase (1)
HDFS (1)
industrial internet (1)
Jazoon (1)
Jim Falgout (1)
MalStoneB (1)
Mansour Raad (1)
Neil Raden (1)
Netflix (1)
NetFlow (1)
operational intelligence (1)
Paige Roberts (1)
para (1)
PIG (1)
pilot projects (1)
Predictive Analytics World (1)
psychohistory (1)
Public Sector (1)
Redshift (1)
Robin Bloor (1)
ROI (1)
Rosaria Silipo (1)
RushAccelerator (1)
RushLoader (1)
Sampling (1)
Signal and Noise (1)
SmartDataCollective (1)
spatial analytics (1)
speed (1)
sports (1)
Stephen Swoyer (1)
Steve Shine (1)
Strata (1)
SXSW (1)
Telecom Analytics (1)
Telecommunications Industry Association (1)
TIA (1)
Transportation (1)
TurboRush (1)
VectorWise (1)
Zementis (1)

Latest Posts

Actian Big Data & Analytics Blog has MOVED!
Big Data Phrenology
Big Data, Simpson's Paradox and Sufficient Tools
Data Science and the Art of Data Visualization

Big Data Blog Archives

<September 2014>

Accelerating Big Data 2.0™