Big Data Blog

Big Data Blog

Use Your Disillusion

Feb 14

Written by:
2/14/2013 6:00 AM  RssIcon

According to Svetlana Sicular, Research Director at Gartner, Big Data is Falling into the Trough of Disillusionment.  Basically, what that really means is that people are starting to understand that big data analytics solutions are not going to bring about world peace, make everyone instant billionaires, and cure the common cold. Don’t Cry, though. This doesn’t mean the promise of big data is all Dust N’ Bones or Double Talkin’ Jive.  What this really means is that we have a chance now to get past the hype, and find the real kernel of truth underlying the bigger-than-life myth. Guns N’Roses may not have known anything about big data, but they almost had the right idea.

This disillusionment is an opportunity for savvy companies to make a leap forward. If you think that the overwhelming media focus on new big data analytics methods and technologies is just a Bad Obsession, well You Ain’t the First. But the thing about the trough of disillusionment is that it leads eventually to the plateau of productivity, and that’s where businesses get the real benefit from new technology. When people begin to understand the limitations of new tech, that’s when they Get in the Ring and start really putting it to good use.

Neil Capel on Wired hit the nail on the head in his article: Smart Data: For Communications, Big Data Becomes Transformational

It’s not Big Data’s fault that it’s disorganized and not actionable by itself. In some respects, the chatter around the technology’s potential has done the actuality of its deliverables a disservice. Many looked at what it could do; others critiqued what it should do and very few companies actually committed to their Big Data strategy.

To get a little more specific, when I talk about big data tech, I’m usually thinking of Hadoop. There are other options, of course: appliances, HPCC, even traditional databases scaled to the extreme. The biggest advantage of Hadoop over the other options is one that matters to every business: cost.

On the Harvard Business Review Blog Network, Paul Barth and Randy Bean produced an excellent article on the results of a survey by NewVantage Partners:  Get the Maximum Value Out of Your Big Data Initiative.  There’s a lot of excellent information and advice in the article, but the one thing that clearly shows why Hadoop is the dominant big data technology in practical applications is summed up in this graph:

Yes, Hadoop can process a lot of data for a reasonable price. The thing is, and prepare to be shocked: Hadoop is not a panacea. It can’t do absolutely everything really well. This is where the disillusionment starts to set in, because if you believe all the press, you’d think Hadoop could cure world hunger and wash your socks. Don’t get me wrong. There is huge potential in the Hadoop model, it’s just never going to live up to the kind of insane press it’s been getting. Nothing could.

The MapReduce programming framework is the source of some of Hadoop’s limitations. Vincent Granville did an excellent article recently on Analytic Bridgewith some specifics on What MapReduce Can’t Do. Once YARN breaks the Hadoop operating environment free of MapReduce restrictions, I expect an explosion of applications and frameworks designed to run on that environment, each specialized to do something really well on Hadoop. Instead of trying to do everything with one tool, many tools will be created, each ideal to one specific aspect of big data processing.

The real advantage of this new technology, aside from the fact that it can handle massive volumes of data without choking and dying, or killing your operating budget, is speed. We are not a patient society. The cutthroat world of business has become all about who can get the right answer faster.

If I can place a correctly targeted ad on a Web page two seconds before you can, chances are, I made the sale and you didn’t. If I can catch cybersecurity intrusions two seconds faster, I might save my company millions. If I can predict when a switch is overloaded two seconds before it fails and re-route, so that my calls drop less than my telecom competitors, I get thousands more loyal customers. Two seconds can be an eternity these days.

From that same HBR blog article by Paul Barth and Randy Bean that had the relative cost graph comes a quote that really brings the point home:

“While 85% of the participating companies have Big Data initiatives underway with the stated reason of improving analytic capabilities and making smarter business decisions, our subsequent briefings highlighted that the real quantum leap for companies comes from the ability to accelerate the speed at which they can get to a decision.

That last bit is the kicker. Welcome to My World. It’s pretty much what I was trying to say in my SmartDataCollective article on Predictive Analytic Strategies to Out-Predict the Competition. Big data technologies applied to normal analytics workloads become predictive analytics accelerators. Big data technologies applied to real modern business analytics needs, give you the answers you need faster, no matter how big your data is.

But that’s all Hadoop and the other new big data technologies do.

They can’t stop North Korea from testing nukes. They can’t predict the lottery numbers for you. And they can’t get your teenage son to put the seat down on the toilet. Get past the disillusionment with what they can’t do, and use your disillusion to finally reach the potential of what big data technologies genuinely can do for your company. Or, you might just find your business Knockin’ on Heaven’s Door.


Pervasive Big Data & Analytics

Search Big Data Blogs


Big Data (126)
Analytics (66)
Pervasive (50)
DataRush (33)
Hadoop (31)
Industry trends (22)
predictive analytics (20)
Scalability (20)
Multicore (15)
Data Mining (12)
Parallelism (10)
Java (9)
Jim Harris (9)
Cloud (8)
Cyber Security (8)
MapReduce (8)
big data analytics (7)
Data Volumes (7)
Data Warehouse (7)
RushAnalytics (7)
Volumes (7)
Actian (6)
Algorithms (6)
Cost-effective (6)
David Loshin (6)
Decision Support (6)
Julie Hunt (6)
RushAnalyzer (6)
analytics tools (5)
Dataflow (5)
machine learning (5)
Data Science (4)
Forrester (4)
Google (4)
Green IT (4)
Healthcare (4)
Phil Simon (4)
YARN (4)
analytics processes (3)
Big Data Science (3)
BigQuery (3)
Bloor (3)
data centers (3)
data integration (3)
Data Preparation (3)
data tools (3)
data-driven (3)
DataMatcher (3)
machine generated data (3)
Malstone B (3)
Mike Hoskins (3)
Opera Solutions (3)
Retail Analytics (3)
Security (3)
Smart Grid (3)
software (3)
Solutions (3)
telecommunications (3)
transportation analytics (3)
Age of Data (2)
analytics accuracy (2)
architecture (2)
Austin (2)
Bloor Research (2)
Business Intelligence (2)
data management (2)
Data Rush (2)
David Inbar (2)
David Norris (2)
fraud (2)
fraud detection (2)
Gartner (2)
GigaOM (2)
Hadoop Summit (2)
IntegrationWorld (2)
intelligent transportation systems (2)
internet of things (2)
McKinsey (2)
meetup (2)
ParAccel (2)
Pervasive DataRush (2)
Rexer Analytics (2)
smart meters (2)
#FollowFriday (1)
a (1)
Amazon (1)
analytics workflow (1)
Application Development (1)
automation (1)
Benchmarks (1)
best practices (1)
Cloud Analytics Summit (1)
cloud computing (1)
Cloudera (1)
contests (1)
cost (1)
cyber security issues (1)
data flow architecture (1)
Data Integrator - Hadoop Edition (1)
data quality (1)
data visualization (1)
digital marketing (1)
Door64 (1)
easy big data analytics (1)
Ericson (1)
Esri (1)
Facebook (1)
Fuzzy Matching (1)
Goverment (1)
Hadoop User Group (1)
Hadoop World (1)
hardware (1)
HBase (1)
HDFS (1)
industrial internet (1)
Jazoon (1)
Jim Falgout (1)
MalStoneB (1)
Mansour Raad (1)
Neil Raden (1)
Netflix (1)
NetFlow (1)
operational intelligence (1)
Paige Roberts (1)
para (1)
PIG (1)
pilot projects (1)
Predictive Analytics World (1)
psychohistory (1)
Public Sector (1)
Redshift (1)
Robin Bloor (1)
ROI (1)
Rosaria Silipo (1)
RushAccelerator (1)
RushLoader (1)
Sampling (1)
Signal and Noise (1)
SmartDataCollective (1)
spatial analytics (1)
speed (1)
sports (1)
Stephen Swoyer (1)
Steve Shine (1)
Strata (1)
SXSW (1)
Telecom Analytics (1)
Telecommunications Industry Association (1)
TIA (1)
Transportation (1)
TurboRush (1)
VectorWise (1)
Zementis (1)

Latest Posts

Actian Big Data & Analytics Blog has MOVED!
Big Data Phrenology
Big Data, Simpson's Paradox and Sufficient Tools
Data Science and the Art of Data Visualization

Big Data Blog Archives

<October 2014>

Accelerating Big Data 2.0™