Big Data Report 2012
description
Transcript of Big Data Report 2012
-
Big DataThe Next Big Thing
MAKING MAR
K
ETSFUNCTIONBETTER
YEARS
GLOBAL RESEARCH & ANALYTICSGLOBAL RESEARCH & ANALYTICS
-
International Youth Centre, Teen Murti Marg, Chanakyapuri, New Delhi - 110 021, India
Phone: 91-11-23010199, Fax: 91-11-23015452, Email: [email protected]
Website: www.nasscom.in
-
Big Data: The Next Big Thing
2
Copyright 2012
International Youth Centre, Teen Murti Marg, Chanakyapuri
New Delhi - 110 021, India
Phone: 91-11-23010199, Fax: 91-11-23015452
Email: [email protected]
Published by
NASSCOM, New Delhi
Designed & Produced by
CREATIVE INC.
Phone: 91-11-41634301
Printed atP.S. Press Services
Disclaimer
The information contained herein has been obtained from sources believed to be reliable. NASSCOM and
CRISIL GR&A disclaim all warranties as to the accuracy, completeness or adequacy of such information.
NASSCOM and CRISIL GR&A shall have no liability for errors, omissions or inadequacies in the information
contained herein, or for interpretations thereof.
Service provider pro les are representative of the Indian players. We have tried to cover players across
the Big Data spectrum hardware, software, analytics, system integration and IT services. Identi cation
of players is based on reliable industry sources, interviews, and organisation websites. This report is not
a recommendation to invest/disinvest in any organisation covered in the report.
The material in this publication is copyrighted. No part of this report may be reproduced either on paper or
electronic media in part or in full without permission in writing from NASSCOM. Request for permission
to reproduce any part of the report may be sent to NASSCOM.
Usage of Information
Forwarding/copy/using in publications without approval from NASSCOM will be considered as
infringement of intellectual property rights.
-
Big Data: The Next Big Thing
3
Every few years, we come across the next big technological idea which radically transforms the way businesses function by opening up new opportunities and effi ciencies. Big Data has now emerged as the next big thing the big idea whose time has come. And like most big ideas in the recent past, Big Data off ers a big opportunity for India.
In this study, jointly conducted by NASSCOM & CRISIL Global Research and Analytics (GR&A), we look at the opportunity, which lies in off ering services around Big Data implementation and analytics for global multinationals. By 2015, Big Data is expected to become a USD 25 billion industry, driven by uses across industries such as manufacturing, retail, nancial services, telecom and healthcare. We expect the Indian Big Data industry to grow from USD 200 million in 2012 to USD 1 billion in 2015 at a CAGR in excess of 83 per cent Indian service providers are already leveraging partnerships, M&As and venture funding to capture Big Data outsourcing opportunity. We are con dent that India will be at the forefront in off ering Big Data analytics and related IT services. The challenge, however, is in meeting the demand of data scientists and IT engineers which is estimated to reach approximately 15,000-20,000, at a CAGR of 80 per cent by 2015. The signs, though, are encouraging.
India follows close on the heels of the US and is well ahead of other outsourcing destinations in terms of Big Data talent availability and service providers initiatives to build such talent for the Big Data opportunity.
To further augment this capacity, organisations are leveraging their academic alliance programmes, with universities in India to introduce courses on various areas of Big Data. Their eff orts are being complemented by private IT training institutes in the country, which are developing talent through courses speci c to Big Data skills.
Today, data is omniscient and omnipresent. This data is getting generated at a rapid pace: around 2.5 billion GB of data is generated every day, and more than 90 per cent of the data available today has been created in the past 3-4 years. This has primarily been because of the explosion in our use of click stream, mobile applications and social media. Its estimated that Twitter alone generates 12 Terabytes of data daily. Its a
gold mine for businesses which can separate the wheat from the chaff to identify the trends. Organisations across segments are now looking at this pool of data to determine how best it can be mined and gauge their customers likes and dislikes.
Storing, analysing and making sense of data of such unwieldy dimension will be a challenge of epic proportions. However, we believe India is on the right path to steal a march over others. In this study, we off er a big perspective on Big Data and how it can be turned into actionable insights.
Foreword
Roopa KudvaManaging Director and CEO, CRISIL
Som MittalPresident, NASSCOM
-
Big Data: The Next Big Thing
4
Acknowledgements 5
Key Takeaways 6
Introduction to Big Data 8
Global Perspective on Big Data 26
Indias Advantage in the Big Data Opportunity 40
The Future of Big Data 71
Annexure 78
Contents
-
Big Data: The Next Big Thing
5
This publication was prepared through a collaborative eff ort by several institutions and individuals. We
would like to acknowledge the support of our Executive Council for providing the essential and gracious
counsel and guidance. NASSCOM has published, and continues to work on, various reports on the
IT-BPO sector; information from these reports have been used in this study.
We gratefully acknowledge the contribution of our members and partners including Genpact, EMC,
Sears Holding, HP Analytics, Mu Sigma, AbsolutData, Computer Sciences Corporation, Deloitte,
Frost & Sullivan, Marlabs, LatentView, EXL Services, Fidelity Investments, Impetus and JP Morgan Chase
in terms of their valuable time and informative case studies.
We deeply appreciate the eff orts of CRISIL Global Research & Analytics (GR&A) and its team comprising
Gaurav Dua, Kumar Rajendran, Priya Khemka, Gunja Rastogi, Mehak Mayor, Praveen Kalani, Hemant Bisht,
Ridhima Sudan, Santosh Kandwal and Sonam Gupta who were instrumental in producing this report.
We also convey our special acknowledgement to NASSCOMs research team for their eff ort and
contribution towards the production of this report.
Acknowledgements
-
Key Takeaways
-
Big Data: The Next Big Thing
7
India showcases competitive advantage in Big Data off erings
-
An Introduction to Big Data
-
Big Data: The Next Big Thing
9
Big Data is de ned by volume, variety and velocity
Organisations worldwide are turning their attention to Big Data as they scramble to derive insights from
the deluge of information generated from various sources. In the past few years, the global marketplace
has seen exponential growth in data volumes, created and consumed by a diverse cross-section of
stakeholders. The term Big Data signi es large datasets in multiple formats, growing at an enormous
rate and posing problems for traditional storage and analytical platforms. Big Data is distinct from large
existing data stored in various relational databases, as it warrants a more advanced mechanism for both
storage and analysis. Technologies such as NoSQL databases and MapReduce/Hadoop frameworks are
at the core of the solutions heralding a paradigm shift. So Big Data is characterised by three attributes
of data: volume, variety and the velocity at which it is generated.
Traditional analytics on transactional or structured data have helped data-driven organisations gain
insights from various enterprise data. As data from weblogs, social media posts, sensors, images, emails,
audio and video les emerge as sources of insights, it presents a huge competitive opportunity for
businesses. The need to derive predictive and actionable insights from this data for improved business
operations and better decision making is what drives Big Data analytics.
-
Big Data: The Next Big Thing
10
The data being generated globally is undergoing exponential growth
Data volume is the primary characteristic of Big Data. With data becoming an indispensable part of
every economy, industry, organisation, business function and individual, it is being actively captured by
organisations to better understand their customers, suppliers, partners and operations. Large datasets
yield more information and hence, improved analysis compared to limited records of data, leading to
better competitive advantage and business operations. This data is being generated at a rapid pace:
around 2.5 billion GB of data is generated every day, and more than 90 per cent of the data available
today has been created in the past 3-4 years. According to IDC, data generated globally is expected to
witness a 41.0 per cent CAGR between 2009 and 2020 to reach 35.0 Zettabytes.
Moreover, the technological landscape has changed with innovation in both managing and storing large
data. As organisations move away from the traditional data storage systems such as le systems and
databases to newer technologies such as cloud-based storage and open source software, data storage
and management costs are seeing a downward trend. According to IDC, storage costs have plummeted
from USD 18.9/gigabyte in 2005 to USD 1.6/gigabyte in 2011, and are expected to further decline to
0.7/gigabyte by 2015. Apart from storage costs, the evolution of several open source analytical tools
and platforms has made data analytics exible, reliable and relatively aff ordable for Big Data.
Volume
Variety
Velocity
-
Big Data: The Next Big Thing
11
Today 80 per cent of data existing in any enterprise is unstructured data
Organisations worldwide are increasingly realising that unstructured data, if analysed, can provide a
competitive edge. While structured data is transactional and can be stored in rows and columns with
an identi able structure, unstructured data such as audio, video and social media messages is raw or
semi-structured. This data is generated in several forms such as web clicks, emails, phone conversations,
weather data, audio and video les, location coordinates and pictures. Moreover, unstructured data
is highly dynamic and does not have a particular format, i.e., it may be in diff erent languages, have
several terminologies, and may exist in the form of X-ray sheets, voice mails, digital photographs, or
phone conversations.
Organisations are overwhelmed by the volume of unstructured data and are looking at ways to manage
and analyze them in a systematic manner. As a result, one of the key focus areas for organisations
wanting to leverage Big Data is to handle unstructured data and adopt new technologies to deal
with them.
It is imperative to develop technologies that can enable storage of such huge data as well as maintain
transactional consistency between structured and unstructured data. Newer technologies such as NoSQL
databases to store unstructured data and processing methods such as Hadoop and massively parallel
processing are gaining prominence in the area of Big Data and Big Data analytics.
Volume
Variety
Velocity
-
Big Data: The Next Big Thing
12
Increased data velocity enables real-time use of Big Data
The proliferation of the internet and the mobile era has increased the rate at which data is created and
stored; hence, there is a need for tools and technologies to analyse data at an equal speed. The shelf-life
of data has dropped from months to hours and seconds.
The ubiquitous nature of the internet, coupled with massive computing power and accessibility, has
transformed data processing from an auxiliary function into an essential mechanism that enables
organisations to transform their businesses. Big Data service providers are increasingly leveraging
technologies such as streaming processing and in-memory computing that mitigates the shortcomings
of batch processing and enable faster storage and data processing.
Earlier, these technologies were popular in verticals considered more critical, such as the nancial and
government sectors. However, as the criticality of analysing data in real-time emerges, several other
industries are also adopting solutions based on these technologies.
Volume
Variety
Velocity
-
Big Data: The Next Big Thing
13
Social media analytics, sentiment analysis and behavioural analysis are the upcoming Big Data analytics services
Big Data analytics is the process of applying advanced analytical techniques to large datasets to
uncover hidden patterns, unknown correlations and other useful information. Big Data analytics
helps businesses:
Take better business decisions: The most important objective of Big Data analytics is to help organisations make better business decisions, taking into account all the available information.
This is achieved by analysing large volumes of structured and unstructured data from sources that
are left unutilised by conventional business intelligence solutions
Predict and identify change: Big Data analytics helps organisations closely monitor their ecosystem, discover what has changed, and decide how they should react. It also enables them to predict
change, which is crucial given the current competitive business environment
Identify new opportunities: Advanced Big Data analytics is an eff ective way to discover new opportunities such as new business segments, best suppliers, associate products of affi nity and
sales seasonality
The evolution of advanced analytical techniques such as machine learning, predictive analytics, data
mining, statistical analysis, arti cial intelligence and natural language processing have enabled
-
Big Data: The Next Big Thing
14
organisations to generate insights across all aspects of their businesses. Organisations are now able
to analyse complete datasets, including unstructured data, instead of smaller samples, resulting in
better outcomes. New visualisation tools and techniques are helping data scientists, and business
users are able to understand Big Data and make decisions based on it. Visual tools for generating
insights have also evolved from simple graphs, PowerPoint presentations and dashboards to heat maps,
cluster analysis and real-time advanced dashboards. Some of the widely used Big Data visualisation
tools are:
Tag cloud: A weighted visual list where words that appear most frequently are larger and words that appear less frequently are smaller
Clustergram: Used to visualise how clusters are formed and how cluster members are assigned to clusters as the number of clusters increases
Heat map: A graphical representation of data where the individual values contained in a matrix are represented as colours
Dashboard: A real-time graphical presentation of data analysis
History ow: Charts the evolution of a document as it is edited by multiple contributing authors
-
Big Data: The Next Big Thing
15
Big Data analytics is the application of advanced techniques on Big Datasets; answer questions previously considered beyond reach
Big Data analytics is an evolving and multifaceted area for analytics players. The key diff erentiating
factors between traditional analytics, advanced analytics and Big Data analytics are:
Big Data analytics diff ers from advanced analytics in terms of diff erent data formats and structures,
and new application requirements for Big Data
While traditional analytics performs rear-view analysis on structured data, advanced analytics and
Big Data analytics provide a progressive view, enabling organisations to anticipate and deal with
future opportunities i.e. Big Data analytics has a de nitive predictive end-result in its use
Big Data analytics has enabled cross-channel analytics and real-time insights at greater speed, access
and collaboration. For example, detection of consumer emotions on a call on mentioning a competitor
or conversion of a service call into an opportunity by leveraging Big Data analytics are more relevant
in real-time rather than after the interaction ends.
-
Big Data: The Next Big Thing
16
Big Data management, analytics, IT services and applications are the key constituents of Big Data ecosystem
The Big Data ecosystem includes multiple elements from the data that is analysed using the IT
infrastructure that supports it and the applications that enable its analysis and usage. Elements of
Big Data include:
Data management refers to systems where the data resides. It comprises the legacy systems as well as Hadoop-based systems and NoSQL databases. Legacy systems include databases that store and
manage structured data, i.e., RDBMS to store and analyse structured data, and MPP systems to scale
up for large structured datasets. Hadoop is an open source software framework to support applications
that enable analysis of petabyte and xetabyte-sized data. Given Hadoops popularity and wide adoption,
several other open-source projects have become associated with it, adding new functionality and
enterprise-ready features to make it a compelling enterprise solution. These sub-projects include
Hadoop Distributed File System (HDFS), Hbase, Hive, Mahout, Pig, ZooKeeper, Avro, Cassandra, and
Chukwa. Once Big Data is collected and processed, it becomes operational data, i.e., it represents Big
Data outcomes or serves as an input data for analytics.
Big Data analytics includes the technologies and tools to analyse the operational data and generate insight from it. After the data is analysed, it becomes available for business users through various
visualisation techniques.
-
Big Data: The Next Big Thing
17
Data consumption involves enabling the Big Data insights to work in Business Intelligence (BI) and end-user applications
IT services enable integration of Big Data framework with the traditional business intelligence infrastructure
-
Big Data: The Next Big Thing
18
Traditional storage architectures limit the potential of Big Data, thereby, compelling businesses to move to new data foundation
The traditional analytics technology stack has evolved into the Big Data analytics technology stack.
The inability of traditional BI applications to process unstructured datasets makes them less relevant
in the Big Data analytics space.
Big Data management, infrastructure and storage systems: Growth in Big Data has led to signi cant infrastructure requirements to support the distributed processing of unstructured data analytics. Unlike
traditional relational databases, which are structured, normalised, and densely populated, Big Data
technology stack mainly comprises Hadoop architecture that has a distributed le system, analytics
and data storage platforms, and an application layer that manages distributed processing, parallel
computation, work ow and con guration management for unstructured data. Other than Hadoop,
there are non-relational databases such as NoSQL databases and MPP systems that are scalable,
network-oriented, semi-structured, and sparsely populated. This layer also comprises servers, networks,
and storage used for scale-out deployment of Big Data technology. With the emergence of Big Data,
traditional RDBMS, MPP and DW are transitioning into a new role of supporting Big Data management
by processing structured datasets as outputs of Hadoop or MapReduce technologies and then input
for BI software and analytical applications.
-
Big Data: The Next Big Thing
19
Big Data analytics: While traditional analytics primarily catered to structured or row/column-based data, Big Data analytics enables analytical processing of multi-structured data for text analytics, predictive
modelling, and social media analytics, using techniques such as MapReduce and in database analytical
functions. Moreover, traditional analytics leveraged basic visualisation techniques such as charts and
graphs to communicate analysis to business users, while Big Data analytics uses new visualisation
tools such as real-time dashboards, heat maps and tag clouds.
-
Big Data: The Next Big Thing
20
Key players across the traditional and Big Data technology stack
As Big Data technologies become mainstream, the vendor landscape is evolving rapidly. Data
management includes vendors of Hadoop-based solutions, other MapReduce technology suppliers
as well as cloud and datacentre providers. The increased demand for Big Data analytics has changed
the competitive landscape for the Big Data analytics service providers. In addition to the incumbent
IT/BPO/Knowledge service players, there are now more pure-play analytics players, some of whom
provide sector-speci c analytics solutions. Some of the larger organisations have set up captives, which
provide data analytics solutions to the other divisions and subsidiaries of those organisations. Even
the breadth of the services provided by analytics organisations has substantially increased from data
storage and management to delivering real-time insights and end-to-end data analytics services.
Big Data management and storage: Many new organisations have emerged as providers of Apache open source Hadoop distributions, with various levels of proprietary customisation for data management.
Cloudera and Hortonworks are the major players for Hadoop distributions. While Cloudera contributes
signi cantly to Apache HBase, the Hadoop-based non-relational database that enables low-latency,
Hortonworks mainly off ers next-generation MapReduce architecture. Other pure players include
MapR, Hadapt and Zettaset. Moreover, mega IT vendors have also entered the Big Data market
through acquisitions. The Big Data warehouse market is mainly led by four players IBM Netezza,
-
Big Data: The Next Big Thing
21
EMC Greenplum, HP Vertica and Teradata Aster Data. Non-Hadoop vendors are also signi cantly
contributing to the Big Data market opportunity Splunk, HPCC Systems and Datastax are some of
the key players.
Big Data analytics: With the deluge of data, it has become pertinent to have applications and platforms that leverage the underlying Hadoop infrastructure for data analytics. Some of the key players in this
segment are: Karmasphere, which off ers an analytical development platform to perform ad-hoc queries
on Hadoop-based data via an SQL interface; Datameer, which provides a Hadoop-based business
intelligence platform that leverages a spreadsheet-like interface to analyse data; and service providers
such as QlikView, Revolution Analytics, Informatica, 1010data, and ClickFox which off er cloud-based
Big Data applications and services.
Big Data use: Big Data analytics engage with large datasets which may be diffi cult to understand for business users. A number of organisations such as Amazon Web Services, Google, and Intellicus are
launching new user applications which facilitate the usage of Big Data analytics.
Additionally, the landscape for Big Data IT services is growing exponentially, with established service
providers such as Oracle, IBM and CSC building their Big Data service portfolio. Moreover, Indian IT/
BPO players such as TCS, Infosys and Wipro are also bolstering their capabilities in Big Data-speci c
software development and implementation.
-
Big Data: The Next Big Thing
22
Big Data enables better customer segmentation, improved productivity and fraud detection across all industry sectors
As organisations adjust to the rapidly changing digital lifestyle of consumers worldwide, they are
beginning to discover the importance of understanding and envisaging the impact of information
generated from non-traditional sources such as blogs, Facebook posts, tweets, emails, smartphone
applications, electronic sensors, images and YouTube videos.
Big Data not only helps organisations gain a multi-dimensional view of their ecosystem, but also
generates powerful insights that can help them better execute their operations and take well-informed
decisions. Big Data is increasingly being leveraged through advanced data analytics tools and techniques
to provide organisations with a better understanding of their customers, competitors, operations,
suppliers and partners. High performance analytics, which previously took days or weeks to perform,
can now be undertaken in seconds, minutes or hours through Big Data technologies.
The public and private sectors are adopting Big Data analytics on a large scale to generate strategic
insights and improve their product/service strategy, operational efficiency and gain a deeper
understanding of their customers, competitors and suppliers. Big Data analytics is enabling them to
predict the trends in near real-time, make more accurate forecasts and adjust their operations quickly
to changing demand or new business opportunities.
-
Big Data: The Next Big Thing
23
Public sector: Big Data can be of immense use in the public/development sectors. It enables government departments and developmental organisations to analyse large amount of data across populations and
to provide better governance and service. Big Data analytics can help them to improve transparency,
enhance decision making, and adopt innovative practices in healthcare, public administration, defence,
disaster management, transportation and energy. For example, Big Data has emerged as a new
focal point for the US Government, which has announced a USD 200 million Big Data Research and
Development Initiative in March 2012.
Financial services: Big Data analytics can enable nancial institutions make better trading and risk decisions, protect themselves from frauds and security threats, and improve their products by
better customer identi cation and marketing campaigns. Further, Big Data analytics is transitioning
investment banks from relying on overnight batch data to make trading decisions. It has improved
the risk decisions by leveraging real-time analysis of current data rather than the risk management
models based on historical data. For example, CITIC Bank Credit Card Center used Big Data technology
to identify customers unlikely to activate their credit card services, and direct marketing incentives
to those most likely to activate, thereby improving the eff ectiveness of the marketing campaign by
65 per cent, while Westpac New Zealand used Big Data technology to analyse social media data to
gain real-time insights into the banks brand health and its product performance across diff erent
geographies by correlating speci c branch performance to customers social data.
Healthcare: The surge in volumes of clinical data on medication, allergies, and procedures owing to the implementation of electronic health records have led healthcare organisations to seek opportunities to
predict and react more rapidly to critical clinical events, resulting in better care for patients and more
eff ective cost management. For example, several of the United States largest integrated delivery
networks such as Cleveland Clinic, MedStar, University Hospitals, St. Joseph Health System, Catholic
Health Partners and Summa Health System use the Big Data platform for real-time exploration,
performance and predictive analytics of clinical data.
Manufacturing: Organisations are increasingly leveraging Big Data and nding new opportunities to predict maintenance problems, enhance manufacturing quality and reduce costs using Big Data.
For example, Volvo leverages Big Data to analyse information received from its vehicles, customer
relationship management systems, product development and design systems, to identify, in advance,
potential issues such as manufacturing and mechanical problems and proactively resolve the problems
by adjusting its manufacturing process.
Telecommunications: Organisations in the telecom industry are increasingly relying on real-time analysis of data generated by mobile devices including phone calls, text messages, applications, and
web browsing for better customer service and to build on retention and loyalty. For instance, while
Nokia collects a huge amount of unstructured data from phones in use, services, log les and other
sources and uses it to gain insights and understand the collective behaviour of consumers to improve
the quality of its phones and their features, Cablecom deploys Big Data analytics to identify when a
-
Big Data: The Next Big Thing
24
customer was most likely to make a decision to leave its network and off ers special deals and incentives
to retain the customer at the right time.
Retail: With large amounts of data being generated from the point-of-sale at stores, online transactions, and social media posts, Big Data off ers numerous opportunities to retailers to improve marketing,
merchandising, operations, supply chain and develop new business models. Retailers are deploying
Big Data analytics to improve the accuracy of forecasts, anticipate changes in demand and react
accordingly. For example, the use of Big Data analytics led to signi cant growth in the number of active
members of Sears loyalty programme (membership crossed 80 million customers).
Other industries: Big Data can also be used in other industries. Data-intensive verticals such as utilities, oil & gas, and transportation, where data is generated through smart meters, GPS systems, and satellites
are gradually using Big Data analytics to make real-time predictions of their operations.
-
Big Data: The Next Big Thing
25
Social gaming, mobile applications, internet search portals are key end-user applications, leveraging Big Data analytics
As adoption of Big Data analytics by enterprises is gaining traction, players are also gearing up towards
mainstream adoption, i.e., B2C applications. Many Big Data players are solving diffi cult problems for
consumers by providing Big Data applications on PCs, smartphones, tablets and other web-enabled
devices. Consumers are using Big Data analytics for everyday chores such as locating vacant parking
spaces more eff ectively, and for real-time comparison of prices. With new applications coming into play
everyday, the B2C market for Big Data is likely to replicate the success of current mobile applications
in the coming years. While innovation is taking place in Big Data technologies, success would be
determined by mass adoption and a large number of businesses getting valuable insights through the
new and compelling end-user applications that allow regular business users or customers to quickly
derive practical and actionable insights.
-
Global Perspective on Big Data
-
Big Data: The Next Big Thing
27
North America drives the Big Data opportunity with over 55 per cent of the worlds data
North America and Europe, the two major data hubs of the world, account for a substantial portion of
the global demand potential for Big Data analytics. Big Data service providers and leading IT players
have signi cantly ramped up their capabilities in these developed regions that embraced the concept
of Big Data, particularly in data-intensive industries such as digital media, manufacturing, healthcare,
retail and nancial services.
While North America and Europe are poised to drive the growth of Big Data for the next 2-3 years,
developing economies such as India and China are expected to catch up soon riding high on the rapid
expansion of multimedia content, increasing popularity of social media and proliferation of mobile
devices. Further, while developed economies are likely to continue to be the major Big Data contributors
in terms of revenue opportunity, emerging economies, particularly India, are all set to emerge as the
preferred Big Data analytics and associated IT service providers.
-
Big Data: The Next Big Thing
28
Global Big Data market is estimated at ~USD 8.0 billion in 2012
Though still in an embryonic stage, with large rms piloting Big Data implementation, the industry is
witnessing exponential growth and market penetration. Statistics suggest that the industry is poised to
grow by more than 50 per cent in 2012 to approximately USD 8.0 billion from USD 5.0 billion in 2011.
Tremendous opportunities have mushroomed for players across the technology spectrum hardware
and software applications providers; systems integrators; technology consultants and analytics
service providers with a large number of organisations implementing Big Data technologies. The
IT-BPO industry is expected to account for about 36-38 per cent of the market opportunity, followed
by applications software at approximately 26-28 per cent.
The market is further expected to experience high penetration rate with investments expanding
beyond the leaders of the Silicon Valley such as eBay, Amazon, Yahoo and Google organisations
that initiated the Big Data revolution, to industry verticals such as manufacturing, nancial services,
healthcare and retail.
-
Big Data: The Next Big Thing
29
Emergence of niche start-ups and technological developments fostering growth in the Big Data industry
-
Big Data: The Next Big Thing
30
New database architectures and innovative analytics tools and techniques to facilitate Big Data implementations
The key stimulus for Big Data implementation is the innovation in database architectures and analytical
tools. Technologies are emerging in the areas of:
Data storage and management (architectures): A number of database architectures and systems such as Hadoop, NoSQL database systems, and MPP systems have emerged, enabling easy storage
and analysis of high volume unstructured data, thus improving scalability and fault tolerance. These
systems perform data management functions much faster through distributed processing and rapid
parallel computations on large clusters of computer nodes.
Data storage, advanced analytics, and data processing: The need for faster data access, storage and analysis has led to the development of in-memory databases such as SAP HANA and Terracottas
BigMemory, which store data in a computers memory, as opposed to disk-based database systems,
thereby enabling faster data processing, low-latency and real-time analytical queries. In-memory
databases particularly help in algorithmic trading, e-Commerce and social media analytics, where
datasets are large and real-time analysis is required. Moreover, analytics tools such as Kognitio, SAP
HANA, and SAS analytics server enable rapid computing and real-time analysis by reducing the response
time, exible and agile analytical environment through massively parallel processing of queries.
-
Big Data: The Next Big Thing
31
Advanced visualisation: Tools and techniques such as tag clouds, real-time dashboards, and heat maps enable representation of multi-dimensional data in enhancing the quality of analysis and insight by
facilitating rapid and accurate observations. Unlike traditional visualisation tools, these new techniques
facilitate integrated display of performance metrics updated in real-time, enabling users to quickly
visualise complex data and get faster insights.
-
Big Data: The Next Big Thing
32
Emergence of niche Big Data start-ups to boost technological innovation
Tools and technologies required to manage and analyse Big Data present a growth opportunity for start-
ups to innovate and come up with new products. New organisations across the Big Data technology
stack have been thriving on the back of some robust investments anticipated in the Big Data space. The
centrepiece of Big Data technology innovation, the Hadoop distribution, has been put to commercial
use by many start-ups such as Cloudera, HortonWorks, Zettaset, and MaPR, with some customisation
of the open source software. Furthermore, the business environment is witnessing a slew of start-ups
in the non-Hadoop systems such as NoSQL, Next Generation (MPP) Data Warehousing like CouchBase,
Splunk and VoltDB. The industry also has many start-ups emerging in the analytics platforms and
cloud-based applications as well as in the advanced data visualisation space. While the past 2-3 years
have mainly seen new organisations coming up in the data management space, analytics applications is
the impetus for growth in the next few years. Some of the start-ups in this eld include Karmasphere,
Kognitio, 1010Data, Revolution Analytics and QlikView.
The Big Data technology space is witnessing a lot of venture capital activity, with funding in Big Data
start-ups reaching ~USD 2.5 billion in 2011, compared with ~USD 1.5 billion in 2010. These start-ups are
innovation hubs that are gaining importance across industry verticals. Most of theseorganisations are
witnessing high double-digit revenue growth driven by the huge demand for their solutions. Moreover,
-
Big Data: The Next Big Thing
33
many start-ups are being acquired by larger IT players given the growth opportunities and the need to
build Big Data capabilities. For instance, IBM has acquired Tealeaf Technologies, Vivisimo and Varicent;
Teradata acquired eCircle, and EMC acquired Greenplum.
-
Big Data: The Next Big Thing
34
Large IT players leveraging M&As to add Big Data capabilities to their service portfolios
The Big Data space is witnessing a string of M&A driven by the need to quickly ramp up capabilities and also to have a complete set of capabilities to service clients who are keen to have Big Data implementation. Leading technology players such as Oracle, IBM, SAP, and EMC are aggressively acquiring smaller Independent Software Vendors (ISVs) and data analytics rms to strengthen their Big Data portfolio.
IBM is in the forefront of this phenomenon through multiple acquisitions over 2010-12 in the Big Data space. It acquired Vivisimo and TeaLeaf Technology in 2012, i2 Limited in 2011 and Coremetrics and Netezza Corporation in 2010, for bolstering its Big Data capabilities. Further, HP acquired Autonomy for more than USD 10 billion, making it the largest deal in the Big Data industry. HP aims to cater to the Big Data market by leveraging Autonomys pattern matching technology that recognises and processes Big Data.
-
Big Data: The Next Big Thing
35
Emergence of cloud-based development and deployment for Big Data solutions
As data is increasingly becoming unstructured, complex and varied, it has become imperative to
process and analyse it in real-time. New data-centric solutions such as Database Platform-as-a-
Service (PaaS), on-demand database service, analytics Software-as-a-Service (SaaS), as well as
on-demand data preparation, storage or enrichment through Data-as-a-Service (DaaS) are now
commercially available.
These Big Data cloud solutions enable traditional enterprises to scale up their data management
and storage at lower costs and provide them real-time insights about the data that could not be
stored before.
While the existing SaaS application service providers are working towards product/service diff erentiation
to ensure that customers derive more value from their applications, new pure-play service providers
are launching Big Data-speci c cloud applications and services. For example, Google, Amazon Web
Services and Microsoft have enhanced their cloud off erings to off er PaaS and analytics SaaS for
Big Data. Leading technology players are launching Big Data cloud solutions in June 2012, CSC launched
its DaaS ClimateEdge, a suite of reports that leverages data from NASA, the National Oceanic and
Atmospheric Administration (NOAA) and other government sources and uses on-demand advanced
analytics to manage climate-related risk and exposure. New players such as 1010Data, and Kognitio
-
Big Data: The Next Big Thing
36
are also off ering their cloud-based Big Data solutions to their customers, enabling them to analyse
Big Data on-demand.
However, the adoption of Big Data through cloud applications may witness a few roadblocks in terms
of data privacy and security concerns. For example, regulations such as Health Insurance Portability
and Accountability Act (HIPAA) Privacy Rules that ensure patient privacy of shared data may inhibit
the adoption of Big Data analytics on-demand.
-
Big Data: The Next Big Thing
37
Potential shortfall of 1.5 million data-savvy managers and ~150,000 data scientists in the US in 2018
The Big Data phenomenon has led to an increasing demand for data scientists professionals
conversant with both the business context and data analytics who play a crucial role in extracting
insights from large datasets, analysing these and then presenting the value-added information to
business users or non-data experts. Big Data needs a new breed of professionals with a deep expertise
in statistics and machine learning, as well as managers and analysts who can leverage insights for
Big Data. The shortage of such talent is a signi cant challenge that organisations need to address
for successful Big Data implementation. According to McKinsey, the US alone faces a shortage of
140,000-190,000 analysts and 1.5 million managers who can analyse Big Data.
To address the shortage, organisations have embarked on initiatives to train their existing employees
and develop new talent. Organisations such as EMC, Oracle and IBM are partnering with universities
to off er courses on various elements of Big Data. Internally, enterprises are creating organisational
cultures that are favourable for data-driven decisions by hiring employees from academic elds such
as statistics, and mathematics, as well as through on-the-job training on emerging technologies in
the Big Data space.
-
Big Data: The Next Big Thing
38
Slow enterprise adoption due to lack of awareness about bene ts of the Big Data
While there is a lot of attention on Big Data and organisations worldwide have started investing in
it, adoption by traditional enterprises has been slower than expected. This is partly due to diffi culties
in understanding the Big Data paradigm and how to integrate it with legacy systems and extract
business value.
Industry studies show that majority of respondents, mainly senior executives from diverse industry
verticals world over, acknowledge that Big Data holds signi cant business opportunities; however, there
is a lack of understanding about how data can be used to drive businesses forward. Further, ensuring
that investing in Big Data implementation would achieve a high RoI is also a major concern. Given the
gap in understanding the bene ts and opportunities of Big Data, many enterprises are less inclined
to give it high priority for immediate investments. However, the market appears receptive as most of
the leading organisations across industry verticals are willing to integrate Big Data into their existing
systems, and are engaging in pilot projects to examine their success.
The value off ered by Big Data is not currently out of doubt as there are skeptics who are still questioning
if it is worth all the investments being poured into it. This is in part due to the lack of abundant and
well-publicised business cases on successful implementation and the bene ts accrued. Therefore, as
executives lack an understanding, and in some cases the sponsorship of Big Data, IT organisations
may witness additional complexities in terms of budget and bandwidth constraints in the process of
implementing Big Data.
-
Big Data: The Next Big Thing
39
Data related regulations like Dodd-Frank and Basel III to impact Big Data implementations
An increasing number of regulations are driving organisations to source, analyse and report large
amount of data. Regulations such as Dodd-Frank, Basel III and HITECH mandate more transparency
and real-time reporting for data collected from multiple systems/sources, their aggregation, analysis
and storage. Consequently, organisations in various industry verticals are leveraging Big Data analytics
to comply and provide more transparency. This has prompted data management, storage and analysis
to be more comprehensive and real-time.
While regulations in industry verticals are driving Big Data adoption, regulations such as the EU Data
Protection Directive may impact adoption of Big Data analytics, particularly in cloud-based delivery
models. Further, with businesses collecting and storing large amount of customer data, privacy-related
concerns have also increased. Some countries have already enacted legislations to protect the privacy
of individuals and many are in diff erent stages of formulating them. Therefore, businesses will also
have to consider certain regulatory aspects as they move towards leveraging Big Data analytics using
stored customer data.
-
Indias Advantage in the Big Data Opportunity
-
Big Data: The Next Big Thing
41
Indias Big Data market opportunity estimated at ~USD 200 million in 2012
India is rising to play an important role as a key outsourcing destination in the overall global Big Data
landscape for services relating to Big Data technology implementation and analytics, capitalising on its
already well-established IT/BPO and knowledge service outsourcing industry, which off ers signi cant
cost and intellectual arbitrage to global multinationals.
Indias domestic demand for Big Data analytics is at a nascent stage since most Indian organisations
still consider Big Data as a mere hype. The opportunity for Indian service providers arises from
off ering Big Data technology implementation and analytics outsourcing services, which is growing
robustly. In 2011, Indias Big Data outsourcing opportunity was estimated by CRISIL GR&A to be around
USD 90 million and is projected to grow by ~110-115 per cent in 2012 to USD 200-205 million. The IT
services segment, which primarily comprises the Big Data technology implementation, including data
collection, integration, and designing of Big Data architecture and data analytical tools, is expected
to account for 82-84 per cent of this growth projection, while the Big Data analytics services is likely
to account for 16-18 per cent.
Although immense amount of data is being generated across all industry verticals including nancial
services, manufacturing, retail, healthcare, telecom, logistics, and others, nancial services and telecom
are early adopters of the Big Data technologies.
-
Big Data: The Next Big Thing
42
Key factors that are pushing organisations to adopt Big Data analytics include large volumes of data
being generated across global organisations as a result of the increasing use of Internet, mobile, social
media marketing, as well as Machine-to-Machine (M2M) conversations that need to utilise this data
to derive meaningful insights to help organisations make well-informed decisions.
-
Big Data: The Next Big Thing
43
Global In-house centres, pure-play analytics rms and IT/BPO players expected to bene t from the Big Data opportunity
The Big Data outsourcing market, though still at an embryonic stage, is being tapped aggressively by
the global in-house centres (captive centres of multinationals) as well as the Indian service providers
comprising IT/BPO players, pure-play analytics rms and knowledge service providers.
Global In-house Centres: Global multinationals have set up these centres across India to off er support on various back-end processes such as accounting, HR, and payroll as well as to off er
an off shore base for knowledge services such as business research, nancial research, data
management and analytics and legal services. With growing interest in Big Data, organisations
are leveraging their already established in-house centres for Big Data technology implementation
as well as to handle large volumes of unstructured data to provide business intelligence and data
analytics solutions.
Global in-house centres have been successfully leveraged to unleash the power of Big Data as
they enable seamless sharing of data given that they are a business unit/division of the parent
organisation. This is because there are no data security/privacy issues and there is a high level of
data integration with the parent. Further, the management enjoys tighter control over the data and
applies analytics closely related to business needs given that these centres have built-in domain
knowledge. Some of the key players who have set up in-house centres to deliver Big Data analytics
-
Big Data: The Next Big Thing
44
to their parent organisation are:
- Retailers such as Sears Holdings and Walmart
- IT/technology service providers such as Google, Yahoo, HP, SAP, Oracle, IBM and Dell
- Financial service organisations such as JPMorgan Chase, Merrill Lynch, HSBC, American Express,
Goldman Sachs, Barclays, Bank of America, Citigroup and Wells Fargo
Pure-play Analytics Players: These primarily comprise Indian as well as global pure-play analytics rms as well as major knowledge service outsourcing providers who off er analytics and are now
establishing their presence in the Big Data analytics eld. Key pure-play analytics rms operating in
the industry are: Bridge i2i, Nuevora, MuSigma, Cognilytics, Fractal and AbsolutData. Key knowledge
services outsourcing players such as CRISIL GR&A, Ugam Solutions, and SmartCube are increasingly
taking interest in expanding their analytics capabilities to harness the potential of Big Data. These
service providers enjoy strong subject matter expertise, leverage the best practices in the industry
to off er analytics services, and off er optimum priced services, given the economies of scale coming
from serving various clients with Big Data analytics. These players face key challenges such as low
levels of data integration with the clients, intellectual property and data security.
Integrated IT/BPO Providers: Several integrated IT/BPO players engaged in application development & management, and infrastructure management as well as BPO players providing outsourcing
services for back-end functions have also entered the Big Data market and are moving from
simpler business process services to providing Big Data implementation, tools, and technologies.
To strengthen their presence in Big Data, these players leverage their global presence and existing
multinational client base looking at Big Data implementation as well as utilise their strong
technology orientation to provide Big Data tools and technologies. This business model mainly
comprises two categories of players:
- IT-BPO providers such as Infosys, TCS, Wipro, and HCL. TCS and Infosys are helping their global
multinational clients in designing and implementing Big Data technology
- Key BPO vendors such as Genpact, EXL, and WNS
-
Big Data: The Next Big Thing
45
Pure-play providers and integrated IT service providers are active in providing services in the Big Data environment
-
Big Data: The Next Big Thing
46
Global in-house centres to be the front-runners in Big Data servicing; but IT/Analytics players follow closely
Big Data analytics came into play globally in late-2011. In 2011, many multinationals were skeptical about Big Data implementation and trying to quantify the Return on Investment (RoI) to build a
case for Big Data implementation. The early adopters of Big Data analytics have tried to leverage
their in-house global centres in India, given the talent shortage in the developed world, to generate
meaningful insights from Big Data. The ease of seamlessly sharing data and information also prompted
multinationals to leverage their analytics and knowledge centres in India to conduct Big Data analytics.
Global multinationals across verticals such as nancial services, retail, technology, and healthcare have
started leveraging their Indian centres for Big Data implementation and analytics.
In 2012-13, the success of global in-house centres in the Big Data market is expected to catapult the emergence of a hybrid service model in which the in-house centres of global organisations would off er
analytical services to external clients in addition to their internal business units. Further, pure-play
analytics rms present in India are increasingly deploying advanced analytical tools and techniques on
Big Data sets to gain signi cant business traction as more and more Big Data business opportunities
move to India. Integrated IT/BPO service providers are building Big Data architecture and off ering
analytics services to their clients.
-
Big Data: The Next Big Thing
47
Some of the key initiatives taken by Indian service providers and global multinationals are:
In 2012, Sears Holdings, the fourth largest retailer in the US, created a wholly-owned
subsidiary, MetaScale, to target and sell its managed Hadoop services (or Big Data services) to
customers with revenue of between USD 1.0 million and USD 10.0 million across healthcare and
entertainment verticals
Walmart expanded its e-Commerce operations to India by opening a @Walmartlabs facility in
Bengaluru, India, in April 2012, to develop social media analytics and Big Data infrastructure
In July 2012, Yahoo also set up a Grid Computing Lab at the IIT-Madras campus in partnership with
the institute to enable researchers to access web-scale data and conduct research on Big Data
issues such as search, personalisation and digital advertising
Infosys aggressively focuses on off ering major enablers for Big Data analytics adoption including
solutions, services, and expertise across key industry verticals such as financial services,
manufacturing, healthcare, and telecom
In 2012, TCS won Big Data contracts to deliver next-gen insights using Big Data frameworks for a
global airline, a US-based bank and a global market research rm as well as to set up a leading-
edge distributed data warehouse for a hi-tech rm using Big Data
BPO service providers such as Genpact and IBM Daksh are also being seen as strong contenders in
the analytics domain and are well poised to capitalise on the Big Data trend
The Big Leap in Big Data is expected to come by 2014 when the stage of testing waters would have been successfully crossed and Big Data pilot projects would have delivered pro table results or expected
ROI for clients. Once the multinational organisations realise the potential opportunity off ered by
Big Data analytics, more and more organisations are expected to undertake Big Data implementation in
a big way to strengthen their business and enhance pro tability. All the players are expected to expand
their operations to tap the growth in the market. Hence, the industry is expected to witness:
The emergence of several new Big Data analytics rms to cash in on the growing Big Data opportunity.
Further, these analytics rms and knowledge service players are expected to play a dominant role
in the Big Data analytics space
Integrated IT services providers who are likely to off er services across the Big Data value chain from
implementation, consulting to analytical services
Global in-house centres are likely to continue to grow, and more and more multinational organisations
are expected to leverage this business model and set up/expand their in-house centres for
Big Data implementation
-
Big Data: The Next Big Thing
48
Service providers are leveraging partnerships, M&As and venture funding to capture the Big Data outsourcing opportunity in India
Major services providers across the country are undertaking several strategic initiatives to capitalise
on the Big Data outsourcing opportunity. The industry is witnessing an increasing thrust on leveraging
venture capital funding; collaboration for developing Big Data technologies and joint go-to-market;
mergers and acquisitions to enhance capability across Big Data software and services as well as
expanding overseas presence to capture the market.
Venture Capital (VC) funding: In the recent months, venture and growth capital rms have invested huge amounts in Big Data organisations, primarily to enable these rms to strengthen
their operations
Partnerships with foreign players: Big Data service providers are entering into technology partnerships and collaborations to expand their capabilities to serve new markets and
industry verticals
- In August 2012, Intel built partnerships with India-based Independent Software Vendors (ISVs)
across various business segments such as nancial services, manufacturing, education, retail,
telecom, and healthcare to foster its presence in the Big Data ecosystem in India
-
Big Data: The Next Big Thing
49
- In July 2012, BPO players such as Infosys BPO announced plans to look for partners in the
Big Data analytics eld to strengthen its capabilities
Strategic M&A to gain Big Data capabilities: The hype in the industry has led to the mushrooming of various smaller players off ering Big Data services such as application development, system
integration, consulting, storage and architecture design. Established integrated IT/BPO service
providers and pure-play analytics rms are aggressively acquiring niche players to broaden
their capabilities
- In June 2012, Wipro acquired Australia-based Promax Applications Group, a specialised trade
promotion management rm, for USD 36.6 million to reinforce its presence in the Australian
market and strengthen its capabilities in Big Data analytics solutions
Geographic expansion: Indian organisations are also looking to expand their overseas presence to market their Big Data capabilities and capture the market opportunity
Strengthening workforce: Various organisations are planning to collaborate with the academia to train and certify data scientists to counter the impending shortage of data scientists, analysts,
and managers that is likely to challenge the Big Data market growth
- In August 2012, Intel announced plans to collaborate with educational institutions to bring
innovation in data analytics and research, and has tied up with ~300 colleges and universities
in India including the IITs and other educational institutes such as Pune University to foster
research and innovation in Big Data analytics
-
Big Data: The Next Big Thing
50
India has an early mover advantage vis--vis other geographies in creating a strong base of Big Data workforce
India is expected to be a forerunner in Big Data talent supply, not as a cheaper alternative but as a
go-to-destination for the quality of talent in the country. India churns out more than 2.5 million university
graduates and about 750,000 post graduates every year, of which ~700,000 students are graduates
in Mathematics and Science and ~300,000 are post graduates in these elds. With its repower of
intellectual pool in Mathematics and Science, India is all geared up for the Big Data revolution. Further,
with the ever-increasing number of students having domain expertise in decision sciences, India is
well-positioned to address the global demand for Big Data solutions.
With India already catering to the business analytics needs of global multinationals at the best possible
performance-to-cost ratio, the country has a huge potential to supply data scientists for the Big Data
industry. Tier I cities such as NCR (Delhi, Gurgaon, and NOIDA), Bengaluru, and Mumbai have emerged
as good breeding grounds in India for global organisations to set up their analytics centres of excellence
and they account for more than two-thirds of the analytics professionals in India. Further, more than
60 per cent of the analytical workforce in India has a work experience of 3-10 years, which is a boon to
Big Data analytics. These professionals have the ability to apply advanced analytics and can be trained
internally by organisations to work on Big Data.
-
Big Data: The Next Big Thing
51
Indian academia is also aggressively developing capabilities to match the ever-growing demand for and
dearth in supply of data scientists with analytical training through solemn intervention at the education
level and imparting training on analytical and statistical tools. Premier colleges/universities in India
already have courses in place to impart training in analytics. Key analytics courses in India include:
Business Analytics and Intelligence (BAI) IIM Bengaluru: An executive course, BAI requires at least ve years of work experience and is suitable for professionals who are already working in
analytics to enhance their knowledge as well as for those with an analytical aptitude
Executive Programme in Business Analytics IIM Calcutta: This is a one-year distance programme off ered in association with Hughes Education, and covers topics such as data mining, soft computing,
design of experiments, survey sampling, statistical inference, investment management, nancial
modelling, and advanced marketing research
Advanced Certi cate Programme in Business Analytics IIT Bombay: Designed in partnership with HughesNet Global Education, it is a part-time course for analysts to develop the skills and
competencies of key analytics techniques such as behaviour and data modelling
Business Analytic & Data Mining Indian Statistical Institute ISI Pune: Designed to guide business analytics professionals in analysing large quantities of data to study unknown interesting
patterns through cluster analysis, dependencies (association rule mining), classi cation of data,
and predictive analytics
Post Graduate Certificate in Research and Analytics MICA Ahmedabad: This is a one-year programme based on practical and non-technical approach through various data
analysis software
Indian universities continue to introduce courses in statistics and data analytics to produce graduates to
meet the manpower shortage in the global Big Data market. Recent academia initiatives for developing
the talent pool for Big Data analytics include:
In August 2012, Academy of Decision Science and Analytics started off ering an e-learning Post
Graduate Programme (PGP) course in data analytics in association with Ivory Education
In July 2012, The Institute of Management Technology (IMT), Ghaziabad, signed an MoU with
Genpact to develop and implement analytics elective for the two-year post graduate diploma in
management programme to provide both theoretical and practical work experience in analytics as
applied in diff erent industries
- Pankaj Kulshreshtha, Senior Vice President and business leader Smart Decision Services
Analytics and Research, Genpact, stated, The emergence of big data, regulatory changes and social media are causing a big shift in the way businesses operate and students of IMT
will learn how to combine process, analytics and technology to make organisations smarter in
this dynamic new world. It is also a great example of two organisations, both leaders in their
respective elds working together to build talent in an area which is expected to more than
double in the next 2-3 years in India.
-
Big Data: The Next Big Thing
52
In May 2012, IIMLucknow partnered with the US-based Kelley School of Business to provide two
certi cate programmes in business analytics and global strategy
- Dan Smith, Dean of the Kelley School, said, Our collaborative goal is to fundamentally advance
the quality of decision making by business leaders by improving their ability to draw meaningful
insights from the massive amounts of data available to them today.
In November 2011, Indian School of Business (ISB) Hyderabad launched Asia Analytics Lab for
its students, which is a focal point for data analytics initiatives, education, research and business
applications in the Asian context
In 2011, the Indian Institute of Science (IISC) Bengaluru launched Master of Management, a
two-year course to focus on training students in Technology Management and Business
Analytics
Indian service providers are also making large investments and innovation in creating and grooming
a new breed of talent. For example, IBM has partnered with 500 universities in India to help more
than 30,000 students develop skills in predictive analytics. India is at an advantage vis--vis other
geographies, as apart from the ample number of graduates it produces each year, organisations in
India are also making huge investments in breeding and grooming such talent. Further, India retains
advantages due to demographic factors, and the fact that the education system is producing a huge
pool of analytical talent.
-
Big Data: The Next Big Thing
53
Indian service providers off ering Big Data solutions across verticals
-
Big Data: The Next Big Thing
54
1. Manufacturing: Indian service providers enable manufacturing organisations to analyse large datasets for eff ective decision making
The manufacturing sector generates large volumes of text, image and numerical data in its production
processes, R&D and engineering functions. The sector generates data from a multitude of sources,
including instrumented production machinery (process control), supply chain management systems,
and performance monitoring systems.
Large volumes of datasets thus aggregated are then subjected to diff erent Big Data analytical tools
and techniques to generate useful insights across the value chain. Hence, Big Data nds application
across R&D, product design, supply chain management, production, marketing and sales, and
after-sales service.
R&D and product design: The use of Big Data in the R&D processes off ers opportunities to accelerate product development, help designers focus on product features based on concrete customer inputs
as well as use designs that minimise production costs
- Aggregate customer data and make them available to improve service and enable
design-to-value
- Source and share data through virtual collaboration sites (idea marketplaces to enable
crowd sourcing)
-
Big Data: The Next Big Thing
55
- Build consistent interoperable, cross-functional R&D and product design databases to enable
rapid experimentation, simulation, and co-creation
Procurement: Manufacturing rms use Big Data analytics during procurement process to drive effi ciency in their supply chain and improve demand forecasting processes. Manufacturers deploy
Big Data analytics to
- Gather sales, customer feedback, and demand patterns from distributors/retailers to rectify
any deviation in real-time, thereby improving the supply chain responsiveness
- Conduct a path analysis to design ways to move a product more eff ectively from the factory
to the customer
- Automate stock optimisation and replenishment decisions based on the analysis of
inventory-related data trends
Production: The deployment of the Internet of Things or actuators and sensors also allows manufacturers to leverage real-time data from sensors to track parts, monitor machinery, and
guide actual operations. At the production stage, Big Data analytics is used in
- Digital factory simulations: Manufacturers take inputs from product development and historical production data and apply advanced computational methods to create a digital model of the
production process and thus design optimal production layouts and digital shop oor control
and improved fault detection
- Sensor-based operations: Firms leverage Big Data analytics on the volumes of real-time, highly granular data gathered from the sensors deployed across production lines to forecast operational
costs, schedule predictive systems maintenance, monitor labour and equipment performance,
and improved fault detection by identifying patterns that lead to potential equipment failure
Sales & Distribution: Manufacturing organisations track customer-related transaction data to generate actionable insights on the customer buying patterns and behaviour, strengthen their
marketing and sales strategies and make informed product decisions. Analytics can be applied on
this data to
- Ensure improved customer segmentation and better customer relationship management
- Improve product inventory tracking
- Enhance the eff ectiveness of the sales force and marketing campaigns
After-Sales Service: Warranty analytics as well as real-time analysis of sales and feedback data are the key applications being leveraged by manufacturing rms, which are based on Big Data analytics.
These applications primarily involve analysing large volumes of warranty claims to improve product
development with the aim of improving product quality and reducing warranty costs. Further,
after-sales and feedback data can help enhance after-sales service as well as detect and rectify
manufacturing and design errors to enhance customer satisfaction
-
Big Data: The Next Big Thing
56
Some of the key bene ts delivered by Big Data analytics for the manufacturing sector include:
Product demand forecasting and supply planning: Using real-time data from sales and demand patterns or from customer feedback and purchasing behaviours, manufacturers can rectify any
deviation in real-time, engage in eff ective demand forecasting, adjust production levels and increase
the frequency of planning supply cycles to match with the production cycles
Improved collaborative engineering through crowdsourcing: Leverages crowdsourcing to collect product-/market-related data to enable collaborative engineering that results in innovative design
from customers. For example, auto manufacturing organisations encourage ideas from consumers
to make improvements to new car models. Big Data analytics enables these organisations to gather
and analyse data from tweets, blogs and other social media platforms eff ectively to off er innovative
features in newer versions of the vehicles
Mass customisation: By enabling design-to-value, Big Data analytics allows manufacturers to leverage quantitative customer insights mined from sources such as PoS, customer feedback from
retail surveys, and social media platform, and improve their output quantities as well as facilitate
mass customisation
Effi cient planning and operations: Big Data aids in designing, simulating and testing product or factory plans in a virtual manner, before the actual production or construction. Further, it is used
to predict equipment failures and system replacements to better anticipate any roadblocks in the
manufacturing processes.
To capitalise on this huge opportunity, various Indian Big Data service providers such as Infosys, Intel,
Fractal, and Wipro have built capabilities to win new clients as well as to better serve the existing ones
in the manufacturing sector.
In 2012, Infosys was selected as the sole sourced partner for cloud strategy and Big Data infrastructure
for a North American manufacturer, to devise a Big Data strategy and roadmap
In August 2012, Intel announced the signing of partnerships with India-based ISVs across various
business segments including manufacturing, and others to build Big Data analytics capabilities
across India
-
Big Data: The Next Big Thing
57
Case examples: Indian service providers serving global manufacturers on custom designed Big Data implementations and analytics
-
Big Data: The Next Big Thing
58
2. Retail: Indian service providers help retailers understand customer buying patterns and maintain optimal stock levels
Retailers generate Big Data through various sources such as social media, Point of sale (PoS) and web/
online sales platform (credit cards and rewards cards, purchases), consumer surveys, loyalty programme
pro les, in-store tools and footfalls. This customer-focused data can be used to gain signi cant and
meaningful insights into consumer behaviour, their buying patterns, and changing preferences.
Big Data analytics helps both online as well as brick and mortar retailers to improve their decision making,
manage the supply chain, inventory levels, merchandising and pricing, enhance focus on customer
segmentation and hence introduce targeted products/services as well as marketing/promotional
campaigns. Further, Big Data allows retailers to enhance their margins and productivity by enabling
them to perform real-time analysis of customer response to pricing/product changes/productivity and
re ne their strategies based on such analysis.
Some of the important areas within the retail industry where Big Data analytics is being used are:
Supply chain and procurement: Retailers use Big Data analytics to help them better manage their and their suppliers inventory levels, relationships with suppliers, and make informed decisions on
stock levels. For example, Barnes & Noble deployed Big Data analytics solution from IBM to enable
suppliers to monitor its inventory and take appropriate replenishment decisions. Big Data enables
retailers to
-
Big Data: The Next Big Thing
59
- Improve inventory management, stocking decisions and stock forecasting by combining multiple
datasets such as sales history, weather predictions and seasonal sales cycles
- Optimise transportation and vehicle routing by using GPS-enabled Big Data telematics to
improve eet and distribution management, enhance productivity by rationalizing fuel effi ciency,
preventive maintenance, driver behaviour, and vehicle routing
- Base their supplier negotiations for price discounts, and change in raw material preferences by
analysing customer preferences and buying behaviour data
Merchandising: Big Data implementation and analytics on the POS and RFID data can help retailers to easily strengthen their merchandising-oriented decisions such as
- Assortment optimisation: Retailers make product assortment decisions in stores based on the demographic and purchasing pattern data
- Price optimisation: Retail rms can leverage advanced demand-elasticity models on the pricing and sales data available for deciding the optimum pricing of products and services
- Placement and design optimisation: Brick and mortar retailers optimise the placement of goods and visual designs of their store layout by mining sales data at the SKU level and even
foot-traffi c data and online retailers adjust website placements based on data on page interaction
such as website traffi c, scrolling, clicks, and mouse-overs
Operations: To create operational value and efficiency, retail firms are deploying Big Data implementation to
- Ensure performance transparency by analysing store sales, SKU sales, and sales per
employee data
- Reduce costs while maintaining service levels by leveraging the labour input, time and attendance
data, and tracking labour scheduling information
Sales and marketing: It is the most common business function for which retail rms use Big Data analytics. Key sales and marketing functions where Big Data implementation nds use are:
- Use customers demographics, purchase history, preferences, and real-time location data for
cross-selling and up-selling of goods
- Undertake location-based marketing for off ering promotional discounts, and special off ers,
primarily leveraging the personal data generated by smartphones
- Enable customer micro-segmentation to deliver personalisation of products/services
to customers based on traditional market research data as well as data available from
behavioural tracking
-
Big Data: The Next Big Thing
60
- Use sentiment analysis that leverages consumer data generated by social media platforms
to make informed business decisions such as assessing the real-time response to
marketing campaigns
- Study in-store consumer behaviour to improve store layout, product mix, and shelf positioning
by tracking shopping patterns, real-time location data from smartphone applications, and
shopping cart transponders
Customer services: By applying Big Data analytics on customer behaviour, which can be tracked through service centres (IVR and call centres), social media platforms; retailers can improve their
interaction with customers for better service delivery
Big Data analytics has found signi cant acceptance in the retail sector, especially among the leading
players. Walmart acquired social media rm Kosmix to create WalmartLabs and is using this specialist
R&D unit to redesign its business by merging social, mobile and retail data, to understand consumers
buying habits. Further, in April 2012, Walmart expanded its e-Commerce operations to India and
opened the @Walmartlabs facility in Bengaluru, India, to develop social media analytics and Big
Data infrastructure. Other retailers such as Sears utilise their in-house IT/technology centres in India
to provide Big Data analytics to set product prices in real-time and move inventories. It also has a
subsidiary, Metascale, which helps other organisations in industries such as energy and healthcare,
implementing Hadoop.
Big Data-driven analytics hold much potential for retailers in the realm of customer intelligence.
These include:
The ability to pro le and segment customers based on socioeconomic characteristics can allow
rms to market to diff erent segments based on their discrete preferences and hence generate
better customer retention rates
Online social network analysis enables businesses to monitor consumer sentiments towards their
brands, react to trends as they develop, and identify in uential individuals within networks for
direct marketing
Using Big Data to construct predictive models for customer behaviour and purchase patterns
facilitates the accurate appraisal of each Customers Lifetime Value (CLV) to a rm, allowing
resource allocation towards acquiring and retaining profitable clients, thereby raising the
overall pro tability
-
Big Data: The Next Big Thing
61
Sears is leveraging Big Data analytics to turn itself around, and is also keen on off ering analytics services to external clients
-
Big Data: The Next Big Thing
62
3. Financial services: Witnessing increased adoption of Big Data analytics, to reduce risk and uncover new market opportunities
Financial services is considered to be a very data-intensive sector, with more data per million of revenue/
operating expenditure or per employee, than almost all other sectors. Within the sector, structured
and unstructured data is available from a variety of sources such as customer and transaction data
from various channels such as branch, kiosks, mobile and web; social media; emails; credit cards
data; insurance claims data; stock market data; statistical data, PDF & excel les, news, videos, and
government lings.
With the industry facing a multitude of challenges such as higher customer expectations, uncertain
operating environment, strict regulations, stiff competition, and slowing economic growth, Big Data
analytics can help banks, capital markets and insurance organisations by providing tools to reduce
costs and improve productivity. Increasing regulatory compliances and the need for collecting every
piece of data and standardising them is driving the growth of Big Data analytics. Several areas within
the nancial services sector are expected to gain from Big Data technologies. They include:
Banking
Credit reward programme analysis: Banks are increasingly using unstructured data to understand customer pro le and introduce successful credit cards with innovative rewards programme
- For example A national bank used a Big Data solution to analyse data from sources such as call
centres, customer service emails, and social media conversations to create a credit card off ering
-
Big Data: The Next Big Thing
63
with a rewards programme to attract a young, professional demographic. This helped in providing
information to the marketing department to create a targeted promotion campaign, including
strategically placed social messaging and monitoring
Capital Markets
Trading surveillance: The nancial sector leverages Big Data to monitor trading activities and identify abnormal trading patterns. In surveillance, Big Data analytics allow online access to trade-by-trade
history for investigation, trending, and discovery to be combined with real-time data to provide a
real-time and historical context to behaviour
- For example Organisations combine data about the parties that participate in a trade with the
complex data that describes relationships among those parties and how they interact with one
another. The combination allows the bank to recognise unusual trading activity and to ag it
for review
Insurance
Insurance organisations are increasingly using unstructured data to predict client longevity, along
with examining the prospective clients medical status by analysing their general comments, visits to
particular websites, and enquiry about some speci c products.
Using weather and calamity information for managing claims exposures and losses based on
unstructured data from weather measurements, and soil observations.
- E.g. An insurance organisation sells Total Weather Insurance, which pays local farmers
when they are impacted by weather events that aff ect their pro ts. The organisation uses a
cloud-driven Big Data analytics service to predict the possibility of extreme weather, along
with the potential impact. It prices its insurance policies accordingly, based on 2.5 million daily
weather measurements, 150 billion soil observations, and 10 trillion scenario data points to
build and price their products
Big Data is being extensively used across all domains of the nancial services for risk management,
fraud detection, compliance and customer relationship management:
Risk management: Predictive modeling of customer behaviour and scoring techniques enable nancial sector organisations to access and minimise default risks at an individual level and make
customised off erings, in line with the customers risk pro le
- E.g. A large bank wanted to use 12 years of monthly account-level credit card data, credit
bureau information and bank account information to better assess the risk before granting
loans or raising credit limits. Ideally, it wanted this information in real time. To speed the
computing, it used an in-database Big Data approach, which helped the bank to calculate risk
70 times faster
-
Big Data: The Next Big Thing
64
Fraud detection: Big Data technologies give nancial services organisations the ability to run exploratory modelling and discovery on data, thereby increasing the accuracy of fraud detection
models. The faster processing capability enables organisations to quickly build or refresh fraud
detection models, and also helps in detecting fraud in real-time by analysing and streaming
transaction data
Compliance and regulatory reporting: Increased oversight and scrutiny of the organisations operations, funding and investment portfolio has led nancial services organisations to adopt
sophisticated Big Data technologies to store and process vast amount of data to simplify and
streamline their regulatory and compliance reporting
- For example Reserve Bank of India (RBI) has directed all Indian banks to standardise their
regulatory reporting by following an Automated Data Flow (ADF) approach to ensure
100 per cent accuracy and zero human intervention in every stage of reporting: right from data
extraction from source systems to the actual submission of returns. Firms that could not utilise
complete information and rms that believed reporting did not really require management
attention are increasingly focusing on Big Data analytics
Customer relation management: Big Data analytics also helps nancial service organisations in acquiring new customers and cross-selling their off erings to existing customers by using
Big Data to identify the most pro table customers and run eff ective marketing campaigns. The
large volume of unstructured data from social media is combined with the CRM systems to
study customer behaviour and optimise custo