BIG DATA CHALLENGESAND HOW TO OVERCOME THEM

7

PRESENTED BY:

It’s easy to get caught up in the hype and opportunity of big data.

However, one of the reasons big data is so underutilized is because big data and big data technologies also present many challenges.

One survey found that55% of big data projects are never completed.

So what’s the problem with big data?

55%

7 CHALLENGES:5. Data Quality 6. Security7. Cost Management

1. Hadoop is Hard2. Scalability3. Lack of Talent4. Actionable Insights

While Hadoop and the surrounding ecosystem of tools is lauded for its ability to handle massive volumes of structured and unstructured data, the software isn’t easy to manage or use.

1HADOOP IS HARD

Hadoop frequently requires extensive internal resources to maintain, and many businesses are left devoting most of their resources to the technology rather than to the actual

big data problem they are trying to solve.

73% of Hadoop users claimed understanding the big data platform was the most

significant challenge of a big data project.

73%

Many organizations fail to take into account how quickly a big data project can grow and evolve.

2SCALABILITY

Big data workloads also tend to be bursty, making it difficult to allocate capacity for resources.

To successfully implement a big data project requires a sophisticated team of developers, data scientists and analysts who also have a sufficient amount of domain knowledge to identify valuable insights.

3LACK OF TALENT

Many big data vendors seek to overcome this challenge by providing educational resources or by providing more

automation of the platform management.

A key challenge for data science teams is to identify a clear business objective and the appropriate data sources to collect and analyze to meet that objective.

4ACTIONABLE INSIGHTS

Once key patterns have been identified, businesses must be prepared to act and make necessary changes in order

to derive business value from them.

Dirty data costs companies in the United States $600 billion every year.

5DATA QUALITY

Common causes of dirty data include

1. User Input Errors2. Duplicate Data

3. Incorrect Data Linking

1 2 3

Specific challenges include:1. User authentication for every team and team member accessing the data2. Restricting access based on a user’s need3. Recording data access histories and meeting other compliance regulations4. Proper use of encryption on data in-transit and at rest.

6SECURITY

The challenge lies in taking into account all costs of the project.

7COST MANAGEMENT

Businesses pursuing on-premises projects must remember the cost of training, maintenance and expansion.

Big data in the cloud projects must carefully evaluate the service-level agreement with

the provider to determine how usage will be billed and if there will be any additional fees.

$

While the number of big data challenges can be overwhelming, it also presents an opportunity. Those businesses who are able to identify the right infrastructure for their big data project and follow best practices for implementation will see a significant competitive advantage.

Ready to learn how you can be successful with big data in the cloud?Download the big data in the cloud success sheet to learn

implementation best practices and hangups to avoid.

Download Success Sheet