2018 TREND REPORT: Enterprise AI Adoption
Transcript of 2018 TREND REPORT: Enterprise AI Adoption
AI PROMISE RUNS INTO ENTERPRISE REALITY 1 of 10
eBook
2018 TREND REPORT: Enterprise AI AdoptionHow today’s largest companies are overcoming the top challenges of AI.
SPONSORED CONTENT
2018 TREND REPORT: ENTERPRISE AI ADOPTION 2 of 10
Executive SummaryEnterprises are making significant investments in artificial intelligence (AI) technology as they attempt to retool
their business and create competitive advantage. This CIO survey of global data science and engineering leaders
across multiple industries found that almost 90% of them are making significant AI investments, but very few are
realizing the full benefits of their investments.
Only 1 in 3 AI projects are successful and it takes more than 6 months to go from concept to production, with a
significant portion of them never making it to production — creating an AI dilemma for organizations.
The very thing that makes AI possible is also making it challenging to implement: data. About 96% of organizations
say data-related challenges are the most common obstacle when moving AI projects to production. Enterprise
data is not AI-enabled and is siloed across hundreds of systems such as data warehouses, data lakes, databases
and file systems. And machine learning (ML) frameworks such as TensorFlow and others don’t do data processing.
Since data systems don’t “do AI” and these AI technologies don’t “do data,” organizations end up using on
average 7 disparate tools which create friction and slow down projects. To make matters worse, the survey found
that 80% of them face collaboration challenges as data science and engineering teams are in organizational silos.
So, what will help these organizations conquer the AI dilemma? According to the survey, 90% of the respondents
believe that unified analytics — the approach of unifying data processing with ML frameworks and facilitating
data science and engineering collaboration across the ML lifecycle, will conquer the AI dilemma. Unified Analytics
is a new category of solutions that unify data science and engineering, making AI much more achievable for
organizations. Unified Analytics makes it easier for data engineers to build data pipelines across siloed systems
and prepare labeled datasets for model building while enabling data scientists to explore and visualize data and
build models collaboratively.
The Databricks Unified Analytics Platform, from the original creators of Apache Spark™, unifies data science
and engineering across the ML lifecycle from data preparation to experimentation and deployment of ML
applications — enabling companies to accelerate innovation with AI. The Databricks platform provides one
engine to prepare high quality data at massive scale and iteratively train ML models on the same data while
leveraging all the popular open source frameworks. It also provides collaboration capabilities for data scientists
and engineers to work effectively across the entire AI lifecycle. Organizations that succeed in unifying their
data at scale with the best AI technologies will have a significantly higher chance of success with AI.
TOC3 Introduction
3 AI Dilemma
4 Data Related Challenges
6 Complexity from Explosion of ML Frameworks
7 Data Science and Engineering Silos
8 A New Category — Unified Analytics
9 Databricks Unified Analytics Platform
2018 TREND REPORT: ENTERPRISE AI ADOPTION 3 of 10
IntroductionCIO/IDG Research Services surveyed more than 200 IT executives at larger companies (1,000+ employees) in the
U.S. and Europe who are either considering or using AI technology. We wanted to understand the AI investments,
expected business outcomes, challenges, and the drivers of AI success across the landscape.
AI Dilemma — Nearly 90% investing in AI, very few succeedingIt’s clear that respondents are putting significant resources into their AI projects in hopes of forging new business
models that take advantage of data and ML across industries including the discovery of new life saving drugs,
detecting fraudulent and malicious behavior, improving global supply chain management, and creating a highly
personalized digital experience for their customers.
Despite the challenges, survey respondents are pursuing AI with gusto. It should be no surprise, then, that two-
thirds of respondents expect their AI investments to increase in the coming year. (See graphic below.)
Only 1 in 3 AI projects are
successful and it takes more than
6 months to go from development
to production.
Use/Planned Use of AI Technology Business Benefits Experienced from AI Projects
Predictive analytics
IT automation
Customer analytics
Security, fraud analysis and investigation
IoT analytics
Risk assessment
Improved security
Improved customer experience
Increased innovation
Better quality/more effective decision-making
Increased competitive advantage
Product/service transformation Source: IDG Research
48%
45%
44%
33%
27%
27%
29%
29%
26%
26%
23%
23%
2018 TREND REPORT: ENTERPRISE AI ADOPTION 4 of 10
“AI has massive potential to drive disruptive innovations affecting most enterprises on the planet. It’s pervasive
across all industries. It is used in genomics to accelerate drug discovery and drive personalized medicine. It
is being applied to manufacturing to improve operational efficiencies of product development and delivery
processes,” Bharath Gowda, VP of Product Marketing at Databricks, says. “In spite of the enormous potential,
very few companies are being successful with scaling with AI efforts”
Data-related Challenges Are Hindering 96% of Organizations from Achieving AIBut the CIO/IDG survey shows the full benefits of AI are not yet being realized for a variety of reasons, but with
one overarching theme: Data. Nearly all respondents (96%) cited multiple data-related challenges when it comes
time to move projects to production (see graphic below).
“ Simple models and a lot of data trump more elaborate models based
on less data.”PETER NORVIG
Research Director at Google
Challenges of Moving from Concept to Production
Preparation and aggregation of large datasets in a
timely fashion for analytics
Data exploration and iterative model training
with largesets of data
Deployment of models to production quickly
and reliably
Source: IDG ResearchVery/Extremely Challenging
Not very challenging
Somewhat challenging
1%56% 42%
4%56% 40%
10%53% 37%
2018 TREND REPORT: ENTERPRISE AI ADOPTION 5 of 10
“ Sifting through massive amounts of data to
identify useful signals is an enormous computational challenge; it’s the type of dataset and computation that DevOps nightmares
and Data Science dreams are made of.”
CHRIS ROBISON Lead Data Scientist at Overstock.com
And the data silos far outpace the other issues when talking about the data-related challenges, with technology
complexity also creating the second biggest challenge.
Data-Related Challenges to Move AI Projects to Production
Data silos (in different parts of the business, acrossdifferent locations, etc.)
Too many technologies in place/technology complexity
Accessing large sets of clean data quickly
Difficult for those processing/preparing data and thosecreating the data models to collaborate
Lack of ample access to data engineering and datascience talent to make AI a reality
Difficult for data scientists with varying skills andtechnology knowledge to collaborate
Lack of a scalable, reliable technology platform toprocess large data sets
51%
37%
35%
35%
29%
25%
24% Source: IDG Research
Gowda says, “For data scientists, it’s been proven that simple models built from large amounts of data produce
better results than very sophisticated models built from small sets of data,” he says. “So, more data means better
models — data is the fuel that powers AI. Clean, reliable data that is accessible to data scientists is the key to
success. Therein lies the challenge for enterprises — transforming siloed messy data into clean labeled data for
model development”
2018 TREND REPORT: ENTERPRISE AI ADOPTION 6 of 10
Increasing Complexity: Organizations Invest in an Average of Seven Different ML ToolsThe differences between the data engineering and data science teams also extend to the tools they use, and
there are many.
The vast majority (87%) invest in various sorts of data and AI related technologies to help with data preparation,
exploration, and modeling, including:
85% Data processing tools such as Apache Spark, Hadoop/MapReduce, and Google BigQuery,
used by 85% of respondents.
65% Data Streaming tools such as Flume, Kafka, and Onyx
80% Machine learning tools such as Azure ML, Amazon ML, and Spark MLlib
65% Deep learning tools such as Google TensorFlow, Microsoft CNTK, and Deeplearning4j (DL4J)
Overall, survey results show that organizations are using an average of seven different machine learning and deep
learning tools and frameworks, creating a highly complex environment that can slow efficiencies.
“To derive value from AI, enterprises are dependent on their existing data and ability to iteratively do ML on
massive datasets. Today’s data engineers and data scientists use numerous, disconnected tools to accomplish
this, including a zoo of ML frameworks,” Gowda says.
Click Streams
...
Video/ Speech
SensorData (IoT)
Emails/Web Pages
Customer Data
Great for Data, but not AI Great for AI, but not data
Click Streams
...
Video/ Speech
SensorData (IoT)
Emails/Web Pages
Customer Data
Great for Data, but not AI Great for AI, but not data
x
Divide Between Data & AI Technologies
2018 TREND REPORT: ENTERPRISE AI ADOPTION 7 of 10
Siloed Data Science and Engineering Teams: 80% Experience Reduced Productivity as a Result Technology skills, leadership, and lack of a cohesive strategy are the biggest hurdles faced by data engineering
and data science. (See graphic below.)
Collaboration between data engineering and data scientists
Extremely challenging
Not very challenging
Very challenging
Somewhat challenging
Source: IDG Research
9%
31%
40%
18%
2%Not at all challenging
Challenges for Data Engineering and Science
Technology skills/capabilities
gaps
Lack of project oversight/ leadership
Lack of a unified strategy
Disagreements regarding data
ownership/control
Limited understanding
of roles and responsibilities outside of one’s
own team
55%
47%
28%
34%
49%43%
30%
37%
28%24%
Source: IDG Research
Data Engineering
Data Science
“Disjointed development and data science teams
is a major obstacle in successfully doing data
analytics.”SAMAN MICHAEL FAR
Senior Vice President of Technology at FINRA
2018 TREND REPORT: ENTERPRISE AI ADOPTION 8 of 10
Unified Analytics — Many Need A New Category of Solutions to Conquer AI DilemmaSurvey respondents are clear that they would welcome such tools. Nearly 4 out of 5 (79%) said an end-to-end
analytics platform that unified big data and AI, while fostering better collaboration between data engineering
and data science teams, would be highly valuable.
Other features that would be welcome in such a platform include:
High quality performance for large data sets.
Built-in integration with various data sources.
Collaborative spaces that allow data scientists with different skills to work together.
Cloud-native platform to enable elastic scalability.
Built-in data management capability for building large data pipelines.
Support for multiple clouds.
Apache Spark was the first Unified
Analytics engine to unify data (data
engineering) with AI (data science).
Apache Spark has become the
de-facto data processing and AI
engine in enterprises today due to its
speed, ease of use, and sophisticated
analytics. Spark simplifies data
preparation for AI by unifying data at
massive scale across various sources
— cloud storage, file systems, key-
value stores, and message buses.
Spark also unifies data and AI with a
consistent set of APIs for simple data
loading, batch/stream processing,
SQL Analytics, Stream Analytics, and
Machine Learning.
2018 TREND REPORT: ENTERPRISE AI ADOPTION 9 of 10
Databricks Unified Analytics PlatformDatabricks accelerates innovation by unifying data science, engineering, and business. Through a fully managed,
cloud-based service built by the original creators of Apache Spark, the Databricks Unified Analytics Platform
lowers the barrier for enterprises to innovate with AI and accelerates their innovation.
“Databricks lets us focus on business problems and make data science very simple.”
DAN MORRIS Senior Director of Product Analytics at broadcast
giant Viacom, which used Databricks to help it identify video quality issues, increase customer loyalty,
and improve advertising performance.
DATABRICKS CLOUD SERVICE
DATABRICKS WORKSPACE
DATABRICKS RUNTIME
Reliable & Scalable Simple & Integrated
End to end ML lifecycle
Dashboards
Jobs
UNIFIED ANALYTICS PLATFORM
Databricks Delta ML Frameworks
API's
Notebooks
Models
Data EngineeringSpeed up the preparation of high quality data, essential for best-in-class ML applications, at scale.
Data ScienceCollaboratively explore large datasets, build models iteratively and deploy across multiple platforms.
2018 TREND REPORT: ENTERPRISE AI ADOPTION 10 of 10
Databricks Workspace — Unify data science and engineering teams
The Databricks Workspace empowers data science and engineering to collaborate using interactive notebooks
that are tightly integrated with cloud-native Apache SparkTM clusters. Real-time collaboration capability and
the ability to program in multiple languages increases data science productivity significantly. in the notebooks
increases the It integrates with MLflow, an open source, cross-cloud framework that tracks experiments and
enables deployment across multiples clouds dramatically simplify the ML workflow.
Databricks Runtime — Unify data and machine learning at massive scale
Databricks Runtime allows engineers to quickly build data pipelines at massive scale by using Delta tables which
bring data reliability and performance optimizations to data lakes. Using reliable data in Delta tables, data scien-
tists can continuously train and deploy state-of-the-art ML models by using pre-configured clusters that include
most of the popular ML frameworks such as TensorFlow, Horovod, Keras, XGBoost, and scikit-learn.
Databricks Cloud Service with enterprise-grade security
Databricks automates and simplifies dev-ops by abstracting the complexity of the data infrastructure by auto-
configuring and auto-scaling clusters; and provides enterprise-grade security and compliance, along with best-
in-class Spark support from the original creators of Apache Spark. Databricks Unified Analytics Platform provides
enterprise-grade security with encryption, auditing, role-based control and HIPAA and GDPR compliance.
“Enterprises are rightly pursuing multiple AI projects as they seek to realize the business benefits the technology
can provide. Data science is going to change the world,” as Databricks’ Gowda says. But as the CIO/IDG survey
makes clear, enterprises are often falling short of success in large part because they’re spending too much time
massaging data and struggling to collaborate between teams. The Databricks Unified Analytics Platform can
likewise help your company take advantage of AI technology to address strategic business objectives. It’ll help
you meet data and collaboration challenges and get AI projects out the door on time and on budget.
To learn more, visit: www.databricks.com
IDG Communications, Inc.
SPONSORED CONTENT
Benefits of AI are Streaming into Viacom Viacom turned to AI technology to help it analyze peta-bytes of network data to improve the performance of its network, grow its audience and improve the advertising performance of its 170 cable, broadcast and online networks in some 160 countries.
Viacom built a real-time analytics solution using the Databricks Unified Analytics Platform to constantly monitor the quality of video feeds and reallocate resources as necessary to ensure best-in-class customer experience. Databricks Runtime has the horsepower to keep up with Viacom’s constant flow of streaming data. And the Databricks Collaborative Workspace enables Viacom data scientists and engi-neering groups to collaborate with each other and with the business.
The results speak for themselves:
With the ability to predict video trends and issues, Viacom has reduced video start delays by 33%.
Leveraging customer data, Viacom has increased viewer retention 3.5 to 7 times
The ability to target customers with personalized ads based on comScore ratings and viewing behavior has improved ad conversion rates.
Dan Morris, Senior Director of Product Analytics for Viacom, isn’t done; rather, he’s looking for more ways to apply the technology. “Now it’s a question of how we bring these benefits to others in the organization who might not be aware of what they can do with this type of platform,” he says.