Teradata and Hortonworks  The Unified Data Architecture (UDA)

 16th October, 2014

2

Shift from a Single Platform to an Ecosystem

“The hype around replacing the data warehouse gives way to the more sensible strategy of augmenting it … The influence of the logical data warehouse has created a situation in which multiple repository strategies are now expected.”

"Logical" Data Warehouse

“Big Data requirements are solved by a range of platforms including analytical databases, discovery platforms, and NoSQL solutions beyond Hadoop.”

Source: “Big Data Comes of Age”. EMA and 9sight Consulting. Nov 2012.

Math and Stats

Data Mining

Business Intelligence

Applications

Languages

Marketing

ANALYTIC TOOLS & APPS

USERS

INTEGRATED DISCOVERY PLATFORM

INTEGRATED DATA WAREHOUSE

ERP

SCM

CRM

Images

Audio and Video

Machine Logs

Text

Web and Social

SOURCES

DATA PLATFORM

ACCESS MANAGE MOVE

UNIFIED DATA ARCHITECTURE System Conceptual View

Marketing Executives

Operational Systems

Frontline Workers

Customers Partners

Engineers

Data Scientists

Business Analysts

Math and Stats

Data Mining

Business Intelligence

Applications

Languages

Marketing

ANALYTIC TOOLS & APPS

USERS

INTEGRATED DISCOVERY PLATFORM

INTEGRATED DATA WAREHOUSE

ERP

SCM

CRM

Images

Audio and Video

Machine Logs

Text

Web and Social

SOURCES

DATA PLATFORM

Business Intelligence

Predictive Analytics

Operational Intelligence

Data Discovery

Path, graph, time-series analysis

Pattern Detection

Fast Data Loading & Availability

Filtering & Processing

Deep History: Online Archival

UNIFIED DATA ARCHITECTURE Business Conceptual View

Fast-Fail Hypothesis Testing

Marketing Executives

Operational Systems

Frontline Workers

Customers Partners

Engineers

Data Scientists

Business Analysts

ACCESS MANAGE MOVE

Data Mgmt. (data lake)

5

Discovering Deep Retail Insights with UDA Transforming Web Walks into DNA Sequences

Impact

•  Leverage Aster platform to generate rapid path insights •  Drives 15% increase in market baskets through personalization •  Drives 10-20% increase in conversions by shortening paths •  Can now see what does and doesn’t lead to sales •  Widening use across all the Corporate Group websites

Situation

Largest German online retailer, conglomerate with numerous brands and 50 websites. 1 Millions visitors, viewing 2M products.

Problem

Needed a better way of analyzing consumer behavior on the websites, communicating with category managers

Solution

Treat each web visit sequence like DNA sequence. Built a fast query tools so analysts can express queries easily for their categories, get deeper insights

KNOX

AMBARI

SOURCE DATA

Sensor Log Data

Customer/Inventory

Data

Clickstream Data

Flat Files

Sentiment Analysis

Data

DB

File

JMS

REST

HTTP

Streaming

Analytical Platforms

Teradata IDW

Aster Discovery Platform

Query/Visualization/ Reporting/Analytical

Tools and Apps

JDBC/ODBC Compliant Tool

MAPREDUCE YARN

Viewpoint Alerts Services System

Health Node

Health Space Usage

Capacity Heatmap

Metrics Analysis

TVI – Proactive system monitoring tied to Teradata customer support

HDFS

REFINE HIVE

PIG

CUSTOM

ETL

LOAD SQOOP

FLUME

Web HDFS

NFS EXTRACT

STRUCTURING

HCATALOG

INTERACTIVE

QueryGrid

EXPORT SQOOP / HIVE

LOAD TDCH

BULK COPY

DISTCP AFS

EXTRACT

Modern Data Architecture: Teradata

Bidirectional

7

• Most Trusted and Flexible Hadoop Platforms for Your Next-Generation Unified Data Architecture™

1.  Teradata Aster Big Analytics Appliance

2.  Teradata Appliance for Hadoop

3.  Teradata Commodity Offering with Dell

4.  Hortonworks Data Platform software-only support resell

• Complete consulting and training capability

> Big Analytics Services—across the UDA

> Data Integration Optimization—ETL, ELT across the UDA

> Hadoop deployment and mentoring

> Teradata delivering Hortonworks training

> Hadoop Managed Services—operations and administration

• Customer Support for Hadoop > World-class Teradata customer support, backed by Hortonworks

Teradata Portfolio for Hadoop ” Bringing Hadoop to the Enterprise”

8

Loom is a platform for profiling, preparing and tracking data lineage for data in Hadoop

•  Hadoop Data Governance and Metadata Management –  Rich information model for capturing and managing the relationships –  Data dictionary for the big data landscape –  Support for non-Hadoop sources

•  Automation (Activescan) –  Discovering and introspecting new data in the cluster –  Triggering external processing (e.g. Oozie script for ETL) –  Automatically collecting metadata about the job - lineage, statistics –  Polling YARN job history for lineage

•  User Interactivity (Workbench) –  Advanced user interfaces for data exploration, profiling and preparation –  Data wrangling for interactively cleaning/reshaping raw data into useable data

Teradata Loom® 2.3 “Integrated metadata management, data lineage

and data wrangling for Enterprise Hadoop”

Free version of Loom pre-installed with Hortonworks Sandbox

9

Teradata Appliance for Hadoop

Optimized hardware for Hadoop

BYNET™ V5 40GB/s InfiniBand interconnect

Tera

da

ta V

ital I

nfr

ast

ruc

ture

Teradata Distribution for Hadoop (Based on Hortonworks HDP)

NameNode Failover

Intelligent Start and Stop

Teradata Connector for Hadoop (TDCH)

Teradata QueryGrid ® Teradata Studio with

Smart Loader

Teradata Viewpoint

Value Added Software from Partners

HCatalog

Kerberos

Teradata Loom® ( for data management )

10

Teradata QueryGrid™ Vision

TERADATA ASTER

DATABASE

SQL, SQL-MR, SQL-GR

Multiple Teradata Systems

TERADATA DATABASE

HADOOP

Push-down to Hadoop

System

IDW

TERADATA DATABASE

Discovery

TERADATA ASTER

DATABASE

Business users Data Scientists

COMPUTE CLUSTER

Run SAS, Perl, Ruby, Python, R

RDBMS DATABASES

Push-down to Other

Database

MONGODB DATABASE

Push-down to NoSQL

Databases

11

•  Trusted: Use existing tools/skills and enable self-service BI with granular security

•  Standard: 100% ANSI SQL access to Hadoop data

•  Fast: Queries run on Teradata or Aster, data accessed from Hadoop

•  Efficient: Intelligent data access leveraging the Hadoop HCatalog

Hadoop Layer: HDFS

Pig

Hive

Hadoop MR

QueryGrid: Teradata-Hadoop QueryGrid: Aster-Hadoop

HCatalog

Da

ta

Da

ta F

ilte

ring

Give business users on-the-fly access to data in Hadoop

Teradata QueryGrid™: Teradata - Hadoop

12

Teradata Viewpoint

•  Hadoop Portlets: –  Node Monitor (Aster & Hadoop)

–  Hadoop Services

•  Integration into existing: –  Monitoring: System Health, Metrics

Analysis, Metrics Graph, Capacity Heatmap, Space Usage.

–  Admin: Alert Viewer, Alert Setup, Teradata Systems, Role Manager

Single Operational View (SOV) for Teradata, Aster, & Hadoop

13

•  Key Features –  High-speed connector between Teradata and

Hadoop based on Apache Sqoop framework

–  Both import and export data between Teradata and Hadoop

–  Leverages the JDBC-FastLoad/FastExport mechanism from Teradata

–  Import/export Hive rcfile/sequencefile/textfile format and Hive partitioned files

Teradata Connector for Hadoop (TDCH)

INTEGRATED DATA WAREHOUSE

CAPTURE | STORE | REFINE

• Available through Hortonworks >  Hortonworks

•  Teradata Connector for Apache Hadoop (Release v1.2.0) •  Download link: http://hortonworks.com/download/

14

•  Hadoop View –  Browse through tables

within the Hadoop cluster -  Views table properties

–  Bi-directional table copies -  Drag and drop interface

-  Maps data types between Hadoop and Teradata tables

–  Transfer Status and History -  Track load status

•  Benefits –  Simplifies Hadoop browsing

–  Ad hoc data movement between Teradata and Hadoop

–  No scripting required

–  Point and click

Teradata Studio: Smart Loader for Hadoop Self-Service Load

15

Questions and Next Steps

More about Teradata & Hortonworks http://www.hortonworks.com/partner/teradata/

Teradata Loom for HDP http://www.teradata.com/tryloom

Find Us @Strata

Booth # 324 Teradata Hadoop Station