Cloudera training: secure your Cloudera cluster

The demand for skills is high and Hadoop is the future. Customers cannot afford to move slowly in staffing their Big Data projects. Customers are building plans to ensure projects are staffed with skilled employees, and supported by a qualified services provider.

Job Trends from Indeed.com

What are you most concerned about when it comes to your readiness for big data and hadoop?

Cloudera MDP webinar poll results, July 2016

Why Cloudera training?Aligned to best practices and the pace of change

1Broadest range of coursesLearning paths for Developer, Admin, Analyst

2Most experienced instructorsMore than 40,000 trained since 2009

6Widest geographic coverageMost classes offered: 50 cities worldwide plus online

7Most relevant platform & communityCDH deployed more than all other distributions combined

3Leader in certificationOver 12,000 accredited Cloudera professionals

Trusted source for training100,000+ people have attended online courses4

8Depth of training materialHands-on labs and VMs support live instruction

9Ongoing learningVideo tutorials and e-learning complement training

State of the art curriculumCourses updated as Hadoop evolves5 10 Commitment to big data education

University partnerships to teach Hadoop in colleges

Creating leaders in the fieldTraining enables Big Data solutions and innovation

Would recommend or highly recommend Cloudera training to friends or colleagues

Draw on lessons from Cloudera training on at least a monthly basis

40% Develop new apps or perform business-critical analyses as a result of training alone

Sources: Cloudera Past Public Training Participant Study, December 2012.

Cloudera Customer Satisfaction Study, January 2013.

88% Indicate Cloudera training provided the Hadoopexpertise their roles require

What is available from Cloudera University?

• Private training: Course delivered at location of customer choice to internal audience

• Public training: Courses regularly scheduled around the globe. Schedule available on web

• Virtual training: Live training accessed via the internet; available for public and private courses

• OnDemand training: Pre-recorded lecture with identical content/exercises as live training options

• Certification: Rigorously developed and meaningful bodies of knowledge

OnDemand Virtual live classroom Private onsitePublic live classroom

Suggested Cloudera University curricula

Developers

• Python/Scala Training

• Developer for Spark and Hadoop

• CCA: Spark and Hadoop

Developer

• Spark ML & Kafka modules

• Topic specific training (Search,

HBase)

• Hands on practice

• CCP: Data Engineer

Administrators

• Cloudera Administration training

• CCA: Administrator

• Cloudera Security OnDemand

Data Analysts/Data Scientists

• Data Analyst: Using Hive, Pig & Impala

• CCA: Data Analyst

• Cloudera Data Science

Security for Hadoop

Carlo Lazzaris | Technical Instructor

Security Webinar Agenda

1. The need for Hadoop Security

Hacker news and legal regulations

2. Cloudera Security Implementation

Five levels of security

3. How to secure your Cloudera cluster

Cloudera Documentation

Cloudera professional services

Cloudera OnDemand security course

The need for Hadoop security

Unguarded data stores are the victims

Regulatory Compliance

Organizations can be fined up to 4% of annual global turnover for breaching GDPR

or €20 Million

Cloudera security implementation

Cloudera Enterprise CDH

The modern platform for machine learning and analytics optimized for the cloud

EXTENSIBLE SERVICES

CORE SERVICESDATA

ENGINEERINGOPERATIONAL

DATABASEANALYTIC DATABASE

DATA CATALOG

INGEST & REPLICATION

SECURITY GOVERNANCEWORKLOAD

MANAGEMENT

DATA SCIENCE

S3 ADLS HDFS KUDUSTORAGESERVICES

• Unified security – protects sensitive data with consistent

controls, even for transient and recurring workloads

• Consistent governance – enables secure self-service access

to all relevant data and increases compliance

• Easy workload management – increases user productivity and

boosts job predictability

• Flexible ingest and replication – aggregates a single copy of

all data, provides disaster recovery, and eases migration

• Shared catalog – defines and preserves structure and

business context of data for new applications and partner

solutions

Open platform servicesBuilt for multi-function analytics | Optimized for cloud

Cloudera Enterprise-Grade Security and Governance

Access

Defining what

users and

applications can

do with data

Technical Concepts:

Permissions

Authorization

Protection

Shielding data in

the cluster from

unauthorized

visibility

Technical Concepts:

Encryption at rest & in

motion

Visibility

Reporting on

where data came

from and how it’s

being used

Technical Concepts:

Auditing

Lineage

Cloudera Manager Apache Sentry Cloudera NavigatorNavigator Encrypt &

Key Trustee

Identity

Validate users by

membership in

enterprise

directory

Technical

Concepts:Authentication

User/group mapping

Cloudera Certified Technology Partners

Data Sources Data IngestProcess, Refine

& PrepData Discovery Advanced Analytics

Connected Machines/Data sources

Other Data Sources

A certified product ensures it integrates securely

• Authenticate via Kerberos or LDAP

Authentication

• Handle Apache Sentry with Hive, Impala, Search, HDFS

Authorization

• Support HDFS transport encryption, at-rest encryption; support SSL/TLS connection encryption

Encryption

Vulnerability Response and Process

Vulnerability reports

Upstream

Internal

External

Fix Publish

Cluster Security Levels

Cloudera Enterprise

The modern platform for machine learning and analytics optimized for the cloud

Enterprise Encryption Performance

Disclaimer

This talk serves as a general guideline for

security implementation on Hadoop.

The actual implementation procedures and

scope of implementation vary on a case-by-

case basis, and should be assessed by

Cloudera’s Professional Services team or

certified Cloudera SI Partners.

Non-secure #0Data Free for All

Firewall

ActiveDirectory/KDC

Hadoop cluster

Cloudera Manager

Gateway node

Cloudera Worker nodesDatacenter

Applications

4 modes of Identity Management

1. Simple Authentication2. Kerberos3. LDAP4. SAML

File group ownership• AD integration• SSSD or CentrifyConsideration in large enterprises.

via SSSD

Simple Authentication detect the user

Firewall

ActiveDirectory

Master

Worker Worker Worker

Cloudera Manager

Master

(SSSD/Centrify)

Simple authentication =

no authentication

Minimal Security #1

Reduce Risk Exposure

How it works: Authentication

• LDAP and SAML authentication options

Web UIs

• LDAP/AD and Kerberos authentication options

SQL Access

•Kerberos authentication

•Automation provided by Cloudera Manager to leverage Active Directory (AD)

Command Lines

User authenticates to AD or KDC

Authenticated user gets Kerberos Ticket

Ticket grants access to Services e.g. Impala

User [ssmith]

Password [***** ]

Kerberos

EXAMPLE.COM

user@EXAMPLE.COM

Hadoop

user@EXAMPLE.COM

Strong Authentication

KDC Key Distribution Center

• MIT

• ActiveDirectory (more common)

realmprimary

Kerberos

Consideration in large corporates

Time synchronization

CM Kerberos Wizard

• Configure AD to create a Kerberos

principal for CM server, and to

delegate CM the ability to

create/manage Kerberos

principals

Kerberos

Consideration in large corporates

Time synchronization

CM Kerberos Wizard

• Configure AD to create a Kerberos

principal for CM server, and to

delegate CM the ability to

create/manage Kerberos

principals

Kerberos Authentication

* LDAP over SSL

Authorization/Access Control

HDFS File ACL YARN job submission

Hbase ACLs Oozie ACL

Access Control List (ACLs)

Sentry Managed

(RBAC)

Impala

Auditing

Backup/Disaster Recovery

Cloudera Backup/Disaster Recovery (BDR)

• A high performance data replicator

• Copies incremental data on the source cluster at specified schedules

Supports

Kerberos

Data encryption

HDFS replication to cloud

Kerberized BDR Best Practice

Production DR

Cloudera BDRPROD.EXAMPLE.COM

Cross-realm trustKDC KDC

DR.EXAMPLE.COM

More Security #2

Managed, Secure, Protected

Data In-Motion Encryption

RPC encryption

Data transport encryption

• Supports AES CTR, up to 256-bit

key length

HTTP TLS/SSL encryption

• No self-signed certificates in

production

Master

Worker Worker Worker

Master

Application

RPC encryption

Transport encryption

TLS/SSL

Data At-Rest Encryption

Transparent encryption

Supports any Hadoop applications

Encryption Zone

$ hadoop key create mykey

$ hadoop fs -mkdir /zone

$ hdfs crypto -createZone -keyName mykey -path /zone

/tmp /zone

foo bar

Encryption zone

Key Management Server Deployment (non-prod)

HDFS NameNode

Client

Java Keystore

KMSKeystore file

Separation of duties

• Encryption Zone Key (EZK) is stored in

KMS server

• HDFS super user can not decrypt files

Key Management Server/Key Trustee Server Deployment

HDFS NameNode

ClientKey Trustee

Key Trustee KMS

Firewall

Key Trustee Server

(Active)

Key Trustee Server

(Passive)

synchronization

(or more)

KMS+KTS+HSM Deployment

HDFS NameNode

Client HSM KMS

HSM KMS

Firewall

Key Trustee Server

(Active)

Key Trustee Server

(Passive)

synchronization

Key HSM

(or more)

Key HSM

Troubleshooting: Encryption Performance Anomaly

• Configuration

• AES-NI Hardware acceleration

• OpenSSL library

• Entropy

Fine Grained Access Control with Apache Sentry

Most Security #3

Secure Data Vault

Level 3 Secure Data Vault

• All data, both data-at-rest and data-in-transit is encrypted

• Key management system is fault-tolerant

• Auditing mechanisms comply with industry, government, and regulatory

standards (PCI, HIPAA, NIST, for example)

• Auditing extends from EDH to the other systems that integrate with it.

• Cluster administrators are well-trained

• Security procedures have been certified by an expert

• Cluster can pass technical review

Data Redaction

Personal Identifiable Information

• PCI-DSS, HIPAA

Best practices followed

Password

• stores in credential files, not in configuration

Log, queries

• Cloudera Manager

Full Encryption

Encrypt Data Spills

• MapReduce

• Impala

• Hive

• Flume

OS-level encryption

• Navigator Encrypt

How to secure your Cloudera cluster

Cloudera Documentation

Cloudera Professional Services security engagement

• Review security requirements and provide an overview of data security policies

• Audit architecture and current systems for security policies and best practices

• Custom tailor a security reference architecture

• Optimize OS and Java to take advantage of hardware-based crypto-acceleration

• Install and configure Kerberos with MIT Kerberos KDC or Active Directory

• Install and configure Sentry and Cloudera Navigator (license required)

• Install and configure Navigator Encrypt and Key Trustee with an HSM root of trust

• Review fine-grain permissions on sample data using Sentry

• Review audit and lineage on sample data using Navigator

• Use Cloudera Manager and Hue to review security integration for users

• Enable and configure HDFS encryption

https://www.cloudera.com/more/services-and-support/professional-services/security-integration-pilot.html

Cloudera online ondemand security course

• Online self paced training course https://ondemand.cloudera.com

• Launch planned for mid Feb 2018

• 3 days estimate worth of content at Cloudera level 1 and 2 security level

• Currently 375~ slides with 9 detailed chapters and 16 instructor demonstrations :

1. Security overview

2. Security Architecture

3. Host Security

4. Encrypting Data in motion

5. Authentication

6. Authorization

7. Encrypting Data at Rest

8. Auditing

9. Additional Considerations: Data Governance

Ondemand security course instructor guided demos

1. Potential Attack vectors

2. Securing the cluster hosts

3. Generating and managing keys for TLS

4. Configuring Cloudera Manager for TLS

5. Encrypting Data in Motion

6. Hadoop default authentication

7. Kerberizing Cluster with MIT Kerberos

8. Kerberizing Cluster with Active Directory

9. Configuring Authorising with Cloudera

Manager

10. Controlling access to Yarn

11. Controlling access to HDFS

12. Controlling access to Tables

13. Enabling HDFS Encryption

14. Protecting local data with NavEncrypt

15. Using Navigator for auditing

16. Reassessing cluster security

Ondemand security course disclaimer

THIS IS REALLY IMPORTANT:

The examples in this course are based on CM/CDH 5.12, running in a cloud-based deployment on a

cluster using the CentOS 7.2 operating system.

Given the almost limitless permutations of possible configurations, including different versions of CDH,

Cloudera Manager, operating systems, directory servers, Kerberos servers, web browsers, and other

tools, as well as variations in policies, laws, and practices that affect each organization differently, it's

impossible for a training course to cover all aspects of security.

This course is meant to provide a background that will help you to understand many important concepts

and techniques, but is not intended as a replacement for the relevant documentation or a consulting

engagement with an expert who can provide advice based on your specific requirements.

• Disclaimers ~ due to security variety and permutations

• Versions used: CDH 5.12 and Centos 7.2

Ondemand security course scenario

• Many of our demonstrations are based on a hypothetical scenario

• However, the concepts should apply to nearly any organization

• Loudacre Mobile is a fast-growing wireless carrier

• Employees serving in a variety of roles

• Data ingested from many sources, in many formats

• Data processed by many tools

Ondemand security course environment

Comprehensive demonstration cluster

Sample chapter structure: Encrypting Data in Motion

• Encryption Fundamentals

• Certificates

• Key Management

Instructor-Led Demonstration: Generating and Managing Keys for TLS

• Configuring Cloudera Manager for TLS

Instructor-Led Demonstration: Configuring Cloudera Manager for TLS

• Encrypting Hadoop’s Data in Motion

Instructor-Led Demonstration: Encrypting Hadoop’s Data in Motion

• Essential Points

Register your interest forOnDemand security course:

peter.rizvi@cloudera.com

Thank you

Cloudera training: secure your Cloudera cluster

Business

Transcript of Cloudera training: secure your Cloudera cluster

‹#› © Cloudera, Inc. All rights reserved. Marcel Kornacker // Cloudera, Inc. Friction-Free ELT: Automating Data Transformation with Cloudera Impala.

Managing and Monitoring a Cluster - Cloudera · Introducing Ambari operations Apache Ambari collects a wide range of information from cluster nodes and services and displays that

Building the Enterprise Data Lake with Cloudera & Cisco · cloudera introduces the enterprise data hub and cloudera enterprise 5 2015 cloudera includes kafka, kudu and record service

Advanced Cluster Options - Cloudera...Advanced cluster options Custom images "2.8.0"]}]}} Related Information Example image catalog Register image catalog Once you've created your

Setting Up Hadoop Cluster With Cloudera Manager and Impala

Trust Aware Secure Routing for Cluster-based Wireless ... · Trust Aware Secure Routing for Cluster-based Wireless Sensor Networks - A Comparative Study P.Prittopaul1, Dr.N.Shankarram2

Cloudera User Group SF - Cloudera Manager: APIs & Extensibility

Cloudera cluster setup and configuration

Apache Atlas Reference - Cloudera · Cloudera, Cloudera Altus, HUE, Impala, Cloudera Impala, and other Cloudera marks are registered or unregistered trademarks in the United States

Cloudera User Group Chicago - Cloudera Manager: APIs & Extensibility

Cloudera GoDataFest Deploying Cloudera in the Cloud

SECURE UPDATES › wp-content › uploads › 2019 › 11 › Asvin.io_.pdfMySQL Cluster Server Cluster REDIS Load Balancer Server Version Controller Cluster of Web Servers Hyperledger

Secure Hadoop Cluster With Kerberos

Hadoop Operations: How to Secure and Control Cluster Access

Cloudera Data Management · ImportantNotice ©2010-2020Cloudera,Inc.Allrightsreserved. Cloudera,theClouderalogo,andanyotherproductor ...

A Secure Cluster Based Framework for Preventing … International Journal of Control Theory and Applications A Secure Cluster Based Framework for Preventing Attacks in Vehicular Cyber

Cloudera Administration Handbook - Packt · Cloudera Administration Handbook . ... , ... Cloudera's cluster management application and walks you through the steps

HueGuide - Cloudera

Cloudera Desktop

Dell EMC Ready Solutions for AI - cioconference.ro › lib › foto › Dell-EMC... · Cloudera to build a secure, PCI-certified Hadoop cluster MasterCard's Enterprise Data Hub fully