Fraud Detection Architecture

Real Time Fraud DetectionPatterns and reference architectures

Ted Malaska // PSA Gwen Shapira // Software Engineer

• Intro• Review Problem• Quick overview of key technology• High level architecture• Deep Dive into NRT Processing• Completing the Puzzle – Micro-batch, Ingest and Batch

Overview

• 15 years of moving data• Formerly consultant• Now Cloudera Engineer:– Sqoop Committer– Kafka– Flume

• @gwenshap

Gwen Shapira

• Ted Malaska (PSA at Cloudera)• Hadoop for ~5 years• Contributed to

– HDFS, MapReduce, Yarn, HBase, Spark, Avro, – Kite, Pig, Navigator, Cloudera Manager, Flume, Kafke, Sqoop, Accumulo – And working on a Sentry Patch

• Co-Author to O’Reilly Hadoop Application Architectures• Worked with about 70 companies in 8 countries• Marvel Fan Boy• Runner

Credit Card Transaction Fraud

Ikea Meat Balls

Coupon Fraud

Video Game Strategy

Health Insurance Fraud

• Typical Atomic Card Fraud Detection• Ikea Meat Ball• Multi Coupons Combinations • OP or Negative Video Games Strategies • Ad Serving • Health Insurance Fraud• Kid Coming Home From School

Review of the Problem

How do we React• Human Brain at Tennis – Muscle Memory– Reaction Thought– Reflective Meditation

Overview of Key Technologies

•Messages are organized into topics•Producers push messages•Consumers pull messages• Kafka runs in a cluster. Nodes are called brokers

The Basics

Topics, Partitions and Logs

Each partition is a log

Each Broker has many partitions

Partition 0 Partition 0

Partition 1 Partition 1

Partition 2

Partition 1

Partition 0

Partition 2 Partion 2

Producers load balance between partitions

Partition 0

Partition 1

Partition 2

Partition 1

Partition 0

Partition 2

Partition 0

Partition 1

Partion 2

Client

Producers load balance between partitions

Partition 0

Partition 1

Partition 2

Partition 1

Partition 0

Partition 2

Partition 0

Partition 1

Partion 2

Client

Consumers

Consumer Group Y

Consumer Group X

Consumer

Kafka Cluster

Partition A (File)

Partition B (File)

Partition C (File)

Consumer

Order retained with in partition

Order retained with in partition but not over

partitionsOff

Off sets are kept per consumer group

Sources Interceptors Selectors Channels Sinks

Flume Agent

Short Intro to FlumeTwitter, logs, JMS, webserver, Kafka

Mask, re-format, validate…

DR, criticalMemory, file,

KafkaHDFS, HBase,

Flume and/or Kafka

UpStream

Flume Source

Interceptor

Flume Channel

Flume Sink

Down Stream

SelectorCan Be KafkaCan Be KafkaCan Be Kafka

Interceptors• Mask fields• Validate information against external source• Extract fields• Modify data format• Filter or split events

SparkStreaming

Spark Streaming Example

1. val conf = new SparkConf().setMaster("local[2]”) 2. val ssc = new StreamingContext(conf, Seconds(1))3. val lines = ssc.socketTextStream("localhost", 9999)4. val words = lines.flatMap(_.split(" "))5. val pairs = words.map(word => (word, 1))6. val wordCounts = pairs.reduceByKey(_ + _)7. wordCounts.print()8. SSC.start()

Spark Streaming Example

1. val conf = new SparkConf().setMaster("local[2]”) 2. val sc = new SparkContext(conf)3. val lines = sc.textFile(path, 2)4. val words = lines.flatMap(_.split(" "))5. val pairs = words.map(word => (word, 1))6. val wordCounts = pairs.reduceByKey(_ + _)7. wordCounts.print()

29Confidentiality Information Goes Here

DStream

Spark Streaming

Single Pass

Source Receiver RDD

Filter Count Print

Source Receiver RDD

Single Pass

Filter Count Print

Pre-first Batch

First Batch

Second Batch

30Confidentiality Information Goes Here

DStream

DStreamSpark Streaming

Single Pass

Source Receiver RDD

Filter Count

Source Receiver RDD

Single PassFilter Count

Pre-first Batch

First Batch

Second Batch

Stateful RDD 1

Stateful RDD 2

Stateful RDD 1

Spark Streaming and HBase

Driver

Walker Node

Configs

Executor

Static SpaceConfigs

HConnection

Tasks Tasks

Walker NodeExecutor

Static SpaceConfigs

HConnection

Tasks Tasks

High Level Architecture

Real-Time Event Processing Approach

Hadoop Cluster IIStorage Processing

Hadoop Cluster I

ClientClientFlume Agents Hbase /

Memory

Spark Streamin

Hive/ImpalaMap/

ReduceSpark

Search

Automated & Manual

Analytical Adjustments and Pattern detection

Fetching & Updating Profiles

Adjusting NRT Stats

HDFSEventSink

SolR Sink

Batch Time Adjustments

Automated & Manual

Review of NRT

Changes and Counters

Local Cache

Clients:(Swipe here!)

Web App

Focus on NRT First

Hadoop Cluster I

Memory

Spark Streamin

Hive/ImpalaMap/

ReduceSpark

Search

Automated & Manual

Adjusting NRT Stats

HDFSEventSink

SolR Sink

Automated & Manual

Review of NRT

Local Cache

Web App

NRT Event Processing with Context

Streaming Architecture – NRT Event Processing

Flume SourceFlume Source

Initial Events Topic

Flume SourceFlume InterceptorEvent Processing

LogicLocal

MemoryHBase Client

Answer Topic

Able to respond with in 10s of milliseconds

Partitioned NRT Event Processing

Flume SourceFlume Source

Initial Events Topic Flume SourceFlume InterceptorEvent Processing

LogicLocal

MemoryHBase Client

Answer Topic

TopicPartition A

Partition B

Partition C

Producer

Partitioner

Producer

Partitioner

Producer

Partitioner

Custom Partitioner

Better use of local memory

Completing the Puzzle

Micro Batching

Hadoop Cluster I

Memory

Spark Streamin

Hive/ImpalaMap/

ReduceSpark

Search

Automated & Manual

Adjusting NRT Stats

HDFSEventSink

SolR Sink

Automated & Manual

Review of NRT

Local Cache

Web App

Micro Batching

Micro BatchingMicro Batching

Complex Topologies

Spark Streaming

Dag Topologies

Spark StreamingKafka Receivers Dag Topologies

Kafka Receivers

• Manages Offset• Stores Offset is RDD• No longer needs HDFS for initial RDD check

pointing

• Lets Kafka Manage Offsets• Uses HDFS for initial RDD recovery

MicroBatch Bad-Input Handling

0 1 2 3 4 5 6 7 8 9 10

Kafka – incoming events topic

Dag Topologies

0 1 2 3 4 5 6 7 8 9 10

Kafka – bad events topic

0 1 2 3 4 5 6 7 8 9 10

Kafka – resolved events topic

0 1 2 3 4 5 6 7 8 9 10

Kafka – results topic

Ingestion

Hadoop Cluster I

Memory

Spark Streamin

Hive/ImpalaMap/

ReduceSpark

Search

Automated & Manual

Adjusting NRT Stats

HDFSEventSink

SolR Sink

Automated & Manual

Review of NRT

Local Cache

Web App

Ingestion

Flume HDFS SinkKafka Cluster

TopicPartition A

Partition B

Partition C

SinkSinkSink

Flume SolR SinkSinkSinkSink

Flume Hbase SinkSinkSinkSink

Reflective Thoughts

Hadoop Cluster I

Memory

Spark Streamin

Hive/ImpalaMap/

ReduceSpark

Search

Automated & Manual

Adjusting NRT Stats

HDFSEventSink

SolR Sink

Automated & Manual

Review of NRT

Local Cache

Web App

Research and Searching

Fraud Detection Architecture

Data & Analytics

Transcript of Fraud Detection Architecture

Fraud Detection using Hadoop

The Impact of Fraud Education on Fraud Detection

Credit Card Transaction Fraud Detection - AMiner · Credit Card Transaction Fraud Detection Niall Adams ... Another plastic card fraud anecdote... ... Types of fraud

Adaptive Fraud Detection

Fraud Detection Module Advanced: Checklist · You can access the Fraud Detection Module via the “Fraud detection” link in your account menu. .3 Co nte s The Fraud Detection Module

Fraud Detection Gap between Auditor and Fraud Detection ...

Comparative Election Fraud Detection

Fraud Detection Class Slides

Business Ethics, Fraud and Fraud Detection

Fraud Detection

Medicare fraud detection

Software AG Investor Presentation/media/Files/S/... · •Fraud detection •Reduce fraud loss •Improve speed and accuracy of fraud detection system •Increase rules in fraud detection

A Cure for Ad-Fraud: Turning Fraud Detection into Fraud Prevention

TIBCO Fraud Detection

Automotive Insurance Fraud Detection

Fraud Detection Module Advanced: Checklist...You can access the Fraud Detection Module via the “Fraud detection” link in your account menu. .3 Co nte s The Fraud Detection Module

Affiliate’s Fraud Detection & Prevention

Phone Fraud Detection

Fraud Detection in Hotel

Fraud Detection Architecture