MongoDB and RDBMS: Using Polyglot Persistence at Equifax
-
Upload
mongodb -
Category
Technology
-
view
1.132 -
download
0
Transcript of MongoDB and RDBMS: Using Polyglot Persistence at Equifax
1
MongoDB and RDBMS: Using Polyglot Persistence at Equifax
MongoDB Evenings Atlanta
September 24, 2015Mike Lawrence
2
“I specialize in business development utilizing a strong background in data science and architecture to improve business
Go-To-Market strategies and operation. I enjoy leveraging data to spot industry trends, make predictive decisions about
future growth areas, and improve context capture for data sets. I am also a caffeine-aholic, so please feel free to say hello
to me next time you’re at Starbucks. “
Mike LawrenceAssociate, Pariveda Solutions
@theMrLawrence
3
You will develop a strong understanding of the polyglot persistence usecase
Three Key Takeaways
1. Breaking traditional data storage patterns enabled Equifax to develop data persistence and access patterns for agility
2. The key drivers to implement MongoDB and the benefits to the business and consumer experience
3. Leveraging the strengths of MongoDB and RDBMS provides a versatile data solution that increase the lifetime value of consumer relationships and improve customer experience.
4
From business overview to solution architecture
A Look at our Presentation Agenda
Equifax PSOLA quick overview of the Equifax
Personal Solutions business unit
Data Access PatternsDiving into the data persistence and access patterns by the application
5
Understanding the DataExplore the different types of data
and understanding its use
Document StorageUse case, adoption, advantages of a document storage solution
Cost Savings with MongoDBA document storage solution
provided a reduction of overall storage costs
Relational StorageNot all data gets persisted into MongoDB, some data remains highly relational
6
Let’s begin!
Solution ArchitectureA quick glance into the polyglot
solution architecture between app and persistence
Q & ATime for all of your questions and comments!
7
Fueling New Product Innovation (NPI)
Equifax Personal Solutions
Consumer Impact
Personal Solutions continued to increase the lifetime value of its consumer relationships by improving the customer experience and
introducing new, high-value products
Equifax Personal Solutions, which contributes 10% of the overall Equifax Revenue, supplies consumers with information to help them understand their credit and protect their identity. In 2014 they launched a strategic transformation to ensure long-term, sustainable
growth in the face of a changing market environment.
8
The first step to the future
Equifax PSOL Strategic Transformation
Equifax is re-engineering the consumer application platform to better reach the digital consumer
A core principle of this strategic project is to introduce new technologies to Equifax that further expand their ability to execute on business objectives at lower operating costs and improve overall system performance.
9
Breaking Traditional Data Storage PatternsAdopting new approaches to data persistence and data access for agility
MapReduce
Data Processing for ComplexBI and Reporting
Streaming
Realtime processing and fulfillment
DocumentTransactional
Document Storage for cohesive and large transactional data
Relational Transactional
Relational storage for highly structured transactional data
DocumentArchival
Document Storage for Archival Solutions
Data Access Patterns
MapReduce
Data Processing for ComplexBI and Reporting
Streaming
Realtime processing and fulfillment
DocumentTransactional
Document Storage for cohesive and large transactional data
Relational Transactional
Relational storage for highly structured transactional data
DocumentArchival
Document Storage for Archival Solutions
Data Access Patterns
12
The types of data are wide ranging, but centered around the consumer
Application Data is Consumer Centric
Consumer InformationBasic information about the consumer must be persisted
to track identity
User AuthenticationRole management, user, and
customer verification are required for privileges
Order ManagementOrders are tracked through placement to completion
Product CatalogsAvailable products, offers, and cross sell are managed
through the database
ConfigurationsApplication configurations are stored for light payload
Audit LoggingAll activities must be tracked,
audited, and persisted
Digital ProductsCredit products are large documents of data to be supplied to a customer
Alert ProcessingAlerts are a form of product
that are persisted and supplied to consumers
13
Relational
Consumer InformationBasic information about the consumer must be persisted
to track identity
User AuthenticationRole management, user, and
customer verification are required for privileges
Order ManagementOrders are tracked through placement to completion
Product CatalogsAvailable products, offers, and cross sell are managed
through the database
ConfigurationsApplication configurations are stored for light payload
Audit LoggingAll activities must be tracked,
audited, and persisted
Digital ProductsCredit products are large documents of data to be supplied to a customer
Alert ProcessingAlerts are a form of product
that are persisted and supplied to consumers
14
Consumer InformationBasic information about the consumer must be persisted
to track identity
User AuthenticationRole management, user, and
customer verification are required for privileges
Order ManagementOrders are tracked through placement to completion
Product CatalogsAvailable products, offers, and cross sell are managed
through the database
ConfigurationsApplication configurations are stored for light payload
Audit LoggingAll activities must be tracked,
audited, and persisted
Digital ProductsCredit products are large documents of data to be supplied to a customer
Alert ProcessingAlerts are a form of product
that are persisted and supplied to consumers
Document
15
Document StorageAdoption, advantages and document types
16
As Equifax grows their consumer base, new data storage technologies were explored to keep with increase demand
Performance and costare two key drivers of success
Developing a new platform offered an opportunity to explore
different technologies solve new challenges
17
As development moved forward, the need document storage became clearer
Building a Case forDocument Storage
Lightweight Searchable StorageStorage of documents in RDBMS is bulky, slow retrieval, and
difficult to search
Data VolumeHigh volume of data creation and retrieval requires scalability
Last Mile DeliveryPerformance drives realtime rendering of credit reports, invoices, and other large documents
Realtime PerformanceLarge volumes of data are analyzed realtime by the business
18
The path that led Equifax from concept to adopting MongoDB
Choosing MongoDB as the Solution
Adoption of MongoDB is driven by retrieval, scalability, and cost
• MongoDB offers flexible storage, easy scalability, andhigh-performance searching and document retrieval
• High performance searching and retrieval allowsEquifax to render credit reports instantly
• MongoDB is a low-cost solution compared to RDBMSstorage of documents
• Scalability of MongoDB meets future growth needs ofEquifax as their data continues to grow exponentially
• Independent searching outside of RDBMS
High Volume Data Stored in RDBMS CLOB/BLOB
Fast Retention and Retrieval of Large Data
NoSQL Data Store
MongoDB
19
Determining where to persist data is decided from a few key rules
Rules for Data Persisted in MongoDB
Cohesive Unstructured Unknown Metadata High Volatility
Highly cohesive data or information that cannot be
broken down
Unstructured documents with few or no standards
New products and developments may require
different metadata
Documents susceptible to frequent schema changes
20
ConfigurationsApplication configurations are stored for light payload
Audit LoggingAll activities must be tracked,
audited, and persisted
Digital ProductsCredit products are large documents of data to be supplied to a customer
Alert ProcessingAlerts are a form of product
that are persisted and supplied to consumers
Documents Stored in MongoDB
21
What cost savings have we experienced with MongoDB?
22
A dramatic reduction in cost over relational storage
MongoDB Helped Increase Bottom Line
MongoDB Storage Cost per GB
Reduction of storage costs have been a major driving
force behind the implementation of MongoDB to
supplement the relational database.$2/gb
MongoDB$/gb
RDBMS$/gb
Storage costs were reduced
400% from $8/gb using
RDBMS to $2/gb using
MongoDB
400%Cost savings per GB
$ $ $ $ $ $ $ $
23
Relational StorageLeveraging existing investments
24
Determining where to persist data is decided from a few key rules
Rules for Data Persisted in RDBMS
Loosely Coupled Highly Structured Highly Related Low Volatility
Data that withstands the breakdown into smaller pieces
Hierarchical or other defined structures
Data extends out to many associations
Relational Data tends to go through few and minor
changes over time
25
Relational
Consumer InformationBasic information about the consumer must be persisted
to track identity
User AuthenticationRole management, user, and
customer verification are required for privileges
Order ManagementOrders are tracked through placement to completion
Product CatalogsAvailable products, offers, and cross sell are managed
through the database
26
Polyglot Persistence Solution ArchitectureMongoDB and RDBMS Work Synchronously
27
Pers
iste
nce
Tra
nsa
ctio
n
Polyglot Solution Architecture with Referential Integrity
Persistence Service
Data Entry
Transaction
MongoDB RDBMS
Poly
glo
t Pe
rsis
ten
ce A
rch
itec
ture
Virtual Relational Integrity
28
Polyglot Persistence Provides aVersatile Data Solution
MongoDB and RDBMS working in harmony
29
Leveraging Both Storage PlatformsEnables Scalability, Performance, and Agility
MongoDB and RDBMS each have their place, using both together increases flexibility and growth
RDBMSHighly Relational or Structured
Transactional Data
Loose Cohesion
Low VolatilityRDBMS MongoDB
MongoDBUnstructured
Unknown Metadata
Tight Cohesion
High Volatility
Large Data
30
Branched Consumer Data with Document Leaf NodesThe complete consumer is much like a tree and its leaves
Credit Files
Audit Logs Alerts
Configurations
The polyglot persistence architecture leverages the
strengths of both storage technologies. The natural
structure of the consumer and the product catalog dictate
relational, while the products to be fulfilled are highly
cohesive documents.
The consumer tree and the document nodes.
A tree and its leaves
31
WHAT WHY WHERE WHEN WHO HOW
Q&AMongoDB and RDBMS: Using Polyglot Persistence at Equifax