How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
-
Upload
ontico -
Category
Engineering
-
view
771 -
download
8
Transcript of How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
Copyright © GigaSpaces 2015. All rights reserved.Copyright © GigaSpaces 2015. All rights reserved.
How to run Real Time processing on Big DataRon ZavnerTechnical Director, EMEA Gigaspaces
Copyright © GigaSpaces 2015. All rights reserved. 2
About GigaSpaces
GigaSpaces provides software middleware for deployment, management and scaling of mission-critical applications on
cloud environments.
GigaSpaces serves more than 500 large enterprises & ISVs, over 50 of which are Fortune-listed.
Direct customers300+
Fortune / Organizations40+ / 500+
Cloud Customers75+
ISVs25+
Copyright © GigaSpaces 2015. All rights reserved. 3
Agenda• The challenge of real time analytics today• Introduction to In Memory Data Grids• Meet XAP• XAP on Big Data (plus advanced features)• Use cases + Demo
Copyright © GigaSpaces 2015. All rights reserved. 4
In today’s realityinformation is gathered from numerous sources
Copyright © GigaSpaces 2015. All rights reserved. 5
• Number of transactions grows exponentially• Tolerance for system response time reduces significantly
Copyright © GigaSpaces 2015. All rights reserved. 6
In order to gain a competitive edge, organizations need real time processing of large data sets
Copyright © GigaSpaces 2015. All rights reserved. 7
Alongside this challenge, there are many other challenges for management of enterprise applications
Such as:
Copyright © GigaSpaces 2015. All rights reserved. 8
Current tier based architecture cannot meet these challenges
Copyright © GigaSpaces 2015. All rights reserved. 9
Lets go over these challenges and see why
Copyright © GigaSpaces 2015. All rights reserved. 10
Peak Loads ChallengesYour messaging tier can only scale so far…
Result: Costly over-provisioning
Massive over-provisioning of resources only to meet peak loads
Your database can only scale so far…
Copyright © GigaSpaces 2015. All rights reserved. 11
Performance Challenges
Disk throughput limitations and bottlenecks
Network bottlenecks
Result: Higher latency for your business transactions, which can be very costly
Copyright © GigaSpaces 2015. All rights reserved. 12
Business Continuity Challenges
Too many moving parts
Result: Application becomes more error-prone and harder to troubleshoot
Copyright © GigaSpaces 2015. All rights reserved. 13
TCO Reduction Challenges
Result: Having many moving parts means higher operational costs (purchase, update, maintain, etc).
Copyright © GigaSpaces 2015. All rights reserved. 14
Real-Time Business Insight Challenges
Result: If performance is not ideal, than real time response isn’t either.
In today’s reality, real time event processing of constantly growing data sets gives your company the competitive edge it needs.
Copyright © GigaSpaces 2015. All rights reserved. 15
Tier-based architecture cannot solve today's data & application processing issues
Copyright © GigaSpaces 2015. All rights reserved. 16
Meets All These Challenges
Copyright © GigaSpaces 2015. All rights reserved. 17
XAP scales the Data Tier using its In-Memory Data Grid so you can access your data in real time
Copyright © GigaSpaces 2015. All rights reserved. 18
XAP scales the Data Tier using its In-Memory Data Grid
The database goes to the background
Partition your data and store it in memory
Copyright © GigaSpaces 2015. All rights reserved. 19
Same goes for the Messaging Tier
Partitioned, co-located in-memory messaging
Copyright © GigaSpaces 2015. All rights reserved. 20
XAP scales the entire application – so you get extreme processing of your big data and get real time insights
Business logic, data & messaging co-located & partitioned into processing units
Copyright © GigaSpaces 2015. All rights reserved. 21
XAP ensures High Availability
Hot backup for each partition for high availability
Copyright © GigaSpaces 2015. All rights reserved. 22
Scale the Web Tier
Host your web application on the XAP infrastructure
Copyright © GigaSpaces 2015. All rights reserved. 23
XAP enables auto-scaling of the entire application on demand
Auto-scale out & in based on real-time performance & load
Copyright © GigaSpaces 2015. All rights reserved. 24
Result:Real time processing & analytics of your big data with XAP IMC
Copyright © GigaSpaces 2015. All rights reserved. 25
What can XAP do for you?
Scaling the Data Tier
Multi-site deployment & DR across remote sites
Batch processing of large data sets
Online transaction processing
Real time querying and analysis of large datasets
Real time processing of large event stream
Scaling the Web Tier
Copyright © GigaSpaces 2015. All rights reserved.
Some features…Complex Queries Support
Scala Support
Data Consistency Level
Increase developer productivity and application scalability using new syntax for complex nested conditions
Comprehensive Scala support including Tasks, Objects, APIs and Queries
Update Multiple Entries to different partition with timeout to cope with locks
Optimize XAP to your business requirements setting the Data Consistency Level
Optimized Cross Partition Update
IPv6 LRMI Filters
Support for IPv6 Network with XAP deployments
Web GUI can serve hundreds of nodes with good UX
Allow for encryption and compression for selected LRMI connection
Scalable Web GUI
Better support for customized user credentials
Extended Custom Security
Copyright © GigaSpaces 2015. All rights reserved.
and more…
27
Microsoft Linq providerSupport for queries written using Linq API and syntax
MongoDB External Data Source
Bi directional data and metadata exchange between MongoDB and XAP
Unique constrains in IndexSupport for unique validation of objects in index in each partition
Change API EnhancementsAdditional Change API returns the valuethat had been changed
Advanced Projections supportPartial queries on nested objects
Immutable Objects Support
Zone controlled deploymentSupport for mapping primary nodes into specific zone
Copyright © GigaSpaces 2015. All rights reserved.28
AggregationFramework
Custom Change Operations
Enhanced Initial Data Load
Management & Deployment Automation
Query Analysis
Memory Xtend for SSD
Global HTTP Session Sharing
and more…
Copyright © GigaSpaces 2015. All rights reserved.
Aggregations – Java
29
Copyright © GigaSpaces 2015. All rights reserved.
Aggregations – .Net (using LINQ)
30
Copyright © GigaSpaces 2015. All rights reserved.
Change Operation
31
Copyright © GigaSpaces 2015. All rights reserved.
Custom Change Operation – Java
32
Copyright © GigaSpaces 2015. All rights reserved. 33
The In-Memory Computing Platform Extreme scaling solution across multiple verticles
Copyright © GigaSpaces 2015. All rights reserved. 35
Multi-Site OperationsBy Synchronization of dynamic data
across remote sites
Multi-Site Data Replication Across Remote Data Centers
Real Time Data Processing & AnalysisBy providing a secured private data cloud
Out of the box Disaster Recovery PlanningData is up-to-date across all sites
Near real-time replication of massive data streams – synchronize data across your different sites in a consistent, failure-proof,
scalable way.
Copyright © GigaSpaces 2015. All rights reserved. 36
Selected Customers
Copyright © GigaSpaces 2015. All rights reserved. 37
E-Commerce
Industry
“GigaSpaces XAP performed beautifully, easily withstanding Kohl’s Black Friday load. In fact, we are quite confident that GigaSpaces could handle Kohl’s growth for years to come”
Handle peak loads without over-provisioning for maximum traffic (following 2009 system crash resulting in loss of millions of $$)
Challenge
Implemented inventory management on top of XAP within 4 months
Solution
* Kohl’s was N. America’s best performing e-commerce website on Black Friday 2010Results
Copyright © GigaSpaces 2015. All rights reserved. 38
Banking
Industry
“GigaSpaces technology allows us to increase customer satisfaction by facilitating better, user-friendly services, as well as new services, which ultimately enhances our bottom line”
Scalable transaction processing for trading platform; increase performance and efficiency of banking processes.
Challenge
GigaSpaces provides infrastructure for Avanza’s core services, including trading, customer and data processing, and storage.
Solution
* Zero system cost for data storage (vs. half system capacity before)* Massive cost reduction* Major performance improvement
Results
Copyright © GigaSpaces 2015. All rights reserved. 39
Premium Edition
XAP In-MemoryLite Edition
Extreme app scaling & RT insights
Try for free!
Try and let us know what you think…
Copyright © GigaSpaces 2015. All rights reserved.
Check us out:
Email us:Call us:Follow us:
www.gigaspaces.comwww.getcloudify.orginfo@gigaspaces.com646-421-2830@GigaSpaces, @CloudifySource