Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get...
Transcript of Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get...
![Page 1: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/1.jpg)
Moving Data Analytics to the Cloud
"The views expressed in this presentation are those of the author(s) and do not necessarily reflect the official policy or position of the Air Force, the Department of Defense, or the U.S. Government."
![Page 2: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/2.jpg)
2
• Why the Cloud• Obstacles to Cloud Migration• Our Cloud Data Analytics Experience• Use Case & Strategy Examples• Demonstration AWS Tools & Services• Cloud Migration Demonstration• Cloud Analytics Demonstration
Discussion Topics Cloud Migration Agenda
![Page 3: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/3.jpg)
3
Reasons to Migrate/Why the Cloud
• Reliable backup/recovery planning and implementation• Improved control of development & test environments
• Faster transition of access• Standardize repository use & configuration management for version control• Much finer control of development & test environment costs• Promotes prototyping without long term capital investment
• Much simpler vertical and horizontal scaling• Storage optimization• IO optimization• Compute optimization
• Move from license-centric platforms to service-centric platforms (open source)• Architecting & provisioning unique or ‘fit for purpose” capabilities
![Page 4: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/4.jpg)
4
So what do we do?
• Some easy wins for the cloud
• Program managed development and test environments• Web service/API integration
• Migration/export for data analytics & reporting
![Page 5: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/5.jpg)
5
Obstacles to Migration• Fear of losing control of “my system”
• System is still there to be managed by you or your designee• Security Overload – FedRAMP, NIST, RMF etc.…
• Underlying system is similar and sometimes the same• Majority of your vendors are there• Forward leaning technical solutions face the same issues OnPrem• Cloud is typically more secure than OnPrem out of the box
• Acquisition Process• Treat as modernization or Tech Refresh• CCE migration team
• Fork Lift vs Refactor Freeze• Fork Lift usually faster• Refactor takes more native advantage of cloud architecture
• Complex legacy systems with large data sets• Many aspects of these systems don’t need to be complex or legacy• Data is extremely portable
![Page 6: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/6.jpg)
6
Research Data Exchange (RDE)Project Background
• Platform to transportation data sharing system• Uses IOT devices to gather vehicle based traffic and route based metrics• Includes Real-Time Data Capture• Supports a wide range of development testing and demonstration scenarios for multi-modal
transportation applications• Uses AWS Infrastructure• Effectively manages compute costs• Promotes data use by simplifying data
capture and standardized exposure APIs
![Page 7: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/7.jpg)
7
Data Analytics Use Case
• Have a legacy highly transactional system• Want to expand reporting and analysis capability• Both reporting and analysis capabilities are on demand• Have both canned/scheduled reports and ad hoc queries• Want flexibility in analysis data content• Don’t have the time/resources to move the existing large scale application• Don’t want capital investment associated with hardware and software buildout such as a
traditional Data Warehouse or Data Lake
SQL DBCurrent Data
StoresExisting
Application
Ad Hoc Storage
ExtractMigrate Process/
Analyze Expose
![Page 8: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/8.jpg)
8
Choosing AWS as the cloud solution
• Demonstration AWS Tools & Services
• AWS Direct Connect• AWS EC2 Memory Optimized Instance• AWS DMS Data Migration Service• AWS Schema Conversion Tool• AWS Aurora & Athena• AWS Glue• AWS S3• AWS QuickSight
![Page 9: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/9.jpg)
9
Use Cases for Migration or Hybrid• Example per purpose database
![Page 10: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/10.jpg)
10
Cloud Migration Approach
![Page 11: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/11.jpg)
11
Cloud Migration Demonstration
To AWS Data Migration Demonstration
https://console.aws.amazon.com/console/home?region=us-east-1#
![Page 12: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/12.jpg)
12
Cloud Analytics Approach
![Page 13: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/13.jpg)
13
Cloud Data Analytics Demonstration
To AWS Data Analytics Demonstration
https://console.aws.amazon.com/console/home?region=us-east-1#
![Page 14: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/14.jpg)
14
Summary
• The Cloud offers a lot of advantages over on-prem.• There are several “easy win” use cases to get started.• The obstacles for cloud migration aren’t as daunting as it initially looks.• If you can’t move your current system, you can ‘fit for purpose’ instead of further tying
yourself to the legacy system.• Data Analytics & Reporting are a simple way to get started.• Tooling for Data Migration is excellent.• Tooling for Data Analytics and Reporting is simple to use and powerful.• IndraSoft is ready, willing and able to help you through the process!
![Page 15: Moving Data Analytics to the Cloud · •Data Analytics & Reporting are a simple way to get started. •Tooling for Data Migration is excellent. •Tooling for Data Analytics and](https://reader034.fdocuments.us/reader034/viewer/2022042216/5ebd9c12b3e6e54a6d6f7d9f/html5/thumbnails/15.jpg)
15
Questions