"Cloud databases amazon web services" by Roman Gomolko
-
Upload
ciklum -
Category
Technology
-
view
446 -
download
0
Transcript of "Cloud databases amazon web services" by Roman Gomolko
![Page 2: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/2.jpg)
Let’s get acquired
![Page 3: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/3.jpg)
UserReport
Developing products that allow to learn the audienceStarted using AWS more than 5 years agoFully migrated to AWS more than 1.5 years agoProcessing 3 billions requests monthlyGenerating reports based on 8 billions of requests with batched
reportsOnline reports on 300 millions of recordsUsed ~50% of services provided by AWSTotally happy regarding using AWS
![Page 4: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/4.jpg)
A database is an organized collection of data
![Page 5: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/5.jpg)
RDS
Relational Databases hosted and maintained by Amazon
![Page 6: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/6.jpg)
Different Engines & Editions & Versions
![Page 7: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/7.jpg)
Captain Obvious’s notes
●RDS doesn’t host particular DB but it hosts RDMS●Create your root user, create separate users for each
database/application●Your instance is firewalled with security groups●Advanced configuration is available through parameter groups
![Page 8: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/8.jpg)
Multi A-Z deployments for production workloads
●SLA 99.95% monthly uptime●Doubles prices●Allows to maintain your database without downtime
○ Minor updates○ Major updates○ Disk resize○ EC2 upgrade
●No support for MS SQL Web, Express, Standard
![Page 9: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/9.jpg)
Pricing
RDS price = EC2 + ELB + license
On-Demand or Reserved purchases with up-front payment
![Page 10: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/10.jpg)
Backups
●Automated with automated rotation●Restore to point of time●Restore will create new instance and deploy desired version. It
takes a while●Manual backup via Snapshots
![Page 11: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/11.jpg)
Advanced optimizations
●Read replicas○ you can create on the fly high available read-only copies of your
data●Using ElastiCache for performance boost
○ Using memcache will massively boost your queries
![Page 12: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/12.jpg)
Downsides
●No control over EC2 for very advanced optimizations●Backup works over instance
○ One RDS per DB○ Or custom backups
●No Active Directory integration●No Cross-region replication
![Page 13: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/13.jpg)
Aurora
MySQL compatible database by Amazon with cloud in the mind
![Page 14: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/14.jpg)
Aurora
Available and DurableAmazon Aurora is designed to offer greater than 99.99% availability, replicating 6 copies of data across 3 Availability Zones and backing up data continuously to Amazon S3. Recovery from physical storage failures is transparent and instance restarts typically require less than a minute.
![Page 15: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/15.jpg)
Aurora
Highly ScalableYou can use Amazon RDS to scale your Amazon Aurora database instance up to 32 vCPUs and 244GiB Memory. You can also add up to 15 Amazon Aurora Replicas across three availability zones to further scale read capacity. Amazon Aurora automatically grows storage as needed, from 10GB up to 64TB.
![Page 16: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/16.jpg)
DynamoDB
Document database with biscuits by Amazon
![Page 17: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/17.jpg)
DynamoDB overview
●Operates with tables●Table definition consist of
○ key (required)○ sort (range) key (optional)○ indexes (optional)
●Table contains items●Items is described by
○ key ○ sort (range) key (if defined for table)○ attributes
![Page 18: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/18.jpg)
DynamoDB item overview
●Max 64 KB●Unlimited number of attributes●Attribute types
○ string○ string array○ number○ number array○ binary○ binary array○ json
![Page 19: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/19.jpg)
DynamoDB operations
●Put - insert or update●Get●Delete●Scan ●Query
![Page 20: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/20.jpg)
Demo time
DynamoDB show-case
![Page 21: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/21.jpg)
DynamoDB performance
●You provision read and write capacity●DynamoDB is divided into shards. Each shard has following
limits:○ 2 Gb of data○ 3000 Read Capacity Units○ 2000 Write Capacity Units
●Your requests can be throttled (API cares about retry-logic in most cases)
●You can setup autoscale of DynamoDB
![Page 22: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/22.jpg)
DynamoDB Streams
●Triggers on data changes●Cross-region replication●ElasticSearch integration to allow to search among your data
https://aws.amazon.com/blogs/aws/new-logstash-plugin-search-dynamodb-content-using-elasticsearch/
![Page 23: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/23.jpg)
Backups and maintenance
●All data is replicated on three nodes - no backup required●Change of provisioned throughput does not downgrade
performance●You can setup AutoScale for DynamoDB
https://github.com/sebdah/dynamic-dynamodb
![Page 24: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/24.jpg)
*hit happens
DynamoDB had massive outage (high error rate on API request) in N. Verginia that affected:
●SQS●CloudWatch●AutoScale Groups●SNS
https://aws.amazon.com/message/5467D2/
![Page 26: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/26.jpg)
ElastiCache
Key-value store is also database
![Page 27: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/27.jpg)
Redis
●Extremely fast in-memory database●Different data structures
○ Sets○ Lists○ Ordered sets○ HyperLogLog○ HashSets○ Geo data○ Pub/Sub
![Page 28: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/28.jpg)
Redis hosted in AWS
●Different versions supported●Multi AZ master/slave configuration maintained by Amazon●Automated backups●Monitoring with CloudWatch●No chance to patch Redis for your needs (geeks like custom
operations)
![Page 29: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/29.jpg)
Example 1. Calculating unique visitors
PFADD visitors.20151001 xxxPFCOUNT visitors.20151001
INC pageviews.20151001GET pageviews.20151001
![Page 30: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/30.jpg)
Example 2. Working with sets
# users 1 and 2 add item to basketSADD added_item_to_cart id1SADD added_item_to_cart id2SADD begin_checkout id1# users haven’t began checkoutSDIFFSTORE no_checkout added_item_to_cart begin_checkout# users with email and haven’t started checkoutSINTER known_email no_checkout
![Page 31: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/31.jpg)
Example 3. Top scored users
ZADD gamescore 1 user1ZADD gamescore 4 user2ZADD gamescore 2 user3ZREVRANGE gamescore 0 9
user2user3user1
![Page 33: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/33.jpg)
Redshift
It’s like PostgreSQL but for peta-bytes
![Page 34: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/34.jpg)
Redshift
●Multiple-node cluster deployment that scales up to petabytes●$1000/Tb/year●Good for data mining●Query execution minutes or hours
![Page 35: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/35.jpg)
Table design
●HashKey - how data will be distributed across nodes●SortKey - how data will be sorted within node●Primary key, foreign keys, constraints - they are hints to query
optimizer
![Page 36: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/36.jpg)
Uploading data
●From CSV●From DynamoDB●From EMR●Bulk insert
http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
![Page 37: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/37.jpg)
Loading data from S3
copy tablefrom 's3://mybucket/data/table.txt' credentials 'aws_access_key_id=<access-key-id>;aws_secret_access_key=<secret-access-key>' csv [gzip] [delimiter "|"];
![Page 38: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/38.jpg)
Query Execution
●PostgreSQL compatible syntax with many disabled features●No views●No stored procedures●Recently deployed scalar custom functions●10 parallel queries
![Page 39: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/39.jpg)
Getting query results
unload ('select * from mytable)to 's3://mybucket/unload/result/' credentials 'aws_access_key_id=<access-key-id>;aws_secret_access_key=<secret-access-key>';
![Page 40: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/40.jpg)
S3 + EMR
Why don’t query files?
![Page 42: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/42.jpg)
EMR
EMR can launch Elastic Map Reduce cluster so●Hadoop●Spark●Hive●Presto
Distributed SQL Query Engine for Big Data
![Page 43: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/43.jpg)
Demo time
![Page 45: "Cloud databases amazon web services" by Roman Gomolko](https://reader036.fdocuments.us/reader036/viewer/2022062904/588401661a28ab884b8b5975/html5/thumbnails/45.jpg)