Post on 19-Aug-2014
description
Use case for using the ElastiCache for Redis
in Production.
Table of contents
• Self-introduction• The minimum knowledge needed Redis
operation• gumi’s Redis Operation• Warning for operating ElastiCache for Redis• Summary
Table of contents
• Self-introduction• The minimum knowledge of Redis operation• gumi’s Redis Operation• Warning for operating ElastiCache for Redis• Summary
Self-introduction
• Tomonori Honma • gumi Inc.
• System Operation Engineer• All Game Server’s operarion in Japan• I don’t develop Game Events
• My favorite AWS Service:Amazon S3 @CkReal
Table of contents
• Self-introduction• The minimum knowledge needed Redis
operation• gumi’s Redis Operation• Warning for operating ElastiCache for Redis• Summary
The minimum knowledge of Redis operation
• Key Value Store• In-memory is very fast
• When is storing data over memory in Server, is controlled by maxmemory-policy(Parameters Group) settings. (default:volatile-lru)
• Two methods for Data Persistence• Append-only file• RDB file
• Can use multiple databases
Table of contents
• Self-introduction• The minimum knowledge needed Redis
operation• gumi’s Redis Operation• Warning for operating ElastiCache for Redis• Summary
gumi's Previous Generation Architecture
memcached
RDS(MySQL5.5)
app
player-shard
others
staticjobqredis
M
S
M
S
masterSQS
gumi's RecentGeneration Architecture
redis
RDS(MySQL5.6)
app
jobq
S
SQSstatic(S3)
The Use caseof Application
• User’s actions in game events can be immediately reflected in Redis.
• example• Realtime Ranking• Matching process of each player• Card Deck Information
The Use caseof Operation
• Error Tracking Tool(Sentry) use as a Buffer• Recent Operation tools sometimes uses Redis
• For quick checks, EC2 is very troublesome.
gumi’s Redis Operation
• Using almost master-slave• A few games use master only
• master : AOF and no RDB, slave : AOF and RDB• One GET operation takes sometimes 2.47sec???• Fork time in different systems
• Multiple backups of RDB are saved• This is to avoid problem where master DB is
flushed and that wipes out slave backup too.• The use multiple databases in one Redis node• Instead of memcached, use redis
• Developer is difficult to understand the timing of memcached's eviction.
Table of contents
• Self-introduction• The minimum knowledge needed Redis
operation• gumi’s Redis Operation• Warning for operating ElastiCache for Redis• Summary
Possible and Not (yet) possible
Possible Not (yet) possible
No EC2 instance!!Import data file from S3Configure AOF settingUse awscliPromotion of replica node
Auto failover like RDSBackup RDB file to S3Change instance typePoint-In-Time-Recovery
Possible and Not (yet) possible
Possible Not (yet) possible
No EC2 instance!!Import data file from S3Configure AOF settingUse awscliPromotion of replica node
Auto failover like RDSBackup RDB file to S3Change instance typePoint-In-Time-Recovery
2014/04/24 ReleaseBackup time includes time
taken to upload RDB file to S3?EC2 : few min
ElastiCache : Tens of minutes
Possible and Not (yet) possible
Possible Not (yet) possible
No EC2 instance!!Import data file from S3Configure AOF settingUse awscliPromotion of replica node
Auto failover like RDSBackup RDB file to S3Change instance typePoint-In-Time-Recovery
Possible and Not (yet) possible
Possible Not (yet) possible
No EC2 instance!!Import data file from S3Configure AOF settingUse awscliPromotion of replica node
Auto failover like RDSBackup RDB file to S3Change instance typePoint-In-Time-Recovery
ElastiCache for Redis is still nothigh-performance like RDS
ElastiCache for Redisimpression
• There is nothing wrong in performance• When you create a snapshot, it is recommended to run
from slave• Backup and Restore ElastiCache Redis Clusters
• appendonly settings is not available in cache.t1.micro• We have not experienced a big problem with
ElastiCache for Redis• When AOF file is corrupted, I have restored from
RDB file of slave server.
Table of contents
• Self-introduction• The minimum knowledge needed Redis
operation• gumi’s Redis Operation• Warning for operating ElastiCache for Redis• Summary
Summary
• Can prepare redis nodes in a short period of time• Since we have very few operation enginners in
our company, this is very helpful :)• Initially, functions we assumed to exist actually
were not yet implemented• But feature set has grown
• Currently, there are no big problems• we are increasing the number of production nodes
Thank you!!