Всеволод Поляков (DevOps Team Lead в Grammarly)
-
Upload
provectus -
Category
Engineering
-
view
305 -
download
0
Transcript of Всеволод Поляков (DevOps Team Lead в Grammarly)
what?• RRD-like (gram.ly/gfsx)• so.it.is.my.metric → /so/it/is/my/metric.wsp• Fixed retention (by name\pattern)• Fixed size (actually no)
Retention and size• 1s:1d → 1 036 828 bytes• 10s:10d → 1 036 828 bytes• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)• 10s:365d → 37 843 228 bytes (1 TB ~ 30 000)
whisper calc
Retention and size• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes• 240 864 metrics in 1 TB• aggregation: average, sum, min, max, and last.• can be assign per metric
How• terraform (https://www.terraform.io/)• docker (https://www.docker.com/)• ansible (https://www.ansible.com/)• rocker (https://github.com/grammarly/rocker)• rocker-compose (
https://github.com/grammarly/rocker-compose)
carbon-cache.py• single-core• many options in config file• default
link
Start load testing• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)• retentions = 1s:1d• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND, MAX_CREATES_PER_MINUTE = inf• defaults• almost 1.5h to get limit :(
Try to tune!
• WHISPER_SPARSE_CREATE = true (don’t allocate space on creation) non-linear I\O load.
• CACHE_WRITE_STRATEGY = sorted (default)
Try to tune!
• CACHE_WRITE_STRATEGY = max will give a strong flush preference to frequently updated metrics and will also reduce random file-io.
go-carbon• multi-core single daemon• written in golang• not many options to tune :(
link
Start load testing• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)• retentions = 1s:1d• max-size = 0• max-updates-per-second = 0• almost 1h to get limit :(
try to tune!remaining := whisper.Size() - whisper.MetadataSize()whisper.file.Seek(int64(remaining-1), 0)whisper.file.Write([]byte{0})chunkSize := 16384zeros := make([]byte, chunkSize)for remaining > chunkSize {
// if _, err = whisper.file.Write(zeros); err != nil {// return nil, err// }remaining -= chunkSize
}if _, err = whisper.file.Write(zeros[:remaining]); err != nil {
return nil, err}
Start load testing• c4.xlarge instance (4 CPU, 7.5 GB ram)• ~1 Gb lan• default parameters• hashing• 10 connections
carbon-relay-ng• golang-based• web-panel• live-updates• aggregators• spooling
link
ALSO• zipper:
• https://github.com/grobian/carbonserver• https://github.com/grobian/carbonwriter• https://github.com/dgryski/carbonzipper• https://github.com/dgryski/carbonapi• https://github.com/dgryski/carbonmem
• https://github.com/jssjr/carbonate
feel free to ask• Vsevolod Polyakov• [email protected]• skype: ctrlok1987• github.com/ctrlok• twitter.com/ctrlok• slack: HangOps• Gitter: dev_ua/devops• skype: DevOps from Ukraine• slack.ukrops.club