Scality S3 Server: Node js Meetup Presentation
-
Upload
scality -
Category
Technology
-
view
232 -
download
2
Transcript of Scality S3 Server: Node js Meetup Presentation
![Page 1: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/1.jpg)
CONFIDENTIAL - FOR GARTNER USE ONLY © Scality 20161
Node.js @ScalityExperiences and Lessons Learned
Giorgio Regni, CTOLauren Spiegel, Software Engineer
![Page 2: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/2.jpg)
Disrupting storage – unlimited & everywhere
![Page 3: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/3.jpg)
When to use object storage?
1. Need for capacities beyond 100 TB and growing fast
2. Very large number of clients accessing isolated data
3. Object must be > 100KB, otherwise use a Database
Bucket 1
Object A
Object B
Object C
Bucket 2
Object A
Object B
Object …
Object Z
![Page 4: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/4.jpg)
Our first Node.js project
![Page 5: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/5.jpg)
Copyright Scality 2014Copyright Scality 2014
Our first node.js project - Building a Tivo in the Cloud• 25 million users -> Designed for high degree of
parallelism
• TB/sec –> Need very efficient network transfer
• Scales out by adding nodes and drives
• Proved 30 GB/sec of ingest with 10 servers and 360 drives
Test Case Latency (seconds)
Duration Recordings Batch Size Sockets RPM per Client
Threads per Client Average at 95% at 99%
2 hours 20,000 2500 1000 270 63 0.159 0.319 0.426
![Page 6: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/6.jpg)
The teamSF & Paris
![Page 7: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/7.jpg)
S3-Server AWS S3 compatible server Open source
https://github.com/scality/s3 Can use local storage
S3-MetaData A distributed metadata database service Supports fast Bucket & object listing Stores ACL and Users/Groups
S3-Vault Security, Identity & Authentication Service Provides Accounts/Keys Supports AWS IAM Users & Groups Interoperable with user directory services (via SAML)
What we built: Three Key Components
7
![Page 8: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/8.jpg)
What have we learned??
![Page 9: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/9.jpg)
Logging is hard
• Challenges• Logging is expensive as it taxes the Node.js process• UDP datagrams have expensive DNS lookups• Redundant transformations by bunyan and bunyan-logstash
• Solution: Werelogs• Produces raw JSON logs with the least resistive path• Forward logs to ELK using Filebeat for indexing• Avoids expensive and redundant transformations• Ability to track requests across the components with UIDs• Dump log history on errors
Open source -> http://github.com/scality/werelogs
![Page 10: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/10.jpg)
![Page 11: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/11.jpg)
Our first Node.Js project
![Page 12: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/12.jpg)
Performance, performance & performance
![Page 13: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/13.jpg)
The performance cycleCode, Benchmark, … Repeat
• Socket & Nagle algorithm on by default -> very high latencies
• The event loop can get backed up quickly -> hunt for all cpu intensive tasks in the main loop
• Buffers are much more efficient when writing server response
• Micro optimizations: Date.now() > new Date()• Beware of libraries doing way too many things for you• ES6 support, Babel5 was killing performance -> Babel6
![Page 14: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/14.jpg)
![Page 15: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/15.jpg)
Nifty Node Tools
![Page 16: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/16.jpg)
Nifty Node ToolsGetting going
• Airbnb JavaScript Style Guide + Eslint
• babel — babel5 to babel6 with just imports, destructuring and default parameters
• Commander — cool cli tools in minutes
• Async
![Page 17: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/17.jpg)
Nifty Node ToolsGetting serious
• Level — LevelDB wrapper for node
• Memcached — client library for node
• xml — <parse>yes</parse>
• Profiler — Go fast or go home
![Page 18: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/18.jpg)
Nifty Node ToolsMight as well test
• Mocha
• Istanbul
• lolex
• aws-node-sdk
![Page 19: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/19.jpg)
Nifty Node ToolsDocs and Open Source Code
• Docs are good, but
• Code is even better
• Read the readable stream code and take a nap.
• Then read the transform stream code and create new universes.
![Page 20: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/20.jpg)
What can you do with this thing?
![Page 21: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/21.jpg)
Download the code!
http://s3.scality.com/https://github.com/scality/s3
![Page 22: Scality S3 Server: Node js Meetup Presentation](https://reader035.fdocuments.us/reader035/viewer/2022062400/587ebc941a28abbb688b6f75/html5/thumbnails/22.jpg)
Lauren:github: laurenspiegeltwitter: @notfollowingyet
Giorgio:github: @giorgioregnitwitter: @giorgioregni