Workers and Worker Patterns at Scale
-
Upload
chad-arimura -
Category
Technology
-
view
922 -
download
2
Transcript of Workers and Worker Patterns at Scale
![Page 1: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/1.jpg)
Workers and Worker Patternsat Scale
Workers and Worker Patterns
![Page 2: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/2.jpg)
● 10 team members and growing (all developers)● Funded by Baseline Ventures, Ignition Partners and
other leading cloud angels● Makers of IronMQ and IronWorker● Multi-cloud, multi-platform, multi-language
Who we are
![Page 3: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/3.jpg)
Design Patterns: Recurring solutions to common problems in software design.
![Page 4: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/4.jpg)
Worker Patterns
○ Small Independent Unit of Work○ Think Concurrently○ Optimal Worker Durations○ Scheduling & Quarterbacking○ Pass ID's not large Objects○ Log Everything
![Page 5: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/5.jpg)
Small Independent Unit of Work
Pattern: Keep workers task specific and as light as possible with one purpose.
Much easier to maintain and scale.
![Page 6: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/6.jpg)
Small Independent Unit of Work
![Page 7: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/7.jpg)
Think ConcurrentlyPattern: Designed as collections of non-interacting workers that may be executed in parallel.
Independent and self-sufficient.
Loosely coupled, no shared state.
![Page 8: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/8.jpg)
Think Concurrently
![Page 9: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/9.jpg)
Optimal Task Durations
Pattern: Make task long enough to take advantage of initial setup (db connections, etc), but short enough to be able to quickly fix and retry on errors.
Too short doesn't make sense for worker setup/teardown time (db connections, etc). If too short, batch several tasks into one.
Too long increases opportunity for failure and complexity of retry.
![Page 10: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/10.jpg)
Scheduling & Quarterbacking
Pattern: Create a low number of scheduled jobs and queue up lots of concurrent jobs.
![Page 11: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/11.jpg)
Scheduling & Quarterbacking
![Page 12: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/12.jpg)
Pass IDs not large Objects
Pattern: Pass the minimum amount of information necessary for your worker to run. Then get the full data while it's running.
● Better performance while queuing up tasks● Get latest data● Much easier to maintain state on that object.
![Page 13: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/13.jpg)
Pass IDs not large Objects
![Page 14: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/14.jpg)
Log EverythingPattern: Log everything using a cloud logger or other new global logger.
loggly.com, papertrail.com, IronWorker logging
![Page 15: Workers and Worker Patterns at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052907/558fc6741a28ab744e8b4641/html5/thumbnails/15.jpg)
Contact:
Chad [email protected]
skype: chadarimuratwitter: chadarimura
Iron.io Confidential (v1.1)
Contact Information