Monitoring the Dynamic Nature of the Cloud [FutureStack16 NYC]
"Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew."...
-
Upload
new-relic -
Category
Data & Analytics
-
view
569 -
download
2
Transcript of "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew."...
![Page 1: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/1.jpg)
Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew.
Jonathan Owens, Senior Site Reliability Engineer & Jose Fernandez, Lead Software Engineer
![Page 2: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/2.jpg)
This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,”, “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.
![Page 3: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/3.jpg)
![Page 4: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/4.jpg)
![Page 5: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/5.jpg)
Confidential ©2008-15 New Relic, Inc. All rights reserved.
GOOD PROBLEM
![Page 6: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/6.jpg)
Confidential ©2008-15 New Relic, Inc. All rights reserved.
6
![Page 7: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/7.jpg)
Shopping list for replacing a Monolith
New Relic APM + Insights
Kafka
Microservices
DevOps
Containers
![Page 8: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/8.jpg)
CollectCollectCollectCollectCollect
![Page 9: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/9.jpg)
Collect
Collect
Collect
CollectCollect
![Page 10: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/10.jpg)
Collect
Collect
Collect
Collect
Collect
![Page 11: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/11.jpg)
![Page 12: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/12.jpg)
Great job refactoring services this year!“– No Manager Ever
![Page 13: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/13.jpg)
Updating last-seen timestamps for metrics
Metric ID resolution
Computing cluster agent metrics
Writing metric metadata
Writing summary records
Applying black / whitelist rules to accounts
Processing and storing environment values
Legacy alerting
Agent connect() calls
Agent run state
Orchestrating thread profiles
Applying rename rules to incoming metrics
Aggregation of 1-minute and 1-hour timeslices
from raw harvestsWriting 1-minute and 1-hour
timeslices to MySQL
![Page 14: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/14.jpg)
Updating last-seen timestamps for metrics
Metric ID resolution
Computing cluster agent metrics
Writing metric metadata
Writing summary records
Applying black / whitelist rules to accounts
Processing and storing environment values
Agent connect() calls
Agent run state
Orchestrating thread profiles
Applying rename rules to incoming metrics
Legacy alertingAggregation of 1-minute
and 1-hour timeslices from raw harvests
Writing 1-minute and 1-hour timeslices to MySQL
![Page 15: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/15.jpg)
Shopping list for replacing a Monolith
New Relic APM + Insights
Kafka
Microservices
DevOps
Containers
![Page 16: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/16.jpg)
Collect ConsumeProcess
Process
![Page 17: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/17.jpg)
Tap into the monolith and emit data for your use
![Page 18: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/18.jpg)
collector minuteaggregator
houraggregator
hourwriter
houraggregator
![Page 19: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/19.jpg)
collector minuteaggregator
houraggregator
hourwriter
houraggregator
![Page 20: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/20.jpg)
Validate your data
MySQL
query
QueryService
“In the dark”
Insights
Cassandra
query, result, and performance
QueryValidator
![Page 21: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/21.jpg)
MySQL
query
QueryService
“In the dark”
Insights
Cassandra
query, result, and performance
QueryValidator
![Page 22: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/22.jpg)
Shopping lists for replacing a Monolith
New Relic APM + Insights
Kafka
Microservices
DevOps
Containers
![Page 23: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/23.jpg)
collector minuteaggregator
hourwriter
![Page 24: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/24.jpg)
collector minuteaggregator
houraggregator
hourwriter
houraggregator
![Page 25: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/25.jpg)
collector minuteaggregator
houraggregator
hourwriter
houraggregator
minuteaggregator
summaryrecord
generator
![Page 26: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/26.jpg)
![Page 27: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/27.jpg)
Experimentally proven value
Service boundaries match communication boundaries
![Page 28: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/28.jpg)
Photo credit: theimpulsivebuy
cc by Aranami http://www.flickr.com/photos/z0/
![Page 29: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/29.jpg)
![Page 30: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/30.jpg)
Shopping list for replacing a Monolith
New Relic APM + Insights
Kafka
Microservices
DevOps
Containers
![Page 31: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/31.jpg)
![Page 32: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/32.jpg)
Alice GoldfussSite Reliability Engineer New Relic @alicegoldfuss
DevOps means empathy and communication over the wall until there is no more wall.
![Page 33: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/33.jpg)
JonathanOwens
JoseFernandez
OpsDev
… long wait… frustration due
to lack of visibility…
cyclerepeats
“I need 3 Cassandra servers
ASAP!!”(vague request)
“Here they are. Sorry it took so long.
Our queue was backed up.”
<different goals>
“Doh. I needed the
newer version of Cassandra…”
![Page 34: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/34.jpg)
OpsDev
![Page 35: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/35.jpg)
Team Culture
![Page 36: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/36.jpg)
How did we do it? How can you do it?
* insert picture for Upscale event
Upscale
![Page 37: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/37.jpg)
Product Reliability Engineer
Site Reliability Engineer
![Page 38: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/38.jpg)
Shopping list for replacing a Monolith
New Relic APM + Insights
Kafka
Microservices
DevOps
Containers
![Page 39: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/39.jpg)
Docker is opinionated about software architecture in a way that encourages more robustly crafted applications.
“– Karl Matthias & Sean P. Kane
Docker: Up and Running
![Page 40: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/40.jpg)
PUSH and then
PULL door
Push plate?
Policy statement
Push on a pull handle
Visual instruction to use badge
(with light indicator)
![Page 41: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/41.jpg)
PUSH PUSH PULL
![Page 42: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/42.jpg)
This part fits the problem
This part fits the person
![Page 43: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/43.jpg)
![Page 44: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/44.jpg)
![Page 45: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/45.jpg)
Start Small
![Page 46: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/46.jpg)
Shopping list for replacing a Monolith
New Relic APM + Insights
Kafka
Microservices
DevOps
Containers
![Page 47: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/47.jpg)
![Page 48: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/48.jpg)
Q&A
![Page 49: "Containers, DevOps, Microservices and Kafka: Tools used by our Monolith wrecking crew." [FutureStack16]](https://reader031.fdocuments.us/reader031/viewer/2022030317/586e8b7e1a28aba0038b8067/html5/thumbnails/49.jpg)
Thank you.
Jonathan Owens, Senior Site Reliability Engineer & Jose Fernandez, Lead Software Engineer