Managing Microservices with Neo4j
-
Upload
ashley-chloe -
Category
Technology
-
view
778 -
download
1
Transcript of Managing Microservices with Neo4j
![Page 1: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/1.jpg)
Managing Microservices with Neo4j
Rob SchoeningAshley Sun
![Page 2: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/2.jpg)
Overview
• Intro• MacGyver + Neo4j• Use Cases• Q&A
![Page 3: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/3.jpg)
• World’s largest online credit marketplace– Peer-to-peer personal loans– Small business loans– Medical financing
![Page 4: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/4.jpg)
Micro ServicesAll the cool kids are doing it…
…and it makes complex softwareeasier to write and maintain
![Page 5: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/5.jpg)
A long, long time ago…2 years ago…We ran 5 production services…Now we run…
139
![Page 6: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/6.jpg)
That’s a lot of stuff to manage!
Continuously!
![Page 7: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/7.jpg)
How do we do it?
![Page 8: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/8.jpg)
Excel
![Page 9: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/9.jpg)
ExcelJust Kidding
Neo4j, of course!
![Page 10: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/10.jpg)
So, like, seriously, how do we do it?
![Page 11: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/11.jpg)
MacGyver
![Page 12: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/12.jpg)
What is MacGyver?
“Is that like MacGruber?”
![Page 13: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/13.jpg)
What is MacGyver?
“Is that like MacGruber?”
No, but it’s on GitHub!
![Page 14: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/14.jpg)
MacGyver provides…• What we can’t easily accomplish with Jenkins• API Integration• A batch scheduler• Data aggregation• An extensible UI• Tools for engineering, operations, infosec, QA
and finance• Performs monitoring and alerting
![Page 15: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/15.jpg)
Integrations
![Page 16: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/16.jpg)
MacGyver+
Neo4j
![Page 17: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/17.jpg)
MacGyver+
Neo4j
Jenkins
GitHub
Load Balancer
Pager Duty SAN Storage
![Page 18: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/18.jpg)
MacGyver+
Neo4j
Jenkins
GitHub AWSJira
Confluence HipChat
Active Directory
Pager Duty
New RelicLoad Balancer
Pager Duty SAN Storage
Graphite
![Page 19: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/19.jpg)
MacGyver+
Neo4j
Jenkins
GitHub AWS
Artifactory
Jira
Confluence HipChat
VMWare
Active Directory
Pager Duty
New RelicLoad Balancer
Pager Duty
Catchpoint
SAN Storage
Graphite
Splunk
![Page 20: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/20.jpg)
Why Neo4j?
• Perfect for capturing/modeling interdependencies
• Cypher’s ad hoc query capability can’t be beat• Easy to extend, build more relationships and
layers incrementally• JSON native, FTW
![Page 21: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/21.jpg)
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
![Page 22: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/22.jpg)
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
![Page 23: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/23.jpg)
App Check-In/Service Discovery
Problem: Keeping track of many rapidly-changing services
Solution: All app servers phone home to MacGyver and are stored in Neo4j as ‘App Instance’ nodes
![Page 24: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/24.jpg)
App Check-In/Service Discovery
Neo4j
App Instance
App Instance
App Instance
MacGyver
![Page 25: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/25.jpg)
App Check-In/Service Discovery
MacGyver
Neo4j
App Instance
App Instance
App Instance
<<REST>><<REST>>
![Page 26: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/26.jpg)
App Check-In/Service Discovery
Real time database of deployed infrastructureNew services get auto-discovered by MacGyverLow maintenanceEasy scalabilityLow latency ad hoc query capability
![Page 27: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/27.jpg)
App Check-In/Service Discovery
“Show me all the instances of app X in environment Y”
![Page 28: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/28.jpg)
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
![Page 29: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/29.jpg)
Deployment and Release Automation
Problem: Highly manual and tedious releases Difficult to answer questions like:– What pool should I deploy to?– Is the most recent revision ‘live’ right now?– Are live pool revisions in sync in different
environments?
![Page 30: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/30.jpg)
Deployment and Release Automation
Solution: Utilize app check-ins and Neo4j to expose info about live and dark pools, enabling us to automate deployments in a release
![Page 31: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/31.jpg)
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
“Live” Pool “Dark” Pool
![Page 32: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/32.jpg)
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
![Page 33: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/33.jpg)
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
“Draining” Pool “Live” Pool
![Page 34: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/34.jpg)
Deployment and Release Automation
• Blue-green deployment
Server 1
Server 2
Server 3
Server 5
Server 4
Server 6
Server 7
Server 8
Service Group
DEPLOY
Pool Cut-over
![Page 35: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/35.jpg)
App Instances
MacGyver
Neo4j
<<update app instance>>MacGyver
![Page 36: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/36.jpg)
Virtual Servers
MacGyver
Neo4j
Load Balancer<<REST API Polling>>
![Page 37: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/37.jpg)
Virtual Servers
MacGyver
Neo4j
<<update virtual server>>
Load Balancer<<REST API Polling>>
![Page 38: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/38.jpg)
Application Pools
![Page 39: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/39.jpg)
Virtual Service
![Page 40: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/40.jpg)
Deployment and Release Automation
Problem: Highly manual and tedious releases Difficult to answer questions like:– What pool should I deploy to?– Is the most recent revision ‘live’ right now?– Are live pool revisions in sync in different
environments?
![Page 41: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/41.jpg)
Deployment and Release Automation
“What pool should I deploy to?”
![Page 42: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/42.jpg)
Deployment and Release Automation
“Are live pool revisions in sync in different environments?”
![Page 43: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/43.jpg)
Deployment and Release Automation
“Do multiple revisions exist within a single pool?”
![Page 44: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/44.jpg)
Deployment and Release Automation
“Do multiple revisions exist within a single pool?”
Think about the Knight Capital meltdown in 2012!
![Page 45: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/45.jpg)
App Instance
App Instance
App Instance
App Instance
Deployment and Release Automation
MacGyver
Neo4j
Load Balancer<<PUT>>
<<REST API>>
<<REST API>>
![Page 46: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/46.jpg)
How Does DevOps at Lending Club Use Neo4j?
1. App check-in/service discovery2. Deployment and release automation 3. Infrastructure mapping
![Page 47: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/47.jpg)
![Page 48: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/48.jpg)
“Are any servers in a live pool degraded?”
![Page 49: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/49.jpg)
![Page 50: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/50.jpg)
“Do we have a single point of failure among any of our services?”
![Page 51: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/51.jpg)
![Page 52: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/52.jpg)
“If storage volume #3 goes down, what services will be impacted?”
![Page 53: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/53.jpg)
Other Use Cases
• “Are servers in the correct security zones?”• “What is the correct AWS VPC placement for
this application?”• Deliver abstraction across multiple Load
Balancer implementations• etc.
![Page 54: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/54.jpg)
For More Info
• https://github.com/if6was9/macgyver• https://macgyver.readme.io• https://github.com/if6was9/neorx
• Ashley Sun – [email protected], @ashleycsun
• Rob Schoening - [email protected], @rschoening
![Page 55: Managing Microservices with Neo4j](https://reader036.fdocuments.us/reader036/viewer/2022062821/589d68651a28abd91d8b562d/html5/thumbnails/55.jpg)
Questions