MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
-
Upload
mongodb -
Category
Technology
-
view
186 -
download
1
Transcript of MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
MongoDB in the middle of a Polyglot Persistence Architecture
Sigfrido NarváezPrincipal Software ArchitectSage North America
MongoDB World 2015June 2015 NY
2
What is this about?
db.Talk.find({
learnings: {
$in: [
"Polyglot Persistence & Architecture",
"How we use MongoDB",
"Tools & Ops"
]
}
});
System of RecordERP, Accounting, HR & Payroll
ERP’sOn-premise
SMB Sage Data Cloud
Connector
Customers
Sales
Product Catalog
Payroll
3
Domains
Data Taxonomy
Types
Reference
Transactional
Historical
Computed
Metadata
Settings
App-specific data
Files & Logs
Global / Lookup
Behavior
Origin: ERP / Cloud
Querying
Fixed
Unstructured
Growth
Throughput
Searchable
Retention
Binary
Data Taxonomy
Types
Reference
Transactional
Historical
Computed
Metadata
Settings
App-specific data
Files & Logs
Global / Lookup
Behavior
Origin: ERP / Cloud
Querying
Fixed Schema
Unstructured
Growth
Throughput
Searchable
Retention
Binary
Sage ERP’sOn-premise
Domains as Micro-Services
Connector
13
Customers
Sales
Products Catalog
Payroll
REST API
Resources
CRUD & Operations
Business
Entities
Rules
Data Layer
Mappings
Migrations
ERP Plug-in
Contracts
Mappings
Connector
Contracts
BackgroundData Sync
BackgroundData Sync
Mobile Sales
ERPMobile Sales
Cus
tom
ers
Pro
du
ct
Cat
alo
g
…
ERPC
ERPB
ERPA
Mobile Sales
Domain
NE
W
Dom
ain
Mobile Sales
Cus
tom
ers
Pro
du
ct
Cat
alo
g
…
ERPC
ERPB
ERPA
App
NE
W
Dom
ain
Mobile Sales
Cus
tom
ers
Pro
du
ct
Cat
alo
g
NEWApp
…
ERPC
ERPB
ERPA
Sal
es
Cus
tom
ers
Pro
du
ct
Cat
alo
g
…
Platform
Technology
Extending the SDC
04/14/2023 15
Apps & FaaS built ontop of Domains
Each Governs Entities, Rules & Persistence
Organized around Business Capabilities
New Apps can re-use, extend or add
New Technologies can be addedor replace existing
MongoDB Use Cases
• Historical Data• Write bursts• Mongo handles easily!• KPI’s via Aggregation Framework
18
REST API Business Data Layer
KPI
Historical Azure WorkerAggregation
Pipeline
Pre-Calculated
On-The-Fly
Aggregation Framework
{
{ "$match", { .. } },
{ "$sort", { .. } },
{ "$limit", 50}
{ "$unwind", { .. } },
{ "$match", { .. } },
{ "$limit", 50},
{ "$project", { .. } }
}
Dynamic schema
• Settings• Tenant, User, App, Features, etc.• Problematic in Relational model• Outgrows K/V model
• App-specific Data • Extensions to Core Entities• i.e. a SalesPerson is an extension of a User• Personalization
SQL MongoDB
{ "_id" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066SalesSalesTeamMember", "CreatedOn" : ISODate("2014-09-15T17:12:48.917Z"), "UpdatedOn" : ISODate("2014-09-15T17:12:48.917Z"), "CreatedBy" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066", "UpdatedBy" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066", "TenantId" : 3, "EntityStatus" : 0, "SettingKey" : "SalesTeamMember", "AppId" : 0, "ExtendedEntityId" : "4c44592a-1f6f-4a8f-bb87-a3a700a85066", "SalespersonIds" : [new BinData(3, "mcseUhOfQUmPK6OnAKhQfA==")], "Title" : "TestTitle", "IsSalesMgr" : false, "IsSalesRep" : false, "SalesRepMaxDiscPct" : 0.0}
{ "_id" : "2e866412-dbda-47ad-8767-a3a700a78e21ServiceServicesUser", "CreatedOn" : ISODate("2014-09-15T17:10:03.04Z"), "UpdatedOn" : ISODate("2014-09-15T17:10:03.04Z"), "CreatedBy" : "00000000-0000-0000-0000-000000000000", "UpdatedBy" : "00000000-0000-0000-0000-000000000000", "TenantId" : 3, "EntityStatus" : 0, "SettingKey" : "ServicesUser", "AppId" : 1, "ExtendedEntityId" : "2e866412-dbda-47ad-8767-a3a700a78e21", "Notes" : "Test Notes", "Role" : 1, "RoleAsString" : "Administrator", "LunchStartTime" : "12:00:00", "LunchEndTime" : "13:00:00", "StartTimeMon" : "08:00:00", "StartTimeTue" : "08:00:00", "StartTimeWed" : "08:00:00", "StartTimeThu" : "08:00:00", "StartTimeFri" : "08:00:00", "StartTimeSat" : "00:00:00", "StartTimeSun" : "00:00:00", "EndTimeMon" : "17:00:00", "EndTimeTue" : "17:00:00", "EndTimeWed" : "17:00:00", "EndTimeThu" : "17:00:00", "EndTimeFri" : "17:00:00", "EndTimeSat" : "00:00:00", "EndTimeSun" : "00:00:00"}
Extensions
@codinghorror
Lessons Learned
Indexes
• Adjust Indexes as needed
• Background• Not on unique indexes!• Can’t add new background indexes and delete old indexes
Migrations
• Added after launch
• DevOps “book keeping”, i.e. Indexes
• https://github.com/phoenixwebgroup/DotNetMongoMigrations
Lessons Learned
Exceed Namespace
• 16mb limit / ~24K namespaces
• Each collection & index is a namespace
• Aggregation output to temp collections
Aggregations
• Ensure $match & $sort uses indexes
• 100mb limit
• Increase with allowDiskUse: true
• Monitor for increase on Non-Mapped Virtual Memory
Production
DevOps - From code to alert paths!
25
Dev CI/CD Log Mining
WAD, IISCustom
APM
Azure Role Metrics
DB Telemetry
Deploy &DB Migrations
code
warm
cold
Mongo on Azure
Partnered with MongoLab
• Dev & Production clusters
• Slow queries & recommended indexes! (Dex)
• Profiler & Stream logs
• Telemetry: Real-Time Monitoring / link to MMS• http://docs.mongolab.com/monitoring
On your own
• Linux VM - http://docs.mongodb.org/ecosystem/tutorial/install-mongodb-on-linux-in-azure/
• Azure Worker - http://docs.mongodb.org/ecosystem/tutorial/deploy-mongodb-worker-roles-in-azure/
Mongo Monitoring & Performance (MMS)
• Data, Index, Storage
• Page Faults
• Memory & CPU
• Top 5 Alerts• Host Recovering, Repl Lag, Connections, Lock%, Replica• http://
www.mongodb.com/blog/post/five-mms-monitoring-alerts-keep-your-mongodb-deployment-track
• Top 5 Graphs• Lock %, Repl Lag, Journal Writes, Page faults, Non-Mapped Virtual Memory• http://
blog.mongodb.org/post/62152249344/the-top-5-metrics-to-watch-in-mongodb
Application Performance Monitoring
MongoLab plug-in
• Operations
• Queues
• Lock %
• Page Faults
• Non-mapped Virtual Memory
• Memory
• Replication Operations
• Replication Lag
• Oplog Window
SQL
• Query drill-down
• Index profile
Continuous Integration / Delivery
• Builds• Visual Studio aware• C# Code Analysis & Coverage• NUnit test execution• NuGet
• Azure Deployments• Azure plugin• Runbook Azure Automation
• Database Migrations• SQL• Mongo
Log Mining
• Open Access (Engineers, DevOps, Customer Support)
• Dump WAD, IIS & Error logs
• Alerts & Dashboards
• Verify issues have ceased (post deployment)
Alert Paths
• Hot• Less than 10s• “Wake someone up”• Heartbeat (SCOM/NOC)
• Warm • Less than 5m – DevOps engineer• NewRelic, MMS, Azure Portal• Slow Query, Slow Response, Repl Lag, Lock%, ApDex < 0.9, CPU%
• Cold• Longer than 5m - Production Support / Engineering• SumoLogic – Error showed up in logs X times• Forensic purposes – saved queries
33
Summary
• Use the best of what is out there!
• Divide for flexibility
• MongoDB is amazing: multi-faceted
• Enable DevOps with awesome tools!
• Change is constant – Partner up!
THANK YOU!
Sigfrido “Sig” Narváez
• [email protected] | @SigNarvaez
• Principal Software Architect, Sage
• Orange County MUG Organizer - ~3yrs
• 2015 MongoDB Master
Sage
• Solutions for ERP, CRM, HR, Payroll, Payments, Assets, Construction, and Real Estate.
• Headquartered in UK, offices in North America, mainland Europe, South Africa, Australia, Asia and Brazil.
• 13K Employees. 6M Customers. 23 Countries.
• LSE:SGE