Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB

HOW TO USE FME SERVER AS AN

Enterprise Service Bus

WHO WE ARE

Dean HintzSenior Analyst,

Strategic ExpertsSafe Software

Neil HellasEnterprise Solutions Architect

Spatial DNA

Tens of thousands of data pros use FME, the data integration platform with the best support for location data.

Don MurrayPresident,

Safe Software

AGENDA● FME

● Enterprise integration challenges & ESB

● FME Server as ESB

● Spatial DNA examples

● 311 ESB Demo

● Q&A

Technology is always changing

Innovative solutions are possible

New systems are constantly being added

System integration is hard

CHALLENGE

Enables data translation

Integrate virtually any system

Powerful data transformation

No coding required

FME

Connect to Virtually Any Format or Service with FME

400+ SUPPORTED SYSTEMS AND DATA TYPES

Database servers: Oracle, SQL Server, PostGreSQL, MySQL,...

Web systems: MongoDB, CouchDB, Dropbox, Box.com, Amazon S3, OData, Google Sheets, Socrata, Salesforce, Sharepoint, Azure, OGC ...

Business Intelligence: Tableau, Excel, ElasticSearch, Qlik

Extensibility: integration with traditional and emerging systems

● JDBC - often traditional● HTTP - often cloud

Tabular, XML, JSON, CSV, HTML ...

FME TERMINOLOGY

Transformers: Used to connect to systems, extract data, transform and remodel data and then load data. Over 400 available!

Transformers use and produce Features.

Features: Records or rows in spreadsheets and databases or objects in other systems.

Features have Attributes.

Attributes: Fields or columns attached to records, rows and objects.

https://www.safe.com/transformers

Three Ways to use FME

FME Server

Automate Deliver Download

** Events / messages: Our main focus here

FME Server Notification Service

●●●

AUTOMATION, EVENTS AND REAL-TIME

Scheduling

REST API

Messaging:

○ JMS○ SQS ○ WebSockets○ WebHooks○ Web Services

API Integration Challenge

Enterprise Integration Challenge

● Multiple, separate applications for different business systems: HR, CRM, BI, ERP, Accounting

● Data silos, duplicate records

● Integration is manual, ad-hoc, point to point

● The more systems that are added, the more difficult it is to manage

Enterprise Integration Challenge: 311 Reporting

Background: Citizens submit reports on the website about problems they notice in the community: missed garbage, potholes, etc. Those reports need to go to the service delivery team who manages related services.

Challenge: The more systems that need to be connected to the 311 database, the more complex the interconnections between databases become.

Specifics: Use topics within the notification service on FME Server as an ESB. All requests between client systems and the 311 database pass through the ESB topics on Server using a common messaging structure. Each system only needs to manage connections to this bus, not other systems.

ESB Demo


311 DB

EngineeringDB

SafetyDB

Initial architecture

311 DB

EngineeringDB

SafetyDB

TransportDB


Architecture grows

311 DB

EngineeringDB

SafetyDB

TransportDB

Contact DB


And grows ...

311 DB

EngineeringDB

SafetyDB

Contact DB

Enterprise Integration Challenge: ESB Approach

More growth = easy

311

ESB

311 DB

EngineeringDB

SafetyDB

TransportDB

Contact DB

Enterprise Integration Challenge: ESB Approach

More growth = easy

311

ESB

Capabilities of an ESB

An ESB can… has…

An ESB can… has…• Route data between systems • Testers, Filters, Notifiers


An ESB can… has…• Route data between systems• Transform formats and protocols

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller


An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous


Extractors and HTTPCaller• Service Types, JobSubmitter



Aynchronous• Publish/Subscribe


Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications



Aynchronous• Publish/Subscribe• Events and messages


Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols


• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous• Publish/Subscribe• Events and messages• Extensible & pluggable


Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols• Chaining topics



• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous• Publish/Subscribe• Events and messages• Extensible & pluggable• Publish Metadata


Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols• Chaining topics• FME Server API



Spatial DNA

● ESB Implementation Examples

311 IncidentManagement

ESB Demo

ESB Demo

EngineeringUpdate

SafetyUpdate

311 CaseUpdate

ESB 311Topics

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

UpdateDepartmentDB.fmw

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

ProcessSafetyMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessUpdateMessage.fmw

ESB 311Topics


Loader.fmw

311 DB


EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw


Update311DB




ESB 311Topics


Loader.fmw

311 DB


EngineeringDB

ESB DemoESB 311Topics

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw


Update311DB





Loader.fmw

311 DB


EngineeringDB

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

Update311DB





ESB 311Topics

ESB Demo


Loader.fmw

311 DB


EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

Update311DB





ESB 311Topics


Loader.fmw

311 DB


EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw





ESB 311Topics

Update311DB


Loader.fmw

311 DB


EngineeringDB

ESB Demo


EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw


Update311DB



ESB 311Topics

ESB 311 Demo: Loader, Update

ESB 311 Demo: Notifications

ESB 311 Demo: Database Summary Report

ESB Configuration on FME Server : Projects

Project Content:● Repository● Workspaces● Topics● Subscriptions● Schedules● Shared resource● DB connections● Web connections● Web socket

ESB Configuration on FME Server : Topics

ESB Configuration on FME Server : Schedules

ESB Configuration on FME Server : DB Connections

311PublicMessageLoader.fmw

To ESB

MessageTransmitter.fmw - Scheduled

To ESB

Update Message via Engineering Topic on 311ESB

{"Status":"Open","Category":"Traffic", "Description":"Flat tire", "Email":"[email protected]", "FirstName":"Dean", "ws_topic": "ESB_EngineeringUpdate", "json_featuretype":"ServiceMessage", "Time": "2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC", "Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}

mailto:[email protected]

ProcessSafetyMessage.fmw Subscription

From ESB

Update311MessageDB_Safety.fmw Scheduled

To ESB

ProcessUpdateMessage.fmw Subscription

From ESB

Update Message via 311Update Topic

{"Status":"Closed","Category":"Traffic", "Description":"flat tire\nResolved by: Jim", "Email":"[email protected]","FirstName":"Dean", "ws_topic":"ESB_311DB_Update", "json_featuretype":"ServiceMessage", "Time":"2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC","Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}


FME Server as ESB: Key Take-aways

● ESB - connection of many systems without increasing architecture complexity

● FME Server’s notification service provides ESB functions and components

● 311 ESB configuration tested using scheduled publication workspaces and

subscription workspaces connected to a set of topics

● Initial ESB design needs to be carefully laid out so the events and message

interactions are clearly mapped

● Define message structure and common data model: key elements, routing

● A good design is easily extendible: start simple, then expand

EnterpriseIntegrationWorkflows

Safe Software

Common Challenges

➔ Field name mapping

➔ Field value mapping

➔ Lookup field mapping

➔ Validating values

➔ Table mapping

➔ Record type mapping

JSON Transformers

Field Names and Values: AttributeManager

Copy from one field to another.

Provide a hard-coded default for a field.

Concatenate multiple field values together into a single field.

Copy part of a value from one field to another.

Value Mapping for Lookup Fields

Validate Values

Summary

● Dilemma of enterprise growth and complexity

● ESB approach / pattern helps manage growth

● FME Server’s notification service as an ESB

● Easy to add new systems and scale capacity without increasing complexity

● Notification service - a great way to integrate FME into your enterprise workflows whether or not you use ESB

Related Webinars● How to Use FME as an Enterprise Integration Platform (Aug 2017)

https://www.safe.com/solutions/enterprise-data-integration

● Web Services & Web Data – Best Practices for Integration (March 2017)

https://www.safe.com/webinars/web-services-fme-2017/

Upcoming:

● Deep Dive into FME Server 2018 (Feb 22, 2018)● FME Server Automations (March 28, 2018)

https://www.safe.com/solutions/enterprise-data-integration

https://www.safe.com/webinars/web-services-fme-2017/

Resources● Neil’s FME UC 2017 Talk: “Making FME Your ESB”

● FME Server Playground https://playground.fmeserver.com/

● FME Server Notification Docs

● Tutorial: Notifications

● Tutorial: Automation Planning and Deployment

● Tutorial: Reading and Writing JSON

● FME Hub: Custom transformers, templates, coming: Server Projects

ESB Demo project package will be sent to all attendees

https://www.safe.com/presentation/making-fme-your-esb

https://playground.fmeserver.com/

https://docs.safe.com/fme/html/FME_Server_Documentation/Content/ReferenceManual/Notification_Service.htm

https://knowledge.safe.com/articles/764/subscribe-report-and-notify-tutorial-0-basics-of-n.html

https://knowledge.safe.com/articles/44299/tutorial-automation-planning-and-deployment-with-f.html

https://knowledge.safe.com/articles/39188/tutorial-getting-started-with-json.html

https://hub.safe.com/

Thank you

Dean HintzSenior Analyst,

Strategic ExpertsSafe Software

[email protected]

Neil HellasEnterprise Solutions Architect

Spatial [email protected]

Don MurrayPresident,

Safe Software



Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB

Software

Transcript of Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB