NJIN 2.0: BUILDING THE MESSAGE SWITCH OF THE FUTURE · Background XML Message Router (XMR)...
Transcript of NJIN 2.0: BUILDING THE MESSAGE SWITCH OF THE FUTURE · Background XML Message Router (XMR)...
NJIN 2.0: BUILDING THE
MESSAGE SWITCH OF THE FUTURE
Kate Silhol
Background
■ XML Message Router (XMR) installed 12 years ago– Advanced Technology Systems product– First foray into XML, web services
■ XMR converted to Nlets Justice Information Network (NJIN) – Majority of code re-written, now considered in-
house product
Overview
■ Current NJIN is approaching capacity– Scalability limitations inherent to the 12 year old
design– Volume has increased by 209% in last 10 years– Manageable but recurring system issues (slowness)– Low risk mitigating changes made to sustain
current NJIN
■ Design work for new system commenced end of 2015
Design Overview and Principles
■ Distributed architecture
■ Three primary components which guide each transaction through the system
■ Specific processes located in distinct modules – true SOA
High Level Design
Input/Output Layer
Validation and ESB Initiation Layer (VEIL)
Enterprise Service Bus
Modules
SBG Customized PMB
Completely separate code; “black boxes”
mostly based on existing code
Initial validation, broker to the ESB
COTS Product; Backbone of the
NJIN 2.0
Input-Output Layer
■ Provides communication between the NJIN 2.0 and the users
■ Leverages SBG’s Plexus Message Broker Code
■ Communication between Input-Output Layer and the VEIL is via queues (one queue for each user/line)
Enterprise Service Bus
■ Backbone of the NJIN 2.0
■ Provides communication between the system components
■ Dynamic and static metadata will dictate the actions taken for each transaction
ESB Selection ProcessProduct Open Source/COTS Pros ConsWSO2
Open Source CostGreat reputation
Java‐based open source –significant ramp‐up for Nlets staff
NServiceBus
Open Source .NET based platform
Performance may not be adequateFunctionality specifics limit outside contractor helpUnclear roadmap
Neuron ESBCOTS .NET based, local Prior experience with tool
Poor reputationMuleSoft COTS
(Open Source also available but inadequate)
Good reputationFunctionality appears to be adequate
Company is focusing on cloud‐oriented architecture.Requires on‐going support costs
TIBCO
COTS
Reputation/functionality/performance – best in classSupport availabilityAbility to involve contractors
Cost
Home Grown N/A Ability to control functionality Time to implement
Processing Queues
■ Between the ESB implementations and connected layers and modules, communication will be via queueing
■ Message durability, prioritization, guaranteed delivery, etc.
Pipelines
■ The VEIL will apply a Pipeline to the transaction before sending to the ESB
– Set of instructions based upon configurable rules■ Different Pipelines based on message type,
destination, etc.■ Pipeline may be updated along the way – error
conditions, etc.■ Provide for better configurability and enhanced
diagnostics
Sample Catalog of Pipeline Components
Rules ValidationRules Validation
Parsing
Logging
Conversion
Sub Route
Message Header Change
Alerting (PAL/”Graham Scrape”)
Federation
Image Resizing
Image Stripping
Special Instructions
Send Message
Caveating
Sample Pipeline Configuration
Rules Validation
Parsing
Conversion
Image Resizing
Send Message
Transactional Objects
■ Each message traversing the NJIN 2.0 will be wrapped in a Transactional Object
■ Metadata about the transaction and its processing that will follow the message through the process
■ Pipeline will be maintained within the Transactional Object
<NletsTransactionalWrapper><NletsTransactionalObject>
<OriginalUniqueID>111111111111</OriginalUniqueID><UniqueID>111111111120</UniqueID><IPAddress>10.10.10.10</IPAddress><InboundHeaderData>
<MessageKey>DR</MessageKey><OriginatingORI>AZMVD0000</OriginatingORI><DestinationORI>AZNLETS23</DestinationORI><ControlField>CONTROLFLD</ControlField>
</InboundHeaderData><OutboundHeaderData/><MessagePriority>3</MessagePriority><InputHeaderFormat>Text</InputHeaderFormat><InputBodyFormat>Text</InputBodyFormat><OriginalMessage>DR.AZMVD0000.AZNLETS23.CONTROLFLD.TXTDRIVER STATUS
DATA</OriginalMessage><ErrorCondition/><PrimaryIndicator>true</PrimaryIndicator>
</NletsTransactionalObject><TransactionalPipeline>
<PipelineStep><StartDateTime/><EndDateTime/><Service>GetDestinationInfo</Service>
</PipelineStep><PipelineStep>
<StartDateTime/><EndDateTime/><Service>Validation</Service>
</PipelineStep></TransactionalPipeline><OutboundMessage/>
</NletsTransactionalWrapper>
Enhancements to NJIN
■ Better performance■ Scalability■ More detailed and usable logging■ Improved diagnostics■ Easier configurability■ Correlation of inbound message with outbound■ Spill queues
– Side by side queues allowing new traffic to flow alongside queued traffic following a state outage
■ Schema validation as an option (test system)■ Statistics■ Dashboard/console improvements
Monitoring Improvements
Total Traffic Volume (Past Hour)
Monitoring Possibilities
Monitoring Possibilities
0
200
400
600
800
1000
1200
1400
1600
0:0
01
:00
2:0
03
:00
4:0
05
:00
6:0
07
:00
8:0
09
:00
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
Overall Message Volume
SUM
0100200300400500600700800900
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Volume by Message Key
Message Key AM CR
DQ DR FQ
FR RQ RR
0
500
1000
1500
2000
2500
0:0
0
1:0
0
2:0
0
3:0
0
4:0
0
5:0
0
6:0
0
7:0
0
8:0
0
9:0
0
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
0
200
400
600
800
1000
1200
1400
1600
0:0
01
:00
2:0
03
:00
4:0
05
:00
6:0
07
:00
8:0
09
:00
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
Overall Message Volume
Future Possibilities
■ Analytics
■ Proactive scaling and load balancing
■ Proactive alerting functionality
■ Reuse of modules
■ Executive Dashboard
NJIN 2.0 Implementation Plan
Addition of QA/Dev System
■ Current test system doing double duty – we need a true test system!
■ First implementation of NJIN 2.0 will be in new QA/Development system
■ No external users – sandbox
■ Then followed by implementations in test and production
Production NJIN
Test NJIN
Production NJIN 2.0
Test NJIN 2.0
Dev/QA NJIN 2.0
Phased Approach
■ Same phased approach will be used on the Test System and Production System
■ Incremental process to minimize risk
■ All steps may be iterated for each individual state/user in a category
■ All phases have clear roll-back plans
Nlets Users
NJIN 1.0PMB
Nlets Users
Current NJIN
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Current NJIN
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Native MQ (PMB) users sending via new Input Layer
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Native MQ (PMB) users sending and receiving via new Input and Output
Layers
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
All users sending and receiving via new Input and Output
Layers
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
NJIN 2.0 processes in parallel (no messages
outbound) to allow testing and review of
logs
Logging
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Parallel processing turned off to prepare
for cutover
Logging
Nlets Users
NJIN 1.0
Input Layer
Output Layer
NJIN 2.0PMB
Nlets Users
Logging
NJIN 2.0 now performs all processing
Status
■ Procured TIBCO Business Works ESB
■ Procuring hardware
■ Team is scheduling and attending TIBCO training
■ Development work has begun– SQL, Input-Output layers and modules first
■ Timeline: Plan to implement in approximately one year