Natural Rule Language - · PDF fileNatural Rule Language •An open language for...
Transcript of Natural Rule Language - · PDF fileNatural Rule Language •An open language for...
![Page 1: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/1.jpg)
Natural Rule Language
Christian Nentwich, Model Two Zero Immo Hüneke, Zühlke Engineering
Presented at the British Computer Society
8th of June 2011
![Page 2: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/2.jpg)
Agenda
• Issues in System Integration
• The goal of executable specifications
• Quick intro to the Natural Rule Language
• Project Review: Soft Trades Synchronizer (Immo)
• Additional experience report from financial services
![Page 3: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/3.jpg)
Area of Concern
• We are dealing with large-scale system integration
ESB
![Page 4: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/4.jpg)
Problem Context
• Hundreds or thousands of systems
• Lots of variety
– Off-the-shelf
– In-house
– “Tactical” (hacks)
• Great complexity
– Tens of thousands of data items
– Complex processes
![Page 5: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/5.jpg)
Widespread Practice
Business Requirement
“We could save money by pre-populating SMALL SYSTEM from BIG SYSTEM
every day. Do it, we are allocating £200,000 to it.”
Analysis
• Analyse the data and semantics in SMALL SYSTEM and BIG SYSTEM
• Write a big specification
• (Usually extremely data centric, with some process on top)
Implementation
Testing and Go-Live
• Sometimes performed by separate teams
• Errors found here are guaranteed to delay the schedule
• Interpret and implement the specification
• Done by programmers using proprietary or open source tools
![Page 6: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/6.jpg)
Readability By Non-Technical Users
• Critical for gap analysis on future projects
• Important for bridging communication gap between developers / business
(Not readable)
![Page 7: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/7.jpg)
Precise and Formal Semantics
• Plain English is too ambiguous
• Must control that you are talking about right things in system context
> ird-12 (Mandatory)
> English Description:
> Context: CalculationPeriodDates (complex type)
>The frequency specified in calculationPeriodFrequency must divide the [period] precisely.
> This means that by stepping through the period from the start date at the specified frequency,
> it must be possible to reach the end date.
What does happen with stubs? Shouldn't the rule include the cases with stubs?
![Page 8: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/8.jpg)
Microsoft Word/Excel
Are you sure
this exists?
What does
“gives any
value” mean?
“Is any of this
currently in
production??”
![Page 9: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/9.jpg)
Is this as good as it gets?
• Poor processes
• Huge maintenance problems
– No transparency of data flows
– Specification useless after go-live
– Too much code to maintain
• Inefficient use of differently skilled people
• Communication barriers
– Off-shoring problems
• System integration is a specialised discipline – but discipline is what is lacking
![Page 10: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/10.jpg)
Changing Direction
Instead of writing informal specifi-
cations and then code, create
executable specifications.
![Page 11: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/11.jpg)
Executable Specifications
• Critical features:
– Readable by non-technical users
– Independent of a proprietary tool
– Precise and formal semantics
– Easy to write code generators or interpreters
• Desirable features:
– Writeable by non-technical users
– Amenable to easy analysis using tools
![Page 12: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/12.jpg)
Microsoft Word/Excel
• Readable by non-technical users
• Independent of a proprietary tool
• Precise and formal semantics
• Easy to write code generators or interpreters
• Writeable by non-technical users
• Amenable to easy analysis using tools
![Page 13: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/13.jpg)
Java / XSLT / Schematron
• Readable by non-technical users
• Independent of a proprietary tool
• Precise and formal semantics
• Easy to write code generators or interpreters
• Writeable by non-technical users
• Amenable to easy analysis using tools
![Page 14: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/14.jpg)
Natural Rule Language
• An open language for expressing:
– Validation rules (constraints)
– Action rules (e.g. Enrichment rules)
– Transformation / mapping
• Aimed at the core problem areas in integration
• Goals
– Read like an English sentence wherever possible
– Require no customisation to get going
– Offer symbolic and textual alternatives
• Specification: http://nrl.sourceforge.net
![Page 15: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/15.jpg)
The Origins: Validation
public void validate(Transaction transaction) {
if (transaction.getDebit() != null && transaction.getCredit() != null) {
if (transaction.getDebit().getValue() == transaction.getCredit()
.getValue() && getChangeInAmount() != null) {
exceptions.add(new ValidationError(
"Must not specify changeInAmount if a debit is equal to a credit"));
}
}
}
context: Transaction
inv: (self->debit = self->credit) implies (self->changeInAmount->empty())
<rule context="ns:transaction">
<assert test="ns:debit != ns:credit or not(ns:changeInAmount)">
Must not specify changeInAmount if a debit is equal to a credit
</assert>
</rule>
Java
OCL
Schematron NRL
Context: Transaction
Validation Rule "Our Sample Rule"
If a debit is present and a credit is present then
no changeInAmount is present
Report 'Must not specify changeInAmount if a debit is equal to a credit'
![Page 16: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/16.jpg)
NRL
A quick practical introduction!
![Page 17: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/17.jpg)
How to use this?
• NRL started out as a “Swiss army knife”
– “Use it for anything”
• Lesson: don’t build Swiss army knives!
• Here is where we got to eventually...
![Page 18: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/18.jpg)
Proprietary
Format Validation
(NRL Constraints)
Mapping (NRL Actions)
Enrichment (NRL Actions)
Canonical
Format
POJO Classes
Spring Mule ESB
WebSphere
Logical Adapter
Physical Adapter
Weblogic
Camel
JBoss
![Page 19: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/19.jpg)
NRL Parser
• The NRL parser is free and open source, also on sourceforge
• Designed for processing / code generation
ConstraintRuleDeclaration
IfThenStatement
ExistsStatement ExistsStatement
1. Text in NRL concrete syntax 2. Parser-generated AST
3. Code generators (from AST)
![Page 20: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/20.jpg)
Model Specifications
Executable Specification using NRL
UML
XML
Schema
Rule Specifications
CSV
Fixed
Width
XML
SWIFT
MT
NRL
Constraints
NRL
Actions
*SWIFT SDK
Code
Guaranteed Consistency
Plain Java
Datastage
Drools
![Page 21: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/21.jpg)
Project Review
Over to Immo!
![Page 22: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/22.jpg)
What else have we done?
• Almost entire take-up is for mapping – Central reference data publishing service of a bank
– Corporate action processing
– SWIFT messaging
– 60+ smaller projects, world-wide
![Page 23: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/23.jpg)
Lesson: Dates are Important
• 83 out of 207 (40%) of FpML validation rules are about dates
• 220 out of 751 rules (29%) in reference data export adapter about dates
• Very important to treat dates as a primitive type
If DebtInstrumentCharacteristics.LegalMaturityDateTime is present
then set to DebtInstrumentCharacteristics.LegalMaturityDateTime;
![Page 24: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/24.jpg)
Lesson: Rationale is External
Context: InterestRateStream
Validation Rule "ird-2"
paymentDates.paymentFrequency [is an integer multiple
of] calculationPeriodDates.calculationPeriodFrequency
Why???
Pragmatics != Semantics
![Page 25: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/25.jpg)
Lesson: Testing
• How to test this?
• People still want to do TDD, even with executable specs
• Tests form part of the specification!
• We’re looking into it
Context: InterestRateStream
Validation Rule "ird-2"
paymentDates.paymentFrequency [is an integer multiple
of] calculationPeriodDates.calculationPeriodFrequency
![Page 26: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/26.jpg)
Lesson: Analysis
• There are big benefits to be had from metadata management
– See session this afternoon!
• Turing-complete languages are too hard to analyse
• Think hard about how expressive your language needs to be
![Page 27: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/27.jpg)
Traceability Both Ways
Natural
Rule
Language
UML XML
Schema
Generated
Code
Metadata
Tools
Published
Specifications
![Page 28: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/28.jpg)
Lesson: Analysis
public boolean validate(CalculationPeriodDates context) throws Exception {
CalculationPeriodDates current = context;
return current.getTerminationDate().getUnadjustedDate().getValue()
.toGregorianCalendar().compareTo(current.getEffectiveDate().
getUnadjustedDate().getValue().toGregorianCalendar()) > 0;
}
Context: CalculationPeriodDates
Validation Rule "ird-14"
terminationDate.unadjustedDate is after effectiveDate.unadjustedDate
![Page 29: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/29.jpg)
Lesson: Don’t Abuse Rule Engines
• We wrote a code generator for the JESS engine
• Hand-wrote NRL
• Artificially split rules into left-hand and right-hand side for RETE matching
• Didn’t work too well (execution model too hard, and doesn’t fit)
![Page 30: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/30.jpg)
Lesson: Programmers...
... like to turn things into code!
If ([getStaticData] 'getSystemDate') = 'Y' then
Set header.DateOfCreationAmendment to
([padField] ([currentDate]'yyyyMMdd') and 8 and 'B' and '0')
Else
Set header.DateOfCreationAmendment to
([padField] headerrecord.DateOfCreationAmendment and 8 and 'B' and '0')
Camel-case instead of English sentence
Mixing syntax and semantics
![Page 31: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/31.jpg)
Lesson: Discipline
• What makes one model better than another?
• How do a good data architects distinguish themselves?
• What is the best way to gather tacit rules?
• What are best practices for large-scale modelling?
• What is the standard ontology of artefacts and activities in integration?
![Page 32: Natural Rule Language - · PDF fileNatural Rule Language •An open language for expressing: ... Spring Mule ESB WebSphere Logical Adapter Physical Adapter Weblogic Camel JBoss . NRL](https://reader031.fdocuments.us/reader031/viewer/2022021819/5abe78bf7f8b9ac0598d291a/html5/thumbnails/32.jpg)
Lesson: Discipline
• Some inroads
• System integration still broadly lacks discipline
• Many topics are off the radar of computer science curriculum
• Good luck finding the best people!