© 2012 ADAPTIVE COMPUTING, INC. 1

Building an Integration Platform with Grails and Gradle

Brian Saville

Adaptive Computing

18 Oct 2012

© 2012 ADAPTIVE COMPUTING, INC. 2 © 2012 ADAPTIVE COMPUTING, INC. 2

Me

Brian Saville

bsaville@adaptivecomputing.com

Twitter: @bksaville

Plugins:

▪ External Config Reload

▪ Spring Security OAuth2 Provider

▪ New Doc

▪ Artefact Messaging

▪ Contributor on Others

© 2012 ADAPTIVE COMPUTING, INC. 3 © 2012 ADAPTIVE COMPUTING, INC. 3

Adaptive Computing Highlights

▪ Leaders in cloud and HPC software

▪ 10+ years developing Moab® technology

▪ Leadership team w/ $1.5B+ in exits

▪ 15 issued patents, 34 pending

▪ Backed by top-tier investors:

▪ Global Partnerships: HP, IBM, SGI, Cray, Microsoft, etc.

▪ Over 200 customers such as:

© 2012 ADAPTIVE COMPUTING, INC. 4 © 2012 ADAPTIVE COMPUTING, INC. 4

Moab Web Services

▪ What is it?

▪ Grails 2.0.3

▪ RESTful Web Services

▪ Business Logic

▪ Plugin Framework (Beta)

▪ What does it do?

▪ Wraps C Core Product

▪ Integrates “old world” non-standardized API with “new world” APIs

© 2012 ADAPTIVE COMPUTING, INC. 5 © 2012 ADAPTIVE COMPUTING, INC. 5

Atypical Grails Deployment

▪ MWS Deployed on Client Servers

▪ WAR file

▪ Tomcat 6

▪ Oracle/Sun JDK 6

▪ MongoDB 2.0+

▪ Customization on Client Sites

© 2012 ADAPTIVE COMPUTING, INC. 6 © 2012 ADAPTIVE COMPUTING, INC. 6

Integration Platform

▪ Reasons

▪ Extensibility

▪ In-House Developers

▪ External Developers

▪ Dynamic Updates to Application Behavior

▪ Why Create Our Own?

▪ Consists of Plugin Framework (Grails) and SDK (Gradle)

© 2012 ADAPTIVE COMPUTING, INC. 7 © 2012 ADAPTIVE COMPUTING, INC. 7

© 2012 ADAPTIVE COMPUTING, INC. 8 © 2012 ADAPTIVE COMPUTING, INC. 8

Why Do We Care?

▪ Integration Platform Developers

1) Provides a standard pattern to follow

2) Identifies required and optional framework components

3) Treads ground for new integration platforms

4) Identifies hurdles and potential problems

▪ Grails/Gradle Users

1) New advanced application of Grails and Gradle

2) Grails plugin framework can be evaluated with the pattern

3) Gradle plugin framework can be evaluated with the pattern

© 2012 ADAPTIVE COMPUTING, INC. 9

Integration Platform Pattern

© 2012 ADAPTIVE COMPUTING, INC. 10 © 2012 ADAPTIVE COMPUTING, INC. 10

The Pattern

▪ Plugin Identification

▪ Lifecycle and State Management

▪ Dependency Management

▪ Extend Application

▪ Services

▪ Configuration Management

▪ Lifecycle Hooks

▪ Metadata

▪ Development (SDK)

© 2012 ADAPTIVE COMPUTING, INC. 11 © 2012 ADAPTIVE COMPUTING, INC. 11

Plugin Identification

▪ By Convention

▪ Class names

▪ Packaging (JAR, Java or Groovy file)

▪ By Interface(s)

▪ IPlugin

▪ Activator (OSGI)

▪ By Configuration

▪ MANIFEST.MF

© 2012 ADAPTIVE COMPUTING, INC. 12 © 2012 ADAPTIVE COMPUTING, INC. 12

Lifecycle and State Management

▪ Concept of State

▪ Stopped and started

▪ More states optional

▪ Control Over State

▪ How to stop, how to start?

▪ How to add plugin? (Removal not required)

© 2012 ADAPTIVE COMPUTING, INC. 13 © 2012 ADAPTIVE COMPUTING, INC. 13

Dependency Management

▪ Internal Dependencies

▪ Build process

▪ Consider IDE

▪ External Dependencies

▪ Fat JARs

▪ Dependency resolution when adding

© 2012 ADAPTIVE COMPUTING, INC. 14 © 2012 ADAPTIVE COMPUTING, INC. 14

Extend Application

▪ Different in Each Case

▪ Custom Services

▪ Web services

▪ Internal services

▪ Provide Resources

▪ Report or Persist Data

▪ …

© 2012 ADAPTIVE COMPUTING, INC. 15 © 2012 ADAPTIVE COMPUTING, INC. 15

Services

▪ Stateless Services

▪ Control services

▪ Logging

▪ Dependent Services

▪ Configuration

▪ Data persistence

© 2012 ADAPTIVE COMPUTING, INC. 16 © 2012 ADAPTIVE COMPUTING, INC. 16

Configuration Management

▪ Access Application Configuration

▪ (Optional) Access Private Configuration

▪ Preferences

© 2012 ADAPTIVE COMPUTING, INC. 17 © 2012 ADAPTIVE COMPUTING, INC. 17

Lifecycle Hooks

▪ State Transition Hooks

▪ Starting and stopping

▪ Before, after, and/or during transition?

▪ Other Hooks

▪ Setup and configuration

▪ How Are They Defined?

▪ Convention

▪ Interfaces (Configuration)

© 2012 ADAPTIVE COMPUTING, INC. 18 © 2012 ADAPTIVE COMPUTING, INC. 18

Development (SDK)

▪ Dependency Management

▪ Build with internal dependencies

▪ Include external dependencies

▪ Testing

▪ Unit

▪ Integration

▪ Deployment

▪ (Optional) RAD Tool

© 2012 ADAPTIVE COMPUTING, INC. 19 © 2012 ADAPTIVE COMPUTING, INC. 19

Metadata

▪ By Convention

▪ By Configuration

▪ Types of Metadata

▪ Project/Plugin information

▪ Author information

▪ (Optional) Configuration constraints

▪ (Optional) Dependency information

© 2012 ADAPTIVE COMPUTING, INC. 20 © 2012 ADAPTIVE COMPUTING, INC. 20

Non-Essential Pattern Elements

▪ Security

▪ Deployment System/Central Repository

▪ Inversion of Control (IoC) Container

▪ Data Persistence

▪ Individual to Plugin

▪ Application Aware

▪ Common Clients and Parsers

▪ XML/JSON

© 2012 ADAPTIVE COMPUTING, INC. 21 © 2012 ADAPTIVE COMPUTING, INC. 21

Non-Essential Pattern Elements Cont.

▪ Events – Built-In and Custom

▪ Decomposition of Logic

▪ Messaging (i18n)

▪ Startup/Shutdown Concerns

▪ Scheduled Jobs

▪ Inter-Project Dependencies

▪ Environment Handling

© 2012 ADAPTIVE COMPUTING, INC. 22 © 2012 ADAPTIVE COMPUTING, INC. 22

Grails Plugins

▪ What is Missing?

▪ Standardized configuration management

▪ Platform core is addressing this

▪ Framework is Mature

© 2012 ADAPTIVE COMPUTING, INC. 23

MWS Integration Platform

© 2012 ADAPTIVE COMPUTING, INC. 24 © 2012 ADAPTIVE COMPUTING, INC. 24

Introduction

▪ Plugin Type – Plugin Definition Class

▪ One per Class

▪ Stored as Groovy or JAR files

▪ Internally is a Grails artefact

▪ Plugin Instance

▪ Multiple instances for a single plugin type possible

▪ Internally is DB entry and Spring Bean

▪ Dynamic (Running Application)

© 2012 ADAPTIVE COMPUTING, INC. 25 © 2012 ADAPTIVE COMPUTING, INC. 25

Plugin Identification

▪ “Ends With” Class Names

▪ Plugin Types

▪ Translators

▪ Autowired Classes

▪ Spring Annotated Classes

▪ @Component

▪ @Autowired

▪ @Resource

▪ @Scope

© 2012 ADAPTIVE COMPUTING, INC. 26 © 2012 ADAPTIVE COMPUTING, INC. 26

Lifecycle and State Management

▪ States

▪ Started, Stopped, Paused

▪ Plugin Control Service

▪ Start, stop, pause, resume

▪ Adding Plugins

▪ User Interface

▪ REST API

© 2012 ADAPTIVE COMPUTING, INC. 27 © 2012 ADAPTIVE COMPUTING, INC. 27

Dependency Management

▪ SDK for Build Dependencies

▪ Grails JARs

▪ Spring JARs

▪ Support Fat JARs

▪ Dependencies Resolved Before Adding

© 2012 ADAPTIVE COMPUTING, INC. 28 © 2012 ADAPTIVE COMPUTING, INC. 28

Extend Application

▪ Reporting – Application Aware Data Persistence

▪ Expose Custom Web Services

▪ Methods by convention

▪ Almost full REST API

▪ Unsecured annotation

▪ Routed by URL mappings

▪ Built-In Events by Convention

© 2012 ADAPTIVE COMPUTING, INC. 29 © 2012 ADAPTIVE COMPUTING, INC. 29

Services

▪ Logging (Similar to Grails)

▪ Data Persistence Services

▪ Control Service

▪ Individual Datastore Service

▪ SSL Service - SSLSocketFactory

▪ not-yet-commons-ssl with PEM files

▪ In memory keystore

▪ Internal REST Service

▪ Request and response with optional attributes

▪ Returns object with response, easy access properties, parsed data

© 2012 ADAPTIVE COMPUTING, INC. 30 © 2012 ADAPTIVE COMPUTING, INC. 30

Configuration Management

▪ Application Configuration

▪ Grails application bean

▪ Plugin Private Configuration

▪ Metaclass injected “config” property

© 2012 ADAPTIVE COMPUTING, INC. 31 © 2012 ADAPTIVE COMPUTING, INC. 31

Lifecycle Hooks

▪ Implemented as Methods on Plugin Types

▪ beforeStart, afterStart, beforeStop, afterStop

▪ configure

▪ If Not Implemented, Inject Empty Methods

▪ Called from Control Service

© 2012 ADAPTIVE COMPUTING, INC. 32 © 2012 ADAPTIVE COMPUTING, INC. 32

Metadata

▪ Convention

▪ Plugin project compiled class included in JAR

▪ Ends with “Project”

▪ Configuration

▪ Properties on plugin project class

▪ =~ Grails Plugin Class

© 2012 ADAPTIVE COMPUTING, INC. 33 © 2012 ADAPTIVE COMPUTING, INC. 33

Metadata – Dynamic Configuration Constraints

▪ Configuration is Dynamic Map

▪ Utilize Grails Constraints and Conventions

▪ Redo min/max to remove value checking

▪ Add custom plugin constrained property (CP)

▪ Add custom plugin CP builder

▪ Add Custom Constraints

▪ Type (Class) – Infer if not specified

▪ Default value

▪ Required (nullable)

▪ Scriptable URL (exec, file, http)

▪ Customized validator with additional methods

© 2012 ADAPTIVE COMPUTING, INC. 34 © 2012 ADAPTIVE COMPUTING, INC. 34

i18n Messaging

▪ Message Property Files

▪ Included in JAR

▪ Loaded with individual message source per project

▪ Message Method Injected on Plugins

▪ Used to Resolve Constraint Errors

▪ Not quite multi-lingual

© 2012 ADAPTIVE COMPUTING, INC. 35 © 2012 ADAPTIVE COMPUTING, INC. 35

Development (SDK)

▪ Commons JAR

▪ Interfaces, classes, constants, annotations

▪ Testing JAR – TestFor

▪ Gradle Project

▪ Dependencies

▪ Testing and Commons

▪ Creates fat JARs (lib and resolved deps)

▪ Additional build tasks

▪ Handles i18n property files

▪ Cobertura code coverage reports

▪ Unit testing with Spock

© 2012 ADAPTIVE COMPUTING, INC. 36 © 2012 ADAPTIVE COMPUTING, INC. 36

Development (SDK) - Commands

▪ Gradle

▪ create-project -Pproject-name=MyProject

▪ create-plugin -Pplugin-name=sample.MyPlugin

▪ test

▪ upload-test -Pmws.url=http://localhost/mws

▪ (Eventually) Python CLI Wrapper

▪ mplugin create-project MyProject

▪ mplugin create-plugin sample.MyPlugin

▪ mplugin test

▪ mplugin upload --url=http://localhost:8080/mws

© 2012 ADAPTIVE COMPUTING, INC. 37 © 2012 ADAPTIVE COMPUTING, INC. 37

Non-Essential Pattern Elements Status

▪ Security - Yes

▪ Central Repository – No

▪ IoC Container – Spring

▪ Data Persistence – Individual Datastore and Application Aware

▪ Common Clients and Parsers – Client Decided and Groovy

▪ Events – Built-In Only

▪ Decomposition – IoC with Conventions and Annotations

© 2012 ADAPTIVE COMPUTING, INC. 38 © 2012 ADAPTIVE COMPUTING, INC. 38

Non-Essential Pattern Elements Status Cont.

▪ Startup/Shutdown Concerns – Initial Plugins

▪ Scheduled Jobs – Quartz

▪ Inter-Project Dependencies – No

▪ Environment Handling – Application Configuration Parameter

© 2012 ADAPTIVE COMPUTING, INC. 39 © 2012 ADAPTIVE COMPUTING, INC. 39

Comparison to Grails Plugins

Concept MWS Plugins Grails Plugins

RAD Tool Partial X

Configuration X No Standard

Integration Tests X

Multiple Instances

X

Custom REST Partial X

Metadata Partial X

Languages Groovy Groovy & Java

Dynamic X

Lifecycle X X

i18n Partial X

© 2012 ADAPTIVE COMPUTING, INC. 40 © 2012 ADAPTIVE COMPUTING, INC. 40

Demo

© 2012 ADAPTIVE COMPUTING, INC. 41

Problems Encountered

© 2012 ADAPTIVE COMPUTING, INC. 42 © 2012 ADAPTIVE COMPUTING, INC. 42

JVM ClassLoaders

▪ Separating ClassLoaders

▪ Locking JARs

▪ Fat JARs

▪ Unresolved: Reloading Typed References

© 2012 ADAPTIVE COMPUTING, INC. 43 © 2012 ADAPTIVE COMPUTING, INC. 43

Language Support

▪ Groovy – Works Great

▪ Java – Interfaces and Metaclass Issues

© 2012 ADAPTIVE COMPUTING, INC. 44

Questions?