Johan EdstromSOA Executive and Apache developer.Apache Camel PMCApache ServiceMix committerOriginal CXF Blueprint AuthorCassandra client library developerAuthorjedstrom@savoirtech.comjoed@apache.org

And debunking some SOA stuff

Using Apache Camel a bit like AKKA

Savoir Technologies - This is where we started

Where are we now?

Well, what happened there?

We kinda started focusing exclusively on scalability

• We went from the majority of our customers wanting “ESB”

• Or tune our “Messaging”

To

• Very modular software

• Infrastructure design that would last a while

• Key concept structures that were re-usable

• We’ve gotten quite a bit of experience here over the years

So what is that scalability we talk of?The right resources in the right place

• Tomcat doesn’t work better with 1024 threads

Messaging Writing QueueBrowsers to see where messages go isn’t awesome

State driven - Not mentioned in EIP Patterns (Yet)

Conversational - Not mentioned in EIP Patterns (Yet)

Effectively aggregating - Don’t use a RDBMS as a mutex

• select for update 2000 times a second doesn’t scale

Transactionally safe - And that doesn’t have to mean XA

Correctly utilizing EE

• Connectionpools aren’t for cowards

This is what we are trying to avoid....

What is the AKKA statement?

So where does AKKA come in?

It could be part of your deployment in Apache Karaf

It could be stand-alone

What is it that we really think is cool?

• Asynchronous systems!

• Actors

• Supervisors

• Scaling

• Immutability

Is that the goal for every Java shop out there?

So what does that AKKA look like?

Or maybe a bit more complicated?

So what is going on here?

• ActorSystem - umbrella naming

• System bound error handling

• Logging

• Scaling

• Then we introduce some EIP Patterns like “stuff”

We RoundRobin and allow it to scale

And we finally mount it

Are you trying to write Camel stuff?

Sure

• And yes I know that Camel-AKKA exists or is it AKKA-Camel now?

• I know there is a Scala DSL for Camel - fixed a few things there…

• What about the Scalaz stuff?

So…. That was really cool. But….

All projects aren’t greenfield

• Can we refactor?

We have existing staff and investments

• Is it easy to train

We want to do this in pieces

• We really want to try, test, promote

Could we do this with the existing EE Infra we have?

Let’s pretend you have Apache Karaf

This is not far from an EE container

• Add OpenEJB if you want and you’ll have a full stack

It contains a web-services layer

Dependency injection services

OSGi integration and legacy (JBI) support (Don’t use that. - Seriously…)

Camel for routing, transformation and mediation

Aries and Spring for EE support, JNDI, TX, JPA

Web-app deployment support

And we’ll add Apache ActiveMQ

This is another presentation :)

• Jeff did that one earlier today, I hope you did attend!

And I presume ya’ll know Apache Camel

Camel is a Domain Specific Language (DSL) focused on

implementing Enterprise Integration Patterns (EIPs)

• Examples: multicast, splitter, content-based router, routing slip, “dynamic

routers”, aggregator

EIPs are implemented by Camel “Processors”

• Users can develop their own processors

• A processor is a fundamental building block

• Bean language and bindings exists so that not a single piece of Apache Camel

Imports will be necessary when integrating your existing code

Camel can connect to a wide variety of integration technologies

• Examples: JMS, HTTP, FTP, SOAP, File - There are ~ 180 components

• Integrations are implemented by Camel “Components”

• Users can develop their own components

Apache Camel

Do I need all of that?

Nope, many solutions will need just a few things

• jaxb, camel, jms, soap, rest and perhaps jdbc

• Cut the container down to fit your needs

• We don’t need to load all of the 100+ Apache Camel components

• Pick and choose!

Should I run that messaging solution inside the “ESB”

• Entirely up to you, let us look a little deeper at that in a sec.

Can I test these solutions or am I stuck with

System.out.println and a remote debugger?

I’ve heard that OSGi sucks?

Yes, It is a bit more complex to manage a full lifecycle

• On the other hand you get reliable hot-deployment and things like

configuration, service management and service subscriptions out of the box.

Yes, package level import export can be confusing

• So make sure you use packages in a nice modular way.

Yes, it takes a while to learn the tooling

• Then did you wake up one day just knowing that WAR or EAR layout?

Yes, you need to learn the classpath and a bit about

classloading

• Which’ll make you a better developer in the end of the day.

• Many problems in this area actually are created by applying EE classloaders

to an OSGi environment.

Decoupling with OSGi

Enforces knowledge of imports and exports

Allows us to version

Programming model with micro-services allows

for re-use on an Api level vs code level

Promotes contracts

Use case #1

We want to accept incoming SOAP requests

We need to inspect those HTTP headers and then use them

to route

We want this to be dynamic and flexible We might want to add a more complex routing engine later

We want this to be modular and possible to run across

multiple systems

Verify your use-cases!

UseCase #1 outline

Let us use some EIP pattern symbols to outline this

• We haven’t really made any technology statements here

We are going to use Apache ActiveMQ because I said so.

It could easily have been any other JMS provider.

• We know we’ll have a synchronous part

Web-services by nature are.

• We’ll try and make all the other parts asynchronous

• We’ll also make sure that the participants don’t need strong integration or

visibility of each other, thus we’ll (hopefully) end up with some pretty re-

usable modules

• Another good approach is to also use sequence diagrams

What else do we want here?

A “Dynamic” Router!

• We’ll craft a simple header based Dynamic router from eip patterns.

Technically it is in between a routing slip with termination and a real

control channel pattern, we utilize JMS as our control channel for this.

We use this example to play with destinations from SoapUI

Remember that ActorSystem in AKKA?

• This dynamic router is a big part of “pretending” that we are doing that

The “Dynamic Router”

Yep, it is just a Java class.

• Imagine how easily you could integrate this with.... Anything!

We are able to turn “endpoints”, “routing”, “systems” or

whatever you want to call it into simple destination names.

We only need one route…..

We can break all these modules apart

Why build that type of a route?

So let’s sum up what we have at this point

We defined a CXF (JAX-WS) Endpoint, generated stubs and

tied it into Camel

• The MEP (Message Exchange Pattern) is IN/OUT since we use CXF

We then dropped in a router “Our Dynamic Router”

• It’ll listen to a JMS queue called recognizer

• Then it’ll start inspecting headers for a location

What about that testing?We are going to be dealing with an OSGi environment.

• Like an EE container it makes testing a tad less fun....

Full blown container testing with something like SoapUI

• Can be scripted and automated in Maven

• Or you could BDD / Cucumber script it

Full OSGi tests with Pax-Exam

• Allows you to wire up a full container

Simple OSGi registry tests with PojoSr

• Let’s you simulate an environment so you can test BluePrint!

Functional Junit/TestNG tests with Camel-Test.

• Should be combined with at least one of the above!

Testing the router

First of, let us just do a Camel Test, we want to verify that

the Dynamic routing works as expected.

• As you can see, none of these routes know about each other.

And if you can’t see that, they still don't.

Keep this in mind!!

How we’ll be using this dynamic data

We basically can use this to write completely dynamic steps

that will restart on a configuration change from

• File changes

• Code changes

• JMX changes

Proof of concept outline

We now have the following modules

• customer-service : A model library

• customer-service-ws : The webservices endpoint

• dynamic-routing : A generic module for routing

• response-builder : A simple module that returns a payload

• That means that all of the needed patterns are fulfilled.

• Competing consumer is a natural thing in JMS - KACHIIIIING!!!!

• Eventing is a natural part of Apache Camel - KACHIIIIING!!!!

• Request Reply is “Handled for us” - KACHIIIIING!!!!

• Dynamic config - KACHIIIIING!!!!

So why didn’t we just slap a camel route in Tomcat?

That’s why.

Otay - so what the heck did that have to do with AKKA

I’m glad you asked!

• We have, thanks to putting it on ActiveMQ

Immutability

Scalability

Asynchronous behavior

• We have a separation of Concerns - Kinda like the Actor

Are we really there tough?

• Not yet!

Do we think we can make that “Actor”

Lets make it very generic

Lets make it really dynamic

Lets use the power of OSGi to “do stuff”

The basic premise now is -

• JMS is our “Actor System”

• Camel handles Messaging and in/out for us as well as a lot of nice to have

stuff like errorhandling, graceful shutdown, startup, validation and so on.

• We want a “generic” way of handling say… Objects?

Second use-case!!!We are gonna take what we learned and build parts of a

dynamic ETL system - Basically CSV files

What is a mutator in this ETL system?

And how is it working?

Yey! We are manipulating strings!

• OMG, Such data….

• Kidding aside - we now have a tool where all the transformations can be

described, this system contains OGNL, pattern matchers, regular expression

engines and as we can see the MutationConfig contains factories for

invoking these guys.

Lets stick these in Camel (MutationManager)

See that Supervisor destination?

The “brian” of the whole system.

• Checks if this is an initial record

• Checks if there is pre-cleansing needed

• Then goes onto in some sub-processors talking to rules and persistence to

build a chain of MutationSets

Then we add an OSGi - Manager (ServiceFactory)

This is purely config driven in OSGi

And the Master system initiation

And showing all the moving parts

Running the system

How did it get to Mock?

System wide “Role” for that particular incoming source file

was established, it was added last in line.

The roles are just written to a Cassandra layer, the

supervisor and the MutationManager executes them.

And the operations?

Runtime loaded

• Part of a config chain - Applied as roles in a processing chain

• Processing chain determined from config and built by the ServiceManager(s)

• Can be hot deployed and loaded

• Can be changed on running processing

End result?

100% Event driven

100% Asynchronous

No camel knowledge really needed

Competing consumer - Horizontally scalable

Hot-Deployable

• And most of the cool stuff I can’t even show you…..

• This pattern is quite well documented in OSGi, not that often combined with

Camel but you end up with all of the parts that really are interesting in

something like AKKA

And what else is Savoir doing?https://github.com/savoirtech/aetos

• Cooler than Karaf :)

https://github.com/savoirtech/hecate

• Hector based Pojo Mapper

Object Graph

Annotations

• CQL3 based Pojo Mapper

Object Graph

Annotations

True collection handling

column adapters

https://github.com/seijoed/hazelcast-discovery

• HazelCast based ActiveMq discovery system

We might be releasing

Polymorphic CRUD

• Based on CXF

• Fully Cassandra (or whatever) backed

• Event driven

• Single bean wiring with entity Class

And we might be looking more at AKKA :)

Questions?