Apache Camel - Stéphane Kay - April 2011

Apache CamelStephane Kay

\\Ville de Lausanne\\

Scope

IS Integration

EIP

Routing

Mediation

Camel Container

Unit testing

Hands-on

Real Use Case

JMS / Active MQ

JBI / NMR

OSGI / ServiceMix

Scala

Table of contents

I. IS Integration heuristics◦ Common issues

◦ Paradigms

II. Camel’s bleat◦ Concepts responding to integration needs

◦ Getting started

III. What’s under the humps?◦ Camel architecture

◦ Distribution

IV. Camel in Action◦ Persistence, transactions, concurrency, failover

◦ Aggregation

V. A Real Use-Case◦ Description, issues & achievement

I. IS Integration Heuristics

Just do it ! Just coding simple things might be just not so simple …

public class FileCopier {

public static void main(String args[]) throws Exception {

File inboxDirectory = new File("data/inbox");

File outboxDirectory = new File("data/outbox");

outboxDirectory.mkdir();

File[] files = inboxDirectory.listFiles();

for (File source : files) {

if (source.isFile()) {

File dest = new File(

outboxDirectory.getPath()

+ File.separator

+ source.getName());

copyFile(source, dest);

}

}

}

private static void copyFile(File source, File dest)

throws IOException {

OutputStream out = new FileOutputStream(dest);

byte[] buffer = new byte[(int) source.length()];

FileInputStream in = new FileInputStream(source);

in.read(buffer);

try {

out.write(buffer);

} finally {

out.close();

in.close();

}

}

}

I. IS Integration heuristics

A Framework?

The bad :◦ Learning effort consumes time.

◦ Adds anarchical dependencies.

◦ Risk of coming with a new need / not implemented

The good :◦ Gain value after a training period

◦ Leverage your code to a higher abstraction

◦ Potentially a future standard ? Indicator : community commits & posts curve.

The plus-value◦ Not obvious


Separation of concerns

IS Integration involves

1. Endpoints

A common endpoint interface abstraction

2. Data Processing

Using Message / exchange abstraction and

conversion tool

3. Routing

A common vocabulary / pattern as DSL

foundation.


Message Endpoint pattern

Routes should refer to a common abstract

Message Endpoint rather than directly using their

technical interface

A good Endpoint abstraction is a URI description

I. IS Integration | endpoint

Uniformed Exchange

Message’s container during routing

II. Camel’s bleat | data

Data Transformation

Data Format Transformation

◦ Message body is transformed from one

format to another.

CSV -> XML

Data Type Transformation

◦ Java Type of message body is

transformed/casted into another.

java.lang.String -> java.jms.TextMessage

java.io.File -> java.lang.String

I. IS Integration | data

Use case : Filtering Route

◦ How can a component avoid receiving

uninteresting messages ?

◦ …coming from a different system

I. IS Integration | routing

EIP?

A Catalog of Design patterns for common IS integration issues

http://camel.apache.org/enterprise-integration-patterns.html
















EIPs Samples… Filter

◦ How can a component avoid receiving uninteresting messages?

Splitter

◦ How can we process a message if it contains multiple elements, each of which may have to be processed in a different way?

Router

◦ How can you decouple individual processing steps so that messages can be passed to different filters depending on a set of conditions?

Aggregator

◦ How do we combine the results of individual, but related messages so that they can be processed as a whole?


Putting all needs together

Route definition attributes :◦ Endpoints Producer / Consumer(s)

◦ Processor(s) Uniformed exchange processing interface

◦ Predicate(s)

A route language should provide thosekey abstractions

Endpoint A = endpoint(“activemq:queue:quote”);

Endpoint B = endpoint(“mq:quote”);

Predicate isWidget = xpath(“/quote/product = „widget‟”);

from(A).filter(isWidget).to(B);


I. Review

IS Integration paradigms

◦ Endpoint abstraction -> URIs?

◦ Data Tx -> Functions // Java beans ?

◦ Routing -> EIPs


II. Camel’s bleat

Apache Camel is a Powerful Open

Source Integration Framework based

on known Enterprise Integration

Patterns

*

Mission Statement :

Making integration easier and more

accessible to developers

Endpoint URI

A camel endpoint URI consists of

three parts :ftp://riders.com/orders?username=rider&password=secret

Scheme

Context path Options

II. Camel’s bleat | endpoint

Java DSL : Builder pattern The Builder Pattern (GOF) is central to the Camel Java DSL

public class FileCopierWithCamel {

public static void main(String args[]) throws Exception {

// create CamelContext

CamelContext context = new DefaultCamelContext();

// add our route to the CamelContext

context.addRoutes(new RouteBuilder() {

public void configure() {

from("file:data/inbox?noop=true").to("file:data/outbox");

}

});

// start the route and let it do its work

context.start();

Thread.sleep(10000);

// stop the CamelContext

context.stop();

}

}

II. Camel’s bleat | routing

Data Format transformations

Marshal / unmarshal

from("activemq:QueueWithJavaObjects”)

.marshal().jaxb()

.to("mq:QueueWithXmlMessages");

from("activemq:queue:MY_QUEUE").unmarshal().zip().proce

ss(new UnZippedMessageProcessor());

http://camel.apache.org/data-format.html





Message Translator EIP

Aka Adapter (GOF)

Using a Processorfrom("direct:start").process(new Processor() {

public void process(Exchange exchange) {

exchange.getIn().setBody(in.getBody(String.class) + " World!"); }

}).to("mock:result");

Using Bean bindingfrom("direct:start") .bean(MyBean.class, "hello(String,String)") .to("mock:result");

Using transform() DSLfrom("direct:start").transform(body().regexReplaceAll(“\n”,”<br/>”))

.to("mock:result");


Getting started : IDE

Eclipse

Maven integration (m2eclipse)

◦ http://m2eclipse.sonatype.org/sites/m2e

◦ Check your eclipse.ini : locate your JDK

-vm C:/dev/java/jdk1.6.0_18/bin

Optional : Fuse Camel plugin for

Eclipse

◦ http://repo.fusesource.com/beta/eclipse/u

pdate/

II. Camel’s bleat | first ride

http://m2eclipse.sonatype.org/sites/m2e

http://repo.fusesource.com/beta/eclipse/update/

http://repo.fusesource.com/beta/eclipse/update/

Getting started : Maven

Basic Maven configuration


First Ride

Copy contents from one folder to

another, control final name / dir

JUnit

◦ Startup context

◦ Route definition : File component

◦ Processor / Exchange

◦ Simple Language (File Component)

Delegate JUnit test to Main class


Second Ride

Add some features to JUnit Class

◦ Log

◦ Add failure if file already exist

◦ Predicate : Simple Language

◦ Error handling

Router EIP (Choice)


Spring XML : content-based router

<camelContext>

<route>

<from uri="activemq:NewOrders"/>

<choice>

<when>

<xpath>/order/product = 'widget'</xpath>

<to uri="activemq:Orders.Widgets"/>

</when>

<otherwise>

<to uri="activemq:Orders.Gadgets"/>

</otherwise>

</choice>

</route>

</camelContext>

II. Camel’s bleat | spring

II. Review

Endpoint / Component URIs

Java DSL

How to start / process exchanges

Log, error handling / routing

II. Camel’s bleat

III. What’s under the humps?

High-level architecture

◦ Container & Services

Toolkit provided out-of-the box

Distribution

High-level view

Container responsibilities :

◦ Create and activate the necessary Endpoint instances using the available

Component implementations


Camel Context Services Components

◦ Loads components on the fly either by autodiscovery on the classpath or when a new bundle is activated in an OSGi container.

Endpoints

◦ Contains the endpoints that have been created.

Routes

◦ Contains the routes that have been added.

Type converters

◦ Contains the loaded type converters. Camel has a mechanism that allows you to manually or automatically convert from one type to another.

Data formats

◦ Contains the loaded data formats.

Registry

◦ Contains a registry that allows you to look up beans. By default, this will be a JNDI registry. If you’re using Camel from Spring, this will be the Spring ApplicationContext. It can also be an OSGi registry if you use Camel in an OSGi container.

Languages

◦ Contains the loaded languages. Camel defines 3 different DSLs (Java, XML/Spring, and Scala) and allows you to use many different languages to create expressions.


Out of the box : EIPs

50+ Routing Patterns based on EIPs http://camel.apache.org/enterprise-integration-patterns.html







Out of the box : Components

70+ Endpoint impl. aka « Components »

All available via URI/scheme paradigm

http://camel.apache.org/components.html


http://camel.apache.org/components.html

Out of the box : Data Format

18 Data Formats - uniformed usage :from(“file://input").marshal().zip().to("activemq:queue:MY_QUEUE");






Out of the box : Languages

Predicates & Expressions

http://camel.apache.org/languages.html


http://camel.apache.org/languages.html

Camel distribution

http://camel.apache.org/download.html

Components sources

Examples

Maven Central (current v2.7)

◦ Snapshot Repo<repository>

<id>apache.snapshots</id>

<name>Apache Development Snapshot Repository</name>

<url>https://repository.apache.org/content/repositories/snapshots/</url>

<releases> <enabled>false</enabled> </releases>

<snapshots> <enabled>true</enabled> </snapshots>

</repository>

http://camel.apache.org/download.html

III. Review

Camel is an integration framework

Based on Enterprise Integration Patterns

Routing and mediation

Easy to use DSL to define routes

No heavy specification

No container dependency

Payload agnostic

Connectivity to a great wealth of

transports

IV. Camel in Action

Test support

Transactions

Exceptions

Concurrency

Persistence

Aggregation

Runtime

Camel Test Support

Provides context/registry support

MockEndpoint assertionsresultEndpoint.expectedBodiesReceived(expectedBody);

template.sendBodyAndHeader(expectedBody, "foo",

"bar");

resultEndpoint.assertIsSatisfied();

IV. Camel in Action| test

Using test support

CamelTestSupport Junit

@EndpointInject

@Produce

Filter Pattern

IV. Camel in Action| test

Transactions

Component level

◦ Polling strategy

public interface PollingConsumerPollStrategy {

boolean begin(Consumer consumer, Endpoint endpoint);

void commit(Consumer consumer, Endpoint endpoint, int polledMessages);

boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws

Exception;

}

Routing level

◦ Transactional Client EIP http://camel.apache.org/transactional-client.html

◦ Transacted DSLfrom("jetty:http://localhost/myservice/order").transacted()

.to("bean:validateOrder").to("jms:queue:order");

IV. Camel in Action | tx

http://camel.apache.org/transactional-client.html



Propagation

Define transaction behaviour<bean id="PROPAGATION_REQUIRED" class="org.apache.camel.spring.spi.SpringTransactionPolicy">

<property name="transactionManager" ref="txManager"/>

<property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>

</bean>

<bean id="PROPAGATION_REQUIRES_NEW" class="org.apache.camel.spring.spi.SpringTransactionPolicy"> <property name="transactionManager" ref="txManager"/> <propertyname="propagationBehaviorName" value="PROPAGATION_REQUIRES_NEW"/>

</bean>

Use transacted DSLfrom("direct:mixed") .transacted("PROPAGATION_REQUIRED“)

.setBody(constant("Tiger in Action"))

.beanRef("bookService")

.setBody(constant("Elephant in Action"))

.beanRef("bookService“).to("direct:mixed2");

IV. Camel in Action | tx

Exceptions

onException clause

new RouteBuilder() {

public void configure() {

onException(IOException.class).maximumRedeliveries(3);

from("direct:start") .process("processor1")

.process("processor2") // <--- throws a ConnectException

.to("mock:theEnd");

}

◦ Note : By default, a producer will not redeliver

HandleonException(ValidationException.class)

.handled(true).transform(body(constant("INVALID ORDER")));

from("jetty:http://localhost/myservice/order").transacted()

.to("bean:validateOrder").to("jms:queue:order");

IV. Camel in Action | errors

Dead Letter EIP

The Error Handler propagates error back to the client.

Dead Letter Channel will always moved failed exchanges to this queue.

useOriginalMessage is used for routing the original input message instead of the current message that potentially is modified during routing.

from("jms:queue:order:input").to("bean:validateOrder") .to("bean:transformOrder").to("bean:handleOrder");

errorHandler(deadLetterChannel("jms:queue:dead") .useOriginalMessage().mamimumRedeliveries(5)

.redeliverDelay(5000);

IV. Camel in Action | errors

Concurency

Dealing with data flows means mastering

concurency

◦ Seda Component :from("seda:bar?concurrentConsumers=5").to("file://output");

◦ Thread DSL

How do we avoid the messages that does not

expect a reply to block ?from("jms:queue:order") // do some sanity check validation

.to("bean:validateOrder") .to("mock:validate").threads(20)

// do some CPU heavy processing of the message (we simulate and delay)

.unmarshal(mySecureDataFormat).delay(1000)

.to("bean:handleOrder").to("mock:order");

IV. Camel in Action | toolkit

Sync to Async

toAsync Processor

from("direct:start").to("mock:a").toAsync("direct:bar", 5).to("mock:result");

from("direct:bar").to("mock:b").transform(constant("Bye World"));

How does this work?void process(Exchange exchange, AsyncCallback callback) throws Exception;

◦ With AsyncCallback interface :

void onTaskCompleted(Exchange exchange);

Http Async Producerpublic class JettyHttpProducer extends DefaultProducer implements AsyncProcessor

◦ Then invoke the callback when reply is ready on the Exchange.

◦ Routing with Spring DSL<route> 

<from uri="direct:start"/> 

<to uri="log:+++ request +++"/> 

<to uri="jetty://http://0.0.0.0:9123/myapp" async="true" poolSize="10"/>



<to uri="log:+++ reply +++"/> 

<to ref="result"/>

</route>

Persistence SQL, JDBC and JPA components provided out-of the box

File idempotent consumer with pluggable repository

from("file://inbox?idempotent=true").to("...");

<bean id="fileStore" class="org.apache.camel.processor.idempotent.FileIdempotentRepository">



<property name="fileStore" value="target/fileidempotent/.filestore.dat"/>



<property name="maxFileStoreSize" value="512000"/>



<property name="cacheSize" value="250"/> </bean>

<camelContext xmlns="http://camel.apache.org/schema/spring">

<route>

<fromuri="file://target/fileidempotent/?idempotent=true&idempotentRepository=#fileStore&move=done/${file:name}"/>

<to uri="mock:result"/>

</route>

Stateful EIP : Aggregator

Aggregator EIP

Combine a number of messages together into a

single message.

◦ http://camel.apache.org/aggregator2.html

Completion predicatesfrom("direct:start“).

.aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(3000)

.to("mock:aggregated");

from("direct:start“).aggregate(header("id"), new

BodyInAggregatingStrategy()).completionSize(3) .to("mock:aggregated");

from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy())

.eagerCheckCompletion().completionPredicate(body().isEqualTo("END"))

.to("mock:aggregated");

from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()).

completionSize(header("mySize")) .to("mock:aggregated");


http://camel.apache.org/aggregator2.html

Aggregator Strategy & Fail Over

Strategypublic class NumberAggregationStrategy implements AggregationStrategy {

public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {

if (oldExchange == null) { return newExchange; }

Integer num1 = oldExchange.getIn().getBody(Integer.class);

Integer num2 = newExchange.getIn().getBody(Integer.class);

Integer num3 = (num1 != null ? num1 : 0) + (num2 != null ? num2 : 0);

oldExchange.getIn().setBody(num3); return oldExchange;

}

}

http://camel.apache.org/aggregate-example.html





Running Camel

Known Deployment Options

◦ Standalone Java Application

◦ Java Web Start

◦ Spring Application

◦ Web Application

◦ Google App Engine

◦ J2EE Application

◦ JBI

◦ OSGi

IV. Camel in Action | runtime

Running inside App Server

Servlet component

camel-example-servlet-tomcat

CXF Component

camel-example-cxf-tomcat

Camel Web console

actually « Web Component »

IV. Riding Camel | runtime

IV. Review

Camel is not a server

Camel is lightweight and embeddable

Camel supports Enterprise Integration paradigms from the JEE/Spring world.

Camel is not an ESB though itimplements EIPs.

Camel doesn’t need but supports anymessaging infrastructure (JMS / MQ)

Camel supports messaging at JBI abstraction level

V. A real Use-Case

Multiple source FTP import to a centralized repository

◦ Collect files (0-5 x 2-20 Megs) from N different FTP endpoints, every

hours on a 24/7 basis

◦ End-user configuration

◦ Logs : technical + business

◦ Monitor failures

FTP

SAN

OPCON

Constraints

Big binary files : maybe JMS would not fit

Implementation should fit some common corporate approach to entreprise intergation.

Currently only SAN @ client. Do not need App Server yet.

Client is technical-oriented : would like hands-on.

Keep intact current logs / traces, but must be possible to connect on JDBC.

Would be interested in indexing meta –infos within Alfresco (JSR-170).

Central schedule/alerting infrastructure must be connected.

Nagios could be an option at mid-term.

Real issues

Multiple source endpoints pattern :

◦ not found (!!!).

FTP component is an active poller; no

easy way to stop polling after having

consuming all files.

No easy way to put into configuration

the context path of Endpoint URIs

Other applications…

Camel as an ETL : Excel sources

PeopleSoft to ActiveDirectory

Bidirectional Federal <-> City

Entreprises DB official data

Reconcilliation of mutations streams

Conclusions

Open-Source

◦ Help yourself through debug mode on sources

Maven-based development

◦ « Real » modularity

Real Standards

◦ EIPs

◦ Maven, Spring, Java6.

◦ JBI (Glassfish)

Gateway to the future

◦ OSGI (Fuse ESB)

Camel. What else?

More information needed?

Camel website:◦ http://camel.apache.org

Camel article:◦ http://architects.dzone.com/articles/apache-

camel-integration

FuseSource website:◦ http://fusesource.com

Camel in Action book:◦ http://manning.com/ibsen

Camel in Action Source code:◦ http://code.google.com/p/camelinaction/

http://camel.apache.org/

http://architects.dzone.com/articles/apache-camel-integration





http://fusesource.com/

http://manning.com/ibsen

http://code.google.com/p/camelinaction/

Questions

Apache Camel - Stéphane Kay - April 2011

Technology

Transcript of Apache Camel - Stéphane Kay - April 2011