© 2014 IBM Corporation

WebSphere Application Server Liberty profile support for Java Batch and the JSR-352 Specification

Dave Sudlik (IBM)dsudlik@us.ibm.com

Jhonatan Okata (SICOOB)jhonatan.okata@sicoob.com.br

Session ASZ-1841

Please Note:• IBM’s statements regarding its plans, directions, and intent are subject to change or

withdrawal without notice at IBM’s sole discretion.

• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.

• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

2

Online and batch processing differ primarily in the type of interaction and expectations for responsiveness

Request

Response

Request

Response

In general: Interaction is one-for-one ... that is,

request with matching response Expectation is for response to follow

request in a near-immediate time frame Examples: Inventory query, Website

shopping transaction, eBanking account withdrawal

Online Processing

Submit

Complete

Start

EndProcess

In general: Interaction is one-for-many... that is, initial

request results in many results from processing

Expectation is for results to finish within some determined non-immediate time frame

Examples: Month end tax calculation, Period end statements and reports, Data transformation, Data analysis

Batch Processing

Interaction and ResponsivenessHigh Low

Online and Batch Processing Defined

24 x 7 x 365 AccessUsers of your online systems expect availability at all hours.

Users from other parts of the world means availability is expected around the clock.

Mobile UsersUsers are no longer tied to a desk and a computer. Today users have access to mobile computing devices that are with the user wherever they may be. Day or night, home or office.

Online

Batch

Online

Batch

In the past ... Today ...

Windows of time which used to be dedicated to batch processing are shrinking, while batch workloads continue to grow as online transactions increase

The need to process batch work has not gone away.

The need to perform batch concurrent with online has emerged.

The need to process batch work has not gone away.

The need to perform batch concurrent with online has emerged.

Batch processing is evolving

It's not just a shrinking batch window... it's also the pressures to reduce duplication in staff and infrastructures supporting batch and online environments

Batch Support Staff Online Support Staff

Maintaining homegrown infrastructures for separate online and batch processing results in increased costs and decreased business focus

Maintaining homegrown infrastructures for separate online and batch processing results in increased costs and decreased business focus

Redundant staff and IT are under the microscope

Integration with existing processes and technologies allow for preserving investments

and incremental adoption of batch

Integration with existing processes and technologies allow for preserving investments

and incremental adoption of batch

Evolving batch processing to leverage Java enables alignment with online application skills,

and more efficient skills utilization

Evolving batch processing to leverage Java enables alignment with online application skills,

and more efficient skills utilization

Consolidation around common tools and technologies enable shared services, code re-

use and support optimizations

Consolidation around common tools and technologies enable shared services, code re-

use and support optimizations

Online and Batch execution within a common runtime, managing workloads by priority to

respond to capacity fluctuations and meet SLAs

Online and Batch execution within a common runtime, managing workloads by priority to

respond to capacity fluctuations and meet SLAs

How can WebSphere Batch address the issues?

11:00pm Midnight 1:00a 2:00am 3:00am

Common Support Staff

JavaJava

Existing Batch

Existing Batch

ExistingBatch

ExistingBatch

Integrated Batch and Online Shared Services and IT Infrastructure

Incremental AdoptionBetter Resource and Skills Utilization

Execution cost savings

Efficient development tools

Common runtime / management

Abundant availability of skills

A comprehensive batch execution platform, built on WebSphere Application Server and providing development tools, runtime, and management

Container-managed Services

Web Container

Application Web Modules

Container-managed Services

EJB Container

Application EJB Modules

Container-managed Services

Batch Container

Batch Applications

WebSphere Application Server Runtime Environment

Batch job dispatching and management system

Job resiliency services (skip record, step retry)

Data record read and write support services

Parallel job management and execution services

Checkpoint and job restart services

COBOL module call services

Batch Services

Development framework for batch

Incremental adoption through integration with existing processes

Concurrent processing of batch and online workloads

Policy driven workload optimization to meet business goals

Expedites the completion of batch jobs through parallel job execution

Completely and tightly integrated into WebSphere Application Server

What is WebSphere Batch?

WebSphere batch capabilities are fully integrated in WebSphere Application Server (WAS) as of v8.5. For installations prior to WAS

v8.5, the WebSphere Compute Grid product is required.

WebSphere batch capabilities are fully integrated in WebSphere Application Server (WAS) as of v8.5. For installations prior to WAS

v8.5, the WebSphere Compute Grid product is required.

JSR-352 and Java Batch

• JSR-352 Specification started in 2011 by IBM as spec lead

• Based on programming models and experience from Compute Grid, z/OS Batch, and Spring Batch.

• Specification developed through the JCP process by industry leaders in the batch space.

• Approved by expert group and accepted as part of Java EE 7

• Finally, a standard for the batch programming model!

The Java Batch Programming Model

• Batchlets and Chunks• Unmanaged and managed batch step processing• Ability to stop, restart, checkpoint a job

• Listeners• Provide hook points into various lifecycle events of a job

• Splits and Partitions• Allow work to be parallelized

• Restart, skip, and retry capabilities• Allow for increased resiliency of a long running batch job

• Job Specification Language• XML based job definition with built in dependency injection

1010

WAS v8.5.5 Liberty Profile &

WAS Developer Tools for Eclipse (WDT)WDT Remote

Unzip install and deployIM or unzip to install. New option to

deploy “server package” of app + config + required subset of server runtime for highest density deploy

Unzip install and deployIM or unzip to install. New option to

deploy “server package” of app + config + required subset of server runtime for highest density deploy

Liberty ExtensionsAdd custom features and

integrate 3rd party components via Liberty

extensions interface

Liberty ExtensionsAdd custom features and

integrate 3rd party components via Liberty

extensions interface

Fidelity to full profile WASSame reliable containers & QOS.

Develop on Liberty profile and deploy to Liberty or full-profile WAS

Fidelity to full profile WASSame reliable containers & QOS.

Develop on Liberty profile and deploy to Liberty or full-profile WAS

Integrated toolsPowerful tools in WDT Eclipse feature. Enhanced for v8.5.5

prog models, Maven integration, ++

Integrated toolsPowerful tools in WDT Eclipse feature. Enhanced for v8.5.5

prog models, Maven integration, ++

Lightweight cluster MgmtLiberty servers can join a

lightweight cluster for workload balancing and high availability

Lightweight cluster MgmtLiberty servers can join a

lightweight cluster for workload balancing and high availability

Dynamic Server ProfileNot static like Web Profile;

configured by app at a fine-grained level

Dynamic Server ProfileNot static like Web Profile;

configured by app at a fine-grained level

Start fast, run efficientlyStarts in <3s; Mem footprint

<50MB; (TradeLite benchmark)

Start fast, run efficientlyStarts in <3s; Mem footprint

<50MB; (TradeLite benchmark)

Dynamically ExtensibleInstall new features from repository (local or remote) with no svr restart

Dynamically ExtensibleInstall new features from repository (local or remote) with no svr restart

“Developer First” FocusSimplified, shareable XML server config. New

integrated messaging server, DynaCache support, new prog. models, such as Web Services, JMS &

EJB-Lite.

“Developer First” FocusSimplified, shareable XML server config. New

integrated messaging server, DynaCache support, new prog. models, such as Web Services, JMS &

EJB-Lite.

Small Download50MB for Web Profile features

Small Download50MB for Web Profile features

Web Profile CertifiedCreate web apps for the Java

EE Web Profile standard.

Web Profile CertifiedCreate web apps for the Java

EE Web Profile standard.

Why Java Batch and Liberty?We have wrapped all the Qualities of Service of Liberty Profile around the Java Batch programming model.

Java Batch and the Value it Provides

JSR 352//JCL

COBOL or Assembler

Java main()

Java Java

Traditional Batch Processing

Early Java Batch Processing

Standardized Java Batch Processing

The essential business value of batch processing is the same; the methods and tools change and bring added value:

Java for Batch Processing•Availability of programming skills•Offload of Java to z Systems specialty engines•Leverage modern development tooling•Priority schedule both online and batch transactions

JSR 352 Java Batch•All the benefits of Java Batch (left)

•Standardized programming interface

•Standardized = greater skill availability

•Platform / vendor neutrality

WebSphere JSR-352 Java Batch Qualities of Service

• REST API• Java Batch in the Liberty profile

provides an easy to use rest interface to remotely manage your batch jobs

• Ability to start, stop, restart, view job instance and execution data, and access job logs

• Job Logging• Server logs are interleaved with

application records for easy debugging

• External scheduler integration • Provides the ability to combine

enterprise quality scheduling with batch

JSR-352JSR-352

JobJob

LoggingLogging

RESTREST

APIAPI

External SchedulerExternal Scheduler

SupportSupport

Beta1Q15

More about tooling

• WebSphere Developer Tools

• Create Java Batch Applications using the JSR-352 programming model and XML job definitions

• Job Creation Wizards• JSL Editor• Java class wizards• Simple to submit, test, and debug applications• Remote deployment and debug capabilities• Simple graphical UI to monitor jobs

Partnership with SICOOB

• SICOOB has made extensive use of the Java Batch / JSR-352 support available in the Liberty profile beta:

https://developer.ibm.com/wasdev/downloads/liberty-profile-beta/

• They worked closely with onsite IBM team and through direct communication with the development lab to migrate from their existing homegrown batch framework to Liberty Batch.

• We are continuing to work together to scale their batch environment.

• SICOOB provided excellent feedback during the migration process:

•Liberty is released in a much more agile delivery cycle allowing IBM to respond quickly to any feedback•SICOOB was able to pick up our latest runtime and tooling updates as they were released to decrease their end-to-end development time

SICOOB Batch Experience

Who is Sicoob?

16

• The major credit union system of Brazil

Sicoob Growth Rates over previous year

17

Sicoob Legacy Batch (AS IS)

Sicoob Legacy Batch

Three Tiers Applications

Visual Basic 6.0

COM Architecture

MS Com+ as Application Server

Sql Server 2008

Sicoob Legacy Batch Numbers

Legacy

900 Jobs per

Cooperative

458.100

Total of daily job

509 Databases

12 Itanium Server

20 Server

2 vCPU

4 GB

Some of Sicoob Legacy Batch Problems

VB 6.0

Deprecated Technology;Without MS support;STA model;Stateless;

COM+

Difficult to identify bottlenecks;Default isolation level is serializable;Unmanaged distributed algorithm;

Some of Sicoob Legacy Batch Problems

Legacy

Batch

• Without SLA and timeout;• Restrited by COM;• Without grid computing;• Restrited by Cooperative;• Absence of problems analysis tools;

2011: Evolution of Sicoob Batch

COBOL x JavaCOBOL x Java

26

COBOL, COBOL, Sure?Sure?

With 55 years, COBOL is considered the most robust and stable language for environment processing;

In 1997, Gartner estimated that was 180 billion of LOC. Accounting for 80% of business code;

“.. high throughput, high transaction, mainframe environments.”

75 american universities/college teach COBOL, but corresponds only 2%;

+

COBOL in the world

COBOL in the world

“…COBOL skills are highly entrenched in the baby boomer generation” (by Dale Vecchio – Gartner Research VP)

COBOL in the world

In 2007, Cobol topped the list of dying computer skills of ComputerWorld magazine;

Difficult to find skilled developer;

Y Generation considers COBOL archaic, being only attracted by elevated salaries;

Not just knows COBOL, other news skills are necessary, as EBCDIC, VSAM, ADABAS.;

-

And Java And Java support?support?

Java in the world

The most popular language (TIOBe) and the biggest community in the world. Cobol is only 27ªl

Periodic new releases:Open Source;

JCP;

Certification to ensure the professional knowlegde;

Teached in the major of universities/colleges;

+

Java in the world

Java Batch

Sure?Sure?

Sicoob Challenge

• Worldwide reference lack in processing core banking business in Java*;

• Late standardization, only proprietary Frameworks;

• Batch Modernization:• Bring batch concepts to JAVA;• Batch as service;

• Batch execution environment:• Dynamic online transaction and batch runtime;• Share business logic;• Parallel processing of batch jobs; • Enterprise scheduler integration (Job dispatching);

35

Sicoob Java Batch Framework

• In 2012, Sicoob developed his homegrown Java Batch Framework;

36

Flow• Logical grouping of all jobs to be

scheduled and executed;• No code is necessary, it will be used

only in design time;• Cron expression;

Job• Logical grouping of work;• Discover the list of step to

process;

Step• Atomic work to be

performed;• No checkpoint algorithm

is necessessary;

Sicoob Java Batch Framework

Sicoob Java Batch Framework

Interface

Database

Sicoob Java Batch Framework

Design Time

Number of executions

depends on the scheduling

Sicoob Java Batch Framework

Sicoob Java Batch Framework Numbers

• Scheduler runs 24 x 7 with high availability;

• More than 60 Main Flows;

• More than 500K jobs daily;

• 16 Application Server running in Linux and Windows Server;

41

Why Sicoob has decided to migrate to JSR352 and WAS Liberty Profile?

Sicoob Motivations

• Final release of JSR352 was approved in 2013.• In few years will be more expensive to maintain and train

developer in our framework;

• Sicoob’s presented challenges were technically overcome

• IBM Development Tools and IBM Support;

• Effort to migrate AS instances to WAS• http://whywebsphere.com/2014/05/07/right-sizing-your-

application-server-environment-liberty-was-was-nd-vs-tomcat-jboss-weblogic-impact-2014-session/

43

Batch Use Case : JSR 352 + WAS Liberty Profile

Batch Use Case Goals

• Transcribe de code and adept from homegrown framework to JSR 352;

• Use and evaluate WebSphere Developers Tools and WAS Liberty Profile (Beta version);

• Compare execution time between Sicoob Batch Framework and new framework over JSR 352;

45

Development Environment

• Eclipse Luna;

• WebSphere Developers Tools (WDT);

• Liberty Profile October - Beta Version;

• Java Batch Feature• javaBatch-1.0;

• managedBatch-1.0;

46

Overview of Operations Classification (COP)

47

A B C D

D D E F

A A A A

Overview of Operations Classification (COP)

• Daily execution;

• Numbers in Dec/2014• Total of Classification : 6.471.670;• Total of Clients: 1.917.089;

49

Batch Flow

50

COP Main FlowCOP Main Flow

Sub Flow: Preparing Data

3 Jobs

Sub Flow: Preparing Data

3 Jobs

Sub Flow:Calculating Risk Level

Jobs

Sub Flow:Calculating Risk Level

Jobs

Sub Flow:Final

4 Jobs

Sub Flow:Final

4 Jobs

51

How does it work?

52

Job: getSteps()

Step: execute()

or

Sicoob Scheduler

COP using JSR352

53

How does it work?

54

Using REST

Scheduler Tool

Points of our development process

• Understand the difference between our framework and JSR352 to transcribe our code as chunk or batchlet;

• When batchlet, identify itemReader, itemProcessor and itemWriter;

• This is the greatest effort of our work;• A good business logic layer makes the migration easier;

• Abstract flow’s orchestration;

• Transfer the transaction controller to batch container;

56

Batch Use Case Results

57

Batch Use Case Results

58

Batch Use Case Results

59

Overall Batch Use Case Experience

• Performance of JOBs over JSR352 is linked with the architecture model applied, so it is important to anticipate the analysis of batch routines;

• Documentation for batch is quite satisfactory• IBM Knowledge Center• WASDev blog• GitHub JSR352 Repository

60

Overall Batch Use Case Experience

• Support:• WasDev forum has quickly response time;

• WebSphere Development Tool;

61

Get a lightning start: Key InterConnect 2015 sessions

#2829 Mon 2:00-3:00 Liberté, Égalité, Fraternité—Viva the CICS and WebSphere Mini Revolution

#1670 Tue 3:30-4:30 IBM WebSphere Liberty z/OS Connect: A Technical Introduction

#4032 Wed 9:30-10:30 Meet the Winners of IBM’s Mainframe Mobile App Throwdown 2014

#6482 Thu 11:00-11:50 What is the Mainframe Mobile App Throwdown and How Do I Register?

Register now at

ibm.biz/mmathrowdownand submit your app by April 12th

Mobilize your mainframe in our fun app-building contest for

z Systems clients & business partners!

The Mainframe Mobile App Throwdownis back for 2015!

Notices and DisclaimersCopyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Notices and Disclaimers (con’t)

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

•IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank YouYour Feedback is

Important!

Access the InterConnect 2015 Conference CONNECT Attendee Portal to complete your session surveys from your smartphone, laptop or conference

kiosk.