Using XSLT primitives for Data Transformation in ESB

Introduction:

Within a mediation module, the message is received from a service requester before it is sent to a service provider. The structure of the message from a service requester is not always the same as that of a service provider. One of the common uses of mediation is to transform data from one format to another. We have several data transformation primitives in WebSphere ESB. Here we are going to learn about XSLT Primitive. We will see different functionalities which we have in XSLT.

Overview:

In IBM® WebSphere® ESB, XSLT mediation primitives are the usual component for data transformation and mapping. The XSLT module offers basic built-in functions such as string, substring, floor, and ceiling for simple data transformation. But for complex mapping requirements such as Date/time format conversion, or for working with Java objects that XSLT does not support such as java.math.BigDecimal, we can use the custom Java bean functionality and custom Java functions for data transformation. This article will describe the custom Java bean functionality, explore the conditional data mapping techniques using xsl:choose, and conclude with a discussion of error handling.

XSL generation

An XSLT primitive offers two choices for data mapping: generating XSL through mapping created with the Mapping Editor interface, or else using your own XSL, though this second option is discouraged because any changes you make when using the Mapping Editor and rebuilding the project will overwrite the XSLT.
Implementation:

We will complete the following steps to implement application using XSLT primitive.

1. Create BO, Interface and Mediation.
2. Implement XSLT Primitive.
3. Test XSL Transformation.
4. Test the Application.

Create BO, Interface and Mediation:

Create a mediation Module called as ‘EAI_FlightBookingMediation’ under Data Types create the Business Objects for our application to store the date. The following Business Objects create before creating Interface.

Create two interfaces called ‘FlightBookingInterface’ and ‘FlightBookingService’ as shown below.

Open Assembly diagram and add ‘FlightBookingInterface’ as an interface to FlightBookingMediation component and add ‘FlightBookingService’ as a reference.

Dar and drop a Java component in Assembly diagram and rename as ‘FlightBookingService’. Wire from FlightBookingMediation to ‘FlightBookingService’ java component.

Implement the Java component and add the following code to flightBooking (DataObject request) method.

System.out.println("Flight Booking Request Received :: "+request);

String Response = null;

Response = "Request received successfully";

return Response;

Implement the FlightBookingMediation component as a blank mediation flow and add the following primitives as shown below.

Implement XSLT Primitive:

Now we will learn how to create a XSL transformation and implantation here. Drag and drop a XSL transformation from Transformation Pallet rename as ‘XSLT_Request’ drag a wire from MessageLogger_Entry to XSLT_Request and from XSLT_Request to SI_FlightBookingService.

Similarly add one more XSL transformation for response mapping. The flow should be look like below.

Now we will implement the transformation mapping for XSLT_Request and XSLT_Response.

Double click on XSLT_Request and Give the name as ‘XSLT_Request’ then click on next.

Select the Message Root as / and make sure the Input Message body and Output Message Body selected correctly then click on finish.

Now the XSLT_Request mapping will open in new window. It will be look like below.

Expand the source and destination smo body parts and wire as shown below, to Move the source values to destination.

Here the move work like copy the source values to the destination values.

Drag a wire from DepartDate to Depart in destination, from the move drop down select FormateDate under Date and Time Functions.

Select the Formate Date à Properties à General and select the Pattern from drop down as shown below.

Drag a wire from DepartDate to JournyDate and double click on Local map to implement local mapping transformation.

Drag a wire from DepartDate to Day, Date, Month, Year field and select the date functions as shown below.

This will give the Day name, Date, Month name and Year will be sent to service.

Rright click on FlightClass and select Create Assign to set static value to specified field.

Select the assign transformation à Properties à General à Give the value as ‘Business Class’

Drag a wire from ReturnDate, IsRoundTrip to return and select If.

Select the If à Properties à Condition and give the condition as $IsRoundTrip == true.

Double click on If and drag a wire from RetrunDate to Return as shown below.

Import the attached EAI_CommonUtility project which hold the utility java classes and add as a Dependency in ‘EAI_FlightBookingMediation’.

Drag a wire from Destination to CountryOfDeparture and select Custom Java Transfrmation.

Select the Custom Java Transformation à Properties à general, select the class name ,method name and parametes as shown below.

The above java class is available in EAI_CommonUtility. It has java code logic to select the country for destination value.

We have lot of functions to perform in XSL transformation. We can perform different string functions, Date functions, Maths functions similarly.

Similarly implement the XSLT_Response XSL transformation. It should be look like below.

 Test XSL Transformation:

WID has built in function to perform the XSL transformation test. We can directly test the XSLT after implementation.

Open the XSLT_Response XSL transformation and click on Test Map.

It will open a view call Test Map view and select ‘Use the Associated XML files’ button to add a sample input file.

Click on Generate Input, give the XML File name and click on Finish then click on Ok.

Now it will generate Input Sample XML file to test XSL Transformation. Now click on Run Transformations.

Now it will generate the response XML file based on the implementation we have in XSLT_Response transformation. We have passed the input as ‘SUCCESS’ and the implementation have Move, so the final response in Output is ‘SUCCESS’.

This input and output XML files we can save for future reference or for future testing purpose.

Test the Application:

Now save the all files, start the WESB server and deploy the EAI_FlightBookingMediation in server.

Open the Assembly diagram and Right click on FlightBookingMediation select test Component.

Give the values as shown below and click on continue.

The output will be look like below.

Select the Request activity in Test client and see the request message which sent to Service. It will be look like below.

Conclusion:

This article showed how to use XSLT functionality for common data type and date conversions. Although the XSLT module is not as feature-rich as the Business Object Mapping Editor provided for SCA components, it is the only option for data mapping in a mediation module apart from using a full-blown Java custom mediation component.

Downloads

File Name	Description	SIZE	Download
XSL Transformation.pdf	XSL Transformation Document	988 KB	Download
XSL Transformation.zip	XSL Sample	32 KB	Download
EAI_CommonUtility.zip	Common Utility	8 KB	Download

continue reading

Best Practices and Tuning for Large Objects in WESB

Introduction

Ensuring optimum performance is attained on systems processing large objects is an issue commonly faced by users of middle-ware software. In general, objects of 1M or more can be considered to be 'large' and require special attention. This article aims to provide you with the necessary information and advice required to successfully utilize the WebSphere Enterprise Service Bus (ESB) V7 product to process large objects efficiently in a 64 bit production environment.

Considerations and affecting factors

This section provides information on the main considerations and affecting factors when processing large messages.
JVM Limitations

The main advantages of 64bit architectures relate to memory management and accessibility. The increased data bus width enables support for addressable memory space above the 4GB generally available on 32bit architectures. Although the limit for the size of the Java heap is operating system dependent, it is not unusual to have a limit of around 1.4 GB for a 32-bit JVM. The increased memory support that 64bit architectures deliver alleviates the constraints on Java heap sizes that can become a limiting factor on 32bit systems when performing operations on large data objects.

As a general rule you should always run with 64-bit JVMs to service large objects.
Size of in-memory BO

It should be noted that the size of the in-memory business object (BO) can be much larger than the representation available on the wire. There can be several reasons for this, notably character encoding differences, modifications made as the message flows through the system, and copies held of the BO during a transaction to allow for error handling and roll-back.
Number of Concurrent Objects

The achievable response time is primarily inversely proportional to the number of concurrent objects being processed – although modern SMP hardware is helping to alleviate these limitations to a degree. In order to achieve the best possible response times from your system you can limit the number of messages being concurrently processed – this is of particular note when processing large data objects due to possible stresses placed on the Java heap.

Limiting the number of concurrently processed messages can be achieved by:

• Restricting the number of clients used to drive the workload.
• Tuning the appropriate thread pools to restrict the number of concurrent threads.

Network
Network bandwidth can be a limiting factor when processing large messages. If we consider a simple client-server model where the client is sending a negligibly sized request message and receiving a 50MB response over a 1Gbit LAN then the maximum theoretical throughput can be calculated as follows:

Bandwidth (1000Mbits) / Message Size (400Mbits) = 2.5 Messages per Second

This equates to an achievable response time of 400ms assuming a single client thread.

In reality, the nominal transfer rate of a network interface card (NIC) is not achievable at the application layer due to the overheads of the lower layers (TCP/IP etc.). A maximum throughput of around 70% of the NIC rating is not unusual.

When processing messages over a multi-tiered configuration (Figure 1) the network load on the middle tier is effectively double that of the client or service provider – this has the effect of halving the achievable throughput from the scenario defined above.

Application design patterns

This section provides a number of design patterns to improve performance for processing of large messages.

Decomposition of an input message is a technique that aims to decompose a large message into multiple smaller messages for individual submission.

If the large message is primarily a collection of smaller business objects then the solution is to group the smaller objects into conglomerate objects less than 1MB in size. If there are temporal dependencies or an "all or nothing" requirement for the individual objects then the solution becomes more complex.
Claim Check Pattern:

The claim check pattern pertains to a technique for reducing the size of the in-memory BO when only a few attributes of a large message are required by the mediation.

1. Detach the data payload from the message
2. Extract the required attributes into a smaller 'control' BO
3. Persist the larger data payload to a data store and store the 'claim check' as a reference in the 'control' BO
4. Process the smaller 'control' BO, which has a smaller memory footprint
5. At the point where the solution needs the whole large payload again, check out the large payload from the data store using the 'claim check' key
6. Delete the large payload from the data store
7. Merge the attributes in the 'control' BO with the large payload, taking the changed attributes in the 'control' BO into account

CO-Located Services
The most significant solution architecture that should be implemented is to utilize a separate JVM (dedicated server) for processing large messages, especially if you are running a transaction mix of small messages payloads (high throughput / low response times) and large message payloads. This technique should be employed even if the large message payloads are only occasional but exhibit relatively long response times.

On systems that host a number of services, with a mix of services that handle large and small message payloads, the GC and message processing overhead incurred due to handling the larger messages can have a detrimental effect on the performance of the other services.

If we take two example services:

• ServiceA – predominantly handles large message payloads
• ServiceB – predominantly handles small message payloads (high throughput / low response times)

Ensuring that ServiceA is located on a separate JVM to ServiceB has multiple benefits:

• GC and message processing overhead of handling larger messages on ServiceA does not affect the high throughput and low response time performance of ServiceB as dramatically
• You can independently tune the separate JVMs so that they are optimized for the expected workloads

Performance tuning

This section provides information and advice on a number of tuning options that should be understood and correctly configured to obtain optimum performance.
JVM Tuning

This section describes tuning considerations relating to the JVM.

What is Garbage Collection?

Garbage Collection (GC) is a form of memory management for the JVM. The trigger for a GC is usually an allocation failure – this occurs when the allocation of an object to the JVM heap fails due to insufficient available space. The aim of the GC is to clear up the JVM heap of any objects that are no longer required, thus providing enough space for the object that previously failed allocation. If a GC was triggered and there is still not enough room for the object then you have exhausted the JVM Heap.

Generational GC is a policy that is best suited to applications that create many short-lived objects, which is typical of middleware solutions. The JVM Heap is split into three sections (Allocate Space, Survivor Space and Tenured Space) and although this provides performance optimizations in a number of situations, when processing large messages you need to be aware of how the JVM Heap is being utilized. Due to the JVM Heap size constraints this can be a limiting factor on 32 bit JVMs and it is recommended that on such architectures you do not use the Generation GC policy for processing large messages. This is not the case on 64 bit JVMs due to the increased memory support.

Increase the size of the JVM Heap?

Processing a number of large messages, especially when running with concurrent threads, can lead to JVM Heap exhaustion. Increasing the size of your JVM Heap can alleviate the majority of cases where JVM Heap exhaustion has been an issue – however, a balance is needed so that side-effects of this change do not inhibit the performance.

Increasing the size of your JVM Heap to compensate for JVM Heap exhaustion will result in more objects being able to be allocated before a GC is triggered. This has the side-effect of increasing the interval times between GCs, and increasing the time it takes to process an allocation failure.

When in a GC all other JVM threads are temporarily blocked – thus if you have a Global GC that regularly takes 3 seconds to complete, and a Service Level Agreement (SLA) on response times of 1 second, then if a Global GC occurs during that transaction the 1 second response time will be exceeded.

If you are running on a 32-bit JVM (not recommended for large object processing) you can maximize the space available to process large BOs by not using generational garbage collection. This results in a “flat heap”, where the entire heap space is available for transient object allocation rather than just the nursery space.

Is there an alternative approach?

If multiple large messages are being processed by a service at the same time, then available space within the JVM Heap can quickly disappear. Limiting the number of Web Container Threads will give the administrator additional control over the number of messages being concurrently processed. This can help alleviate the issue of Heap exhaustion without the need to increase the JVM Heap to an excessive size.

Additionally, you could ensure that only a single large message is being processed at once by using a single client to drive messages into WebSphere ESB – this will help to reduce memory consumption and provide optimal response times. Throttling incoming client requests with large messages to arrive sequentially into WebSphere ESB can be achieved by a front end server such as a Data Power appliance for instance.

The administrative tuning section of this article details the parameters and settings available from the WebSphere ESB Administrative Console.
Administrative Tuning

This section describes a number of relevant parameters, tuning considerations, recommendations and information of where these can be applied in the Administrative Console:

MDB Activation Spec

There are a few ways to access the MDB ActivationSpec tuning parameters:

Resources > Resource Adapters > J2C Activation Specifications > ActivationSpec Name Resources > JMS > Activation Specifications > ActivationSpec Name

There are two properties that need to be considered when processing large messages:

maxConcurrency – this property controls the number of messages that can be concurrently delivered from the JMS queue to the MDB threads.

maxBatchSize – this property determines how many messages are taken from the messaging layer and delivered to the application layer in a single step.

Thread Pools

The following thread pools will typically need to be tuned:

• Default
• ORB.thread.pool
• WebContainer

The maximum size of these thread pools can be configured under Servers > Application Servers > Server Name > Thread Pools > Thread Pool Name

JMS Connection Pool

There are a few ways to access the JMS Connection Factories and JMS Queue Connection Factories from the Admin Console:

Resources > Resource Adapters > J2C Connection Factories > Factory Name

Resources > JMS > Connection Factories > Factory Name

Resources > JMS > Queue Connection Factories > Factory Name

From the admin panel for the connection factory open Additional Properties > Connection Pool Properties. From here you can control the maximum number of connections.

Conclusion

The increased memory support that 64bit architectures deliver alleviates the constraints on Java heap sizes that can become a limiting factor on 32bit systems when performing operations on large data objects. Increasing the size of your JVM Heap can alleviate the majority of cases where JVM Heap exhaustion has been an issue – however, a balance is needed so that side-effects of this change do not inhibit the performance.

Ø  Tune the JVM appropriately to balance GC intervals and GC pause times.

Ø  Consider available design patterns aimed at reducing the stress on the JVM.

Ø  Use a dedicated server for processing large messages.

Ø  Constrain concurrency or single-thread requests through the large message server.

Downloads

File Name	Description	Size	Download
Handling large objects in WESB.pdf	Best practices of using large objects in WESB Document	270 KB	Download

continue reading

Implement Service Gateway using IBM WESB V7.0

Introduction

WebSphere® ESB V7.0 introduces the Proxy Gateway pattern extending the capabilities of the Static and Dynamic Service Gateway patterns that were introduced in WebSphere ESB V6.2. This sample demonstrates a typical Service Gateway scenario showing built-in routing capabilities of the proxy gateway.

Service Gateway is a typical mediation has a single service consumer and a single service provider (or multiple service providers of the same type). The mediation acts as an intermediary between the two entries completing any processing, such as message manipulation, on the way. There is a separate mediation for each service provider and service consumer. This scenario is shown in Fig 1.

By using the Service Gateway pattern logic the mediation can be written once and used across all service interactions. The Service Gateway pattern provides the ability to have a single mediation that will handle the request for multiple service consumer types and service provider types, as shown in Figure 2. 

Overview

This sample creates a Proxy Gateway, which is a type of Service Gateway.

Architecture of the Proxy Gateway

The concept of a Proxy Gateway is based on the primary goal of a Service Gateway: to provide a generic capability across a multitude of service providers. The diagram below shows the relationship between a Proxy Gateway, Proxy Group, Service Provider and Virtual Service.
General processing template of a Gateway where Service consumers go into the mediation component through a virtual service.

Ø  Proxy Group: collection of Service Providers that can be grouped together into a logical set.

Ø  Proxy Gateway: The entities that will complete the Gateway function for one or more Proxy Groups.

Ø  Virtual Services: the exposed endpoint of the Proxy Gateway for each service provider within the associated Proxy Group

Implementation

We need to complete the following steps to build our application.

Ø  Create a Proxy Gateway.

Ø  Deploy Proxy Gateway.

Ø  Business Space configuration.

Ø  Test the application.

Create a Proxy Gateway:

Use the Proxy Gateway wizard to build a Proxy Gateway:

Open WebSphere® Integration Developer in a new workspace.

Select Window > Show view > Patterns Explorer to open the Patterns view, as shown below.

Create a new Proxy Gateway in the Patterns Explorer tab. Right-click Proxy Service Gateway and select Create New Instance.

Enter EAIProxyGateway as the Integration Solution name and click OK. The New Pattern Instance panel opens, as shown in below.

Accept the default configuration for the Proxy Gateway and click Generate. The Proxy Gateway wizard generates the required artifacts and a summary panel opens.

Scroll down and click the requestResponse Request Flow diagram to open the Mediation Flow Editor. A single Gateway Endpoint Lookup primitive is created by default.

Select the Gateway Endpoint Lookup Primitive and go to properties à Details. The panel should similar to this:

The details section shows the Lookup Method selected (URL) and populates a default Proxy Group called EAIProxyGateway ProxyGroup. When the application is installed, WebSphere ESB will check if EAIProxyGroup exists. If not, WebSphere ESB will create a new Proxy Group within the built-in configuration store and create an association between the Proxy Gateway and Proxy Group.

When the application is removed from the runtime the association between the Proxy Gateway and Proxy Group is removed, but the Proxy Group configuration remains. You can redeploy the application without losing configuration information associated with the Proxy Group (such as Virtual Services).

You can delete the Proxy Group manually using the Business Space console, when there are no associated Proxy Gateways.

The generation of the Proxy Gateway is now complete and the mediation module can be deployed on the runtime environment.

Deploy Proxy Gateway:

To be able to run and test this application it must be built and deploy the application.

If you did not build the sample yourself, you must import the completed sample application. Download the samples from Download Section and import the Proxy Gateway.zip project interchange file.

When you have imported the project, build and deploy the EAI_AccountServiceand the EAIProxyGateway_Gateway projects to a WebSphere ESB 7.0 server. Use the Add/Remove Projects option in the context menu to define a server in the Servers view, or install the server by using the Administration Console.

Business Space configuration:

To access the Business Space console open a Web browser and go to: http://<hostname>:<port>/BusinessSpace.

For example, http://localhost:9444/BusinessSpace

Note: the <hostname>:<port> name varies based on your server configuration.

The login screen is displayed.

In the User ID name field enter admin, and in the Password field, enter admin.

Note: When using Business Space for the first time the welcome screen is displayed. Within Business Space there are spaces, templates, pages and widgets. A space is a collection of pages that can be built based on a template, and a page can contain a number of widgets. WebSphere ESB and WebSphere Process Server include a number of built-in templates and widgets to provide an enhanced ready-to-use experience.

Create a new space so that the Proxy Gateway widget is available. You can do this by creating a new space and page, then adding the Proxy Gateway widget to that page; or by using the Service Administration template:

Go to Manage Spaces > Create Space.

In the Space Name field, enter Proxy Service Administration.

Select the Create a new space using a template check box. Select the Service Administration template from the menu. Click Save. You are returned to the Space Manager page.

Click the Proxy Service Administration Space link.

Select Proxy Gateway. The Proxy Gateway section includes the EAIProxyGatewaywidget. This widget shows the associations between a Proxy Gateway and Proxy Group.

Our first task is to associate a service provider with the Proxy Group, and create and expose a virtual service on the Proxy Gateway:

Click the EAIProxyGatewayProxyGroup row and select the pencil icon at the end of that row.

The Proxy Group view is displayed and all the associated services are listed. Ensure you have deployed the EAI_AccountService file.

In the WSDL Location field, enter http://localhost:9081/EAI_AccountServiceWeb/sca/AccountServiceExport1?wsdl

Click Add Service.

The WSDL will be processed by WebSphere ESB runtime and a subset of the information is displayed, as shown in below figure. Select the Enable Virtual Service check box and click Save.

The widget shows the one virtual service deployed within the Proxy Group.

Test the application:

When we have completed the steps to configure your settings in Business Space, requests can be routed and resolved WSDLs can be returned. This section shows the mechanism of testing the Proxy Gateway using the Web Service Explorer within WebSphere Integration Developer:

In WebSphere Integration Developer, go to Window > Open Perspective > Other > Web.

To start the Web Services Explorer, click Run > Launch Web Services Explorer.

Select the WSDL page icon as shown in below figure.

Click WSDL Main Link. The WSDL page loads.

Enter in the WSDL URL field: 
http://localhost:9081/EAIProxyGateway_GatewayWeb/sca/
EAIProxyGateway_GatewayExport_WS_SOAP11/AccountServiceExport1_AccountServiceHttpService?wsdl,

and click Go. This loads the WSDL into the Web Service Explorer.

Click the getAccountDetails operation. Enter a value for the account number and click Go. The expected result is shown below:

Conclusion

This article introduced you to the proxy gateway pattern, which provides a Web service gateway that is fast to build and meets the majority of requirements with an out of the box experience. The article started with an overview of service gateway patterns and then went into detail on the architecture of a proxy gateway. It showed you how the built-in configuration store is administered using Business Space. The development tools provided, specifically the Patterns Explorer used to generate a proxy gateway implementation, was described. An end-to-end example showed you how to build, configure, and test a proxy gateway. 

Downloads

File Name	Description	Size	Download
Proxy Gateway.zip	Proxy Gateway Sample	32 KB	Download
Service Gateway in IBM WESB.pdf	Service Gateway in IBM WESB Document	635 KB	Download

continue reading