WebSphere MB & MQ V7 - Message Display, Test & Performance Utilities

Abstract:

This Support Pac contains utilities (rfhutil) that are useful for the development and testing of WebSphere MQ and WebSphere Message Broker Version 7 applications.

Description:

This Support Pac contains a utility that is useful for the development and testing of WebSphere MQ and WebSphere Message Broker Version 7 applications. Test messages are stored as files, which are then read by the application and written to a WebSphere MQ queue. The program is GUI based.

Command line performance measurement utilities are also provided.

Possible Uses:

It allows test messages to be captured and stored in files, and then used to drive WBI Message Broker Version 7 applications. Output messages can also be read and displayed in a variety of formats. The formats include two types of XML as well as matched against a COBOL copybook. The data can be in EBCDIC or ASCII. An RFH2 header can be added to the message before the message is sent.

Software version:

6.0, 6.1, 7.0

Operating system(s):

AIX, HP-UX, Linux, OS/390, Solaris, Windows, z/OS

Skill Level Required:

None beyond basic MQ and Message Broker development skills.

New in this Release:

The following changes have been made in this release:

• Support for WebSphere Message Broker V7.0.

• Support for WebSphere MQ V7.0.1.

Prerequisites:

Ø  WebSphere MQ server or client. 

Please note: RFHUtil requires either MQ or the MQ client installed to start up. For MQ client only customers please use RFHUtilc.exe (the client version) all other customers should use RFHUtil.exe.
 Installation Instructions:

URL	LANGUAGE	SIZE(MB)	Download Options
ih03.pdf	US English	1.12	HTTP

Downloads

Download	RELEASE DATE	LANGUAGE	SIZE(MB)	Download Options
ih03.zip	11 Nov 2009	US English	2.6	HTTP
RFHTool.pdf	18 Apr 2014	US English	0.35	HTTP

continue reading

Design Blueprint to implement a lightweight ESB

Introduction

This article introduces a design blueprint for implementing a homegrown, lightweight Enterprise Service Bus (ESB) that leverages commonly used Integrated Development Environments (IDEs) and middleware. The scenario uses IBM® WebSphere® Application Server V8, a robust deployment environment for Java EE applications, and IBM Rational® Software Architect, which provides tooling to design, develop, test, and package Java EE applications for deployment to an application server.

Business scenario Challenge

In order to meet business demands, a service provider is required to deliver multiple releases of its application within a year. Each release results in updating the existing service contract and deploying a new version of the web service into a production environment.

The most cost-effective solution would typically be a single instance of the service deployed in production to reduce maintenance costs. However, due to budget restrictions or maintenance schedules, service consumers are not always in a position to adopt new versions as soon as they are released. This scenario forces the service provider to host multiple versions of the service to support client needs.

Solution

WebSphere Enterprise Service Bus (WebSphere ESB) and WebSphere DataPower® Appliances both provide robust ESB capabilities that enable dynamic routing and transformation of requests between service consumers and providers. These capabilities enable the decoupling of service consumers from the instance of the service that processes the request. By leveraging an ESB, service providers can honor multiple service contracts without hosting multiple instances of the service in production.

When only a subset of the capabilities provided in WebSphere ESB and WebSphere DataPower are required, customers have the option of building lightweight ESB capabilities into their Java EE applications to meet their project-specific needs.

Blueprint for implementing a lightweight ESB:

A blueprint for implementing a lightweight ESB should consist of several layers to handle the different functions needed to receive, route, transform, and respond to requests from a client. The blueprint outlined in this article includes the following four layers:

Ø  Routing Layer -- A Java API for RESTful Web Services (JAX-RS) RESTful endpoint for URL pattern matching and request routing.

Ø  Mediation Layer -- Mediation and mapping modules for managing requests at the data content level between a client and a specific service version.

Ø  Service Invocation Layer -- Manager for the services contract that builds the required request, invokes a physical service endpoint, and returns the content with the response.

Ø  Multi-Protocol Listener Layer -- A listener service such as WebSphere MQ, Java Message Service (JMS), or another service for non-JAX- RS client applications to participate in the lightweight ESB.

Templates and guidelines for implementation

The following sections provide details about the templates and guidelines for each layer in the blueprint.

Routing Layer

The Routing Layer is the entry point into the lightweight ESB. The primary purpose of the Routing

Layer is to provide clients with well-known service endpoints and routing of client requests to the appropriate mediation logic. In doing so, the Routing Layer decouples the service consumers from the physical location of the service provider's endpoint.

The solution template provided for the Routing Layer is a JAX-RS RESTful web service. This layer intercepts client requests and reroutes them, based on URL, to the appropriate Mediation Layer entry point. Below Figure shows the following functions of the Routing Layer:

Ø  Two service consumers -- MyService version A client and MyService version B client -- which use different URLs when invoking the ESB.

Ø  Routing components for MyService version A client and MyService version B client which route requests based on URL to the appropriate mediation component.

The JAX-RS Routing Layer provides two URLs – one for MyService version A clients and one for MyService version B clients. Each URL refers to a method in the URL Routing Resource class that handles the routing of the client request to the appropriate mediation logic:

URL Routing Resource class

public class URL_Routing_Resource {

    private URLRouterImpl urlRouterImpl = new URLRouterImpl();

    // Constructor

    public URL_Routing_Resource() {}

    @GET

    @Path (value="myService/versionA/")

    @Produces("application/xml")

    public Response invokeMyServiceVersionA(String payload)

    {

        String content = null;

        System.out.println("");

        content = urlRouterImpl.myServiceVersionA(payload);

        return Response.ok(content).build();

    }

    @GET

    @Path (value="myService/versionB/")

    @Produces("application/xml")

    public Response invokeMyServiceVersionB(String payload)

    {

        String content = null;

        System.out.println("");

        content = urlRouterImpl.myServiceVersionB(payload);

        return Response.ok(content).build();

    }

    @GET

    @Path (value="otherService/versionX/")

    @Produces("application/xml")

    public Response invokeOtherServiceVersionX(String payload)

    {

        String content = null;

        System.out.println("");

        content = urlRouterImpl.otherServiceVersionX(payload);

        return Response.ok(content).build();

    }

 }

Listener Layer (optional)
The Listener Layer is an optional component of the blueprint and is dependent on the need for batch or non-HTTP type interfaces. The primary purpose of the Listener Layer is to provide ESB access to a broader range of clients that leverage common non-HTTP based transport mechanisms, such as WebSphere MQ, JMS, File Transfer Protocol (FTP), and Remote Method Invocation (RMI). You can create the Listener Layer to intercept an inbound non-HTTP request, build an HTTP request from the incoming data, invoke the Routing Layer via HTTP, and build a non-HTTP client response. Technologies to implement the Listener Layer can include an MQ Listener, JMS Listener, FTP Listener, or EJB Listener. Figure 3 shows the following functions of the Listener Layer:

Ø  Two non-HTTP clients -- An MQ service consumer for MyService version A client, and an FTP service consumer for MyService version B client.

Ø  Two Listener Layer mapping modules that map the incoming request to an HTTP JAX- RS request and invoke the Routing Layer. In addition, the response is also mapped as required by the client and the appropriate response returned to the client.

Mediation Layer

The Mediation Layer serves as the central data content mapping point for all incoming and outgoing service requests. The primary purpose of the Mediation Layer is to provide mapping capabilities between client-aware versions of a service and physical service endpoint versions. In addition, the Mediation Layer ensures that adequate data-related content required by a specific service version is present prior to calling the Service Invocation Layer.

The solution template provided for the Mediation Layer leverages Java classes to programmatically determine a service version, and perform inbound mapping to fill in missing data content required to call a physical service endpoint. This layer invokes the Service Invocation layer and performs outbound-related mapping for the client-specific response based on version. The Mediation Layer could also use XSLT or other mapping technologies or products, such as WebSphere Transformation Extender, to provide data content mapping. Figure 4 shows the following functions of the Mediation Layer:
Ø  An inbound mapping module for clients of MyService that supports versions A and B.
Ø  An outbound mapping module for clients of MyService that supports versions A and B.
Ø  An additional example of a mapping module for clients that would support multiple versions of a future service (depicted as OtherService).

The Mediation Layer provides an invocation entry point for MyService (version A clients and version B clients), in addition to an entry point for OtherService. The Invoke method uses the incoming service name to call the service-specific mediation logic:

Invoke method in Mediation Layer

/*

 * Entry point into Mediation Layer for all services

 */

    public MediationObject invoke(MediationObject mo)

    {

        if(mo.getServiceName().equalsIgnoreCase("myService"))

            return invokeMyService(mo);

        else if(mo.getServiceName().equalsIgnoreCase("otherService"))

            return invokeOtherService(mo);

        else

            return null;

    }

The logic within the service-specific mediation determines which version of client made the request, and executes mapping logic to transform the incoming message content prior to passing the message to the Service Invocation Layer. When the response is returned from the Service Invocation Layer, outbound mapping logic is performed to transform the message content into the format expected by the requesting client:

Mediation logic for MyService

/*

 * Mediation logic for MyService

 */

    private MediationObject invokeMyService(MediationObject mo)

    {

        Mapper mapper = new Mapper();

        MediationObject outgoingMO = new MediationObject();

        // 1. Determine service version

        if (mo.getVersion().equalsIgnoreCase("A"))

        {

        // 2. Invoke the Mapping Layer to perform any required outbound mapping

            ServiceInvocationObject sio =  mapper.serviceRequestMapper(mo);

        // 3. Call the Invocation Layer to physically invoke the service

            InvocationLayer invoke = new InvocationLayer();

            ServiceInvocationResponse sir = invoke.callService(sio);

        // 4. Invoke the Mapping Layer to perform any required inbound mapping

            outgoingMO = mapper.serviceResponseMapper(sir);

        }

        else

            if(mo.getVersion().equalsIgnoreCase("B"))

            {

                // Repeat Steps 2 - 4 above

            }

            else

                return null;

        return outgoingMO;

    }

Service Invocation Layer

The Service Invocation Layer serves as the entry point for calling a physical service. The primary purpose of the Service Invocation Layer is to implement the client code to invoke a physical service endpoint. The Service Invocation Layer takes the data content from the Mediation Layer and constructs the request required by the physical service endpoint. The physical service is then invoked, and the Service Invocation Layer constructs a response object that is returned to the Mediation Layer.

The solution template provided for the Service Invocation Layer leverages Java classes to implement the client code, which constructs the service endpoint request from the data content provided by the Mediation Layer. This Layer constructs the physical service request object, invokes the physical service endpoint, and converts the response object back to the format required by the Meditation Layer. Figure 5 shows the following functions of the Service Invocation Layer:

Ø  A module to construct the request object, invokes the physical service endpoint, and converts the response to the format required by the Mediation Layer.

Ø  An additional example of a module to construct the request object, invoke the physical service endpoint, and convert the response to the format required by the Mediation Layer for a future service (depicted as Other Service).

The Service Invocation Layer provides an invocation entry point for MyService and an entry point for OtherService. The callService method uses the incoming service name to call the service-specific invocation logic:

The callService method in the Service Invocation Layer

/*

 * Entry point for Service Invocation Layer

 */

    public ServiceInvocationResponse callService(ServiceInvocationObject sio)

    {

        if(sio.getServiceName().equalsIgnoreCase("myService"))

            return callMyService(sio);

    else if(sio.getServiceName().equalsIgnoreCase("otherService"))

            return callOtherService(sio);

        else

            return null;

    }

The logic within the Service Invocation Layer determines which version of a service is being requested by a client, constructs the endpoint-specific request, submits the request to the endpoint, and then converts the endpoint response to the format expected by the Meditation Layer:

/*

 * Service Invocation logic for myService

 */

    private ServiceInvocationResponse callMyService(ServiceInvocationObject sio)

    {

        ServiceInvocationResponse sir = new ServiceInvocationResponse();

        // 1. Build physical service endpoint specific object from SIO payload

        // 2. Call physical service endpoint

        // 3. Build SIR payload from endpoint specific response object

        // 4. Return SIR to Mediation Layer

        return sir;

    }

Conclusion

This article introduced a design blueprint for implementing a homegrown, lightweight ESB. The blueprint consists of four layers that outline the different functionalities needed to receive, route, transform, and respond to requests from a client. To illustrate the use of the blueprint, the article described a business scenario and provided code templates and guidelines for implementing a lightweight ESB. You can use this article to design and implement your own lightweight ESB to meet your specific project requirements.
Downloads

File Name	Description	Size	Download
Design blueprint to implement a lightweight ESB	Document	480 KB	Download

continue reading

Resolving out of memory errors on WAS or WPS

Problem (Abstract):

Depending on your flow design and memory capacity, out-of-memory problems can be seen when you run WAS or WPS or WESB.

Resolving the problem:

Increase the heap size to its optimal size to prevent out of memory errors from occurring when you are working with large and complex applications.

Use the steps in the procedure to increase the heap size. However, note that if the heap size is too large, the system does not have enough physical memory, and uses virtual memory to hold the data that it is using. Virtual memory can be used up sooner than expected if you are running other memory intensive applications at the same time. Also, virtual memory is slower than physical memory.

Physical memory size (GB)	Optimal heap size (MB)
1.5	768
2	1024
4	2048

Note: Incase if you are using 64 Bit Operating System and 4 GB Physical Memory, we should keep 4096 MB Heap Size

Procedure

1. In the WebSphere® Application Server administrative console, select the name of your server.
   
2. In the configuration tab, under Server Infrastructure, click Java and Process Definition > Process Definition.
   
3. Click Java Virtual Machine.
   
4. In the Maximum Heap Size field, enter 2048 and in the Initial Heap Size field, enter 1024 then Click OK.
   
5. Click Save.
   
6. Log out, and then restart the server.

continue reading

Best Practices and Tuning for Large Objects in WESB

Introduction

Ensuring optimum performance is attained on systems processing large objects is an issue commonly faced by users of middle-ware software. In general, objects of 1M or more can be considered to be 'large' and require special attention. This article aims to provide you with the necessary information and advice required to successfully utilize the WebSphere Enterprise Service Bus (ESB) V7 product to process large objects efficiently in a 64 bit production environment.

Considerations and affecting factors

This section provides information on the main considerations and affecting factors when processing large messages.
JVM Limitations

The main advantages of 64bit architectures relate to memory management and accessibility. The increased data bus width enables support for addressable memory space above the 4GB generally available on 32bit architectures. Although the limit for the size of the Java heap is operating system dependent, it is not unusual to have a limit of around 1.4 GB for a 32-bit JVM. The increased memory support that 64bit architectures deliver alleviates the constraints on Java heap sizes that can become a limiting factor on 32bit systems when performing operations on large data objects.

As a general rule you should always run with 64-bit JVMs to service large objects.
Size of in-memory BO

It should be noted that the size of the in-memory business object (BO) can be much larger than the representation available on the wire. There can be several reasons for this, notably character encoding differences, modifications made as the message flows through the system, and copies held of the BO during a transaction to allow for error handling and roll-back.
Number of Concurrent Objects

The achievable response time is primarily inversely proportional to the number of concurrent objects being processed – although modern SMP hardware is helping to alleviate these limitations to a degree. In order to achieve the best possible response times from your system you can limit the number of messages being concurrently processed – this is of particular note when processing large data objects due to possible stresses placed on the Java heap.

Limiting the number of concurrently processed messages can be achieved by:

• Restricting the number of clients used to drive the workload.
• Tuning the appropriate thread pools to restrict the number of concurrent threads.

Network
Network bandwidth can be a limiting factor when processing large messages. If we consider a simple client-server model where the client is sending a negligibly sized request message and receiving a 50MB response over a 1Gbit LAN then the maximum theoretical throughput can be calculated as follows:

Bandwidth (1000Mbits) / Message Size (400Mbits) = 2.5 Messages per Second

This equates to an achievable response time of 400ms assuming a single client thread.

In reality, the nominal transfer rate of a network interface card (NIC) is not achievable at the application layer due to the overheads of the lower layers (TCP/IP etc.). A maximum throughput of around 70% of the NIC rating is not unusual.

When processing messages over a multi-tiered configuration (Figure 1) the network load on the middle tier is effectively double that of the client or service provider – this has the effect of halving the achievable throughput from the scenario defined above.

Application design patterns

This section provides a number of design patterns to improve performance for processing of large messages.

Decomposition of an input message is a technique that aims to decompose a large message into multiple smaller messages for individual submission.

If the large message is primarily a collection of smaller business objects then the solution is to group the smaller objects into conglomerate objects less than 1MB in size. If there are temporal dependencies or an "all or nothing" requirement for the individual objects then the solution becomes more complex.
Claim Check Pattern:

The claim check pattern pertains to a technique for reducing the size of the in-memory BO when only a few attributes of a large message are required by the mediation.

1. Detach the data payload from the message
2. Extract the required attributes into a smaller 'control' BO
3. Persist the larger data payload to a data store and store the 'claim check' as a reference in the 'control' BO
4. Process the smaller 'control' BO, which has a smaller memory footprint
5. At the point where the solution needs the whole large payload again, check out the large payload from the data store using the 'claim check' key
6. Delete the large payload from the data store
7. Merge the attributes in the 'control' BO with the large payload, taking the changed attributes in the 'control' BO into account

CO-Located Services
The most significant solution architecture that should be implemented is to utilize a separate JVM (dedicated server) for processing large messages, especially if you are running a transaction mix of small messages payloads (high throughput / low response times) and large message payloads. This technique should be employed even if the large message payloads are only occasional but exhibit relatively long response times.

On systems that host a number of services, with a mix of services that handle large and small message payloads, the GC and message processing overhead incurred due to handling the larger messages can have a detrimental effect on the performance of the other services.

If we take two example services:

• ServiceA – predominantly handles large message payloads
• ServiceB – predominantly handles small message payloads (high throughput / low response times)

Ensuring that ServiceA is located on a separate JVM to ServiceB has multiple benefits:

• GC and message processing overhead of handling larger messages on ServiceA does not affect the high throughput and low response time performance of ServiceB as dramatically
• You can independently tune the separate JVMs so that they are optimized for the expected workloads

Performance tuning

This section provides information and advice on a number of tuning options that should be understood and correctly configured to obtain optimum performance.
JVM Tuning

This section describes tuning considerations relating to the JVM.

What is Garbage Collection?

Garbage Collection (GC) is a form of memory management for the JVM. The trigger for a GC is usually an allocation failure – this occurs when the allocation of an object to the JVM heap fails due to insufficient available space. The aim of the GC is to clear up the JVM heap of any objects that are no longer required, thus providing enough space for the object that previously failed allocation. If a GC was triggered and there is still not enough room for the object then you have exhausted the JVM Heap.

Generational GC is a policy that is best suited to applications that create many short-lived objects, which is typical of middleware solutions. The JVM Heap is split into three sections (Allocate Space, Survivor Space and Tenured Space) and although this provides performance optimizations in a number of situations, when processing large messages you need to be aware of how the JVM Heap is being utilized. Due to the JVM Heap size constraints this can be a limiting factor on 32 bit JVMs and it is recommended that on such architectures you do not use the Generation GC policy for processing large messages. This is not the case on 64 bit JVMs due to the increased memory support.

Increase the size of the JVM Heap?

Processing a number of large messages, especially when running with concurrent threads, can lead to JVM Heap exhaustion. Increasing the size of your JVM Heap can alleviate the majority of cases where JVM Heap exhaustion has been an issue – however, a balance is needed so that side-effects of this change do not inhibit the performance.

Increasing the size of your JVM Heap to compensate for JVM Heap exhaustion will result in more objects being able to be allocated before a GC is triggered. This has the side-effect of increasing the interval times between GCs, and increasing the time it takes to process an allocation failure.

When in a GC all other JVM threads are temporarily blocked – thus if you have a Global GC that regularly takes 3 seconds to complete, and a Service Level Agreement (SLA) on response times of 1 second, then if a Global GC occurs during that transaction the 1 second response time will be exceeded.

If you are running on a 32-bit JVM (not recommended for large object processing) you can maximize the space available to process large BOs by not using generational garbage collection. This results in a “flat heap”, where the entire heap space is available for transient object allocation rather than just the nursery space.

Is there an alternative approach?

If multiple large messages are being processed by a service at the same time, then available space within the JVM Heap can quickly disappear. Limiting the number of Web Container Threads will give the administrator additional control over the number of messages being concurrently processed. This can help alleviate the issue of Heap exhaustion without the need to increase the JVM Heap to an excessive size.

Additionally, you could ensure that only a single large message is being processed at once by using a single client to drive messages into WebSphere ESB – this will help to reduce memory consumption and provide optimal response times. Throttling incoming client requests with large messages to arrive sequentially into WebSphere ESB can be achieved by a front end server such as a Data Power appliance for instance.

The administrative tuning section of this article details the parameters and settings available from the WebSphere ESB Administrative Console.
Administrative Tuning

This section describes a number of relevant parameters, tuning considerations, recommendations and information of where these can be applied in the Administrative Console:

MDB Activation Spec

There are a few ways to access the MDB ActivationSpec tuning parameters:

Resources > Resource Adapters > J2C Activation Specifications > ActivationSpec Name Resources > JMS > Activation Specifications > ActivationSpec Name

There are two properties that need to be considered when processing large messages:

maxConcurrency – this property controls the number of messages that can be concurrently delivered from the JMS queue to the MDB threads.

maxBatchSize – this property determines how many messages are taken from the messaging layer and delivered to the application layer in a single step.

Thread Pools

The following thread pools will typically need to be tuned:

• Default
• ORB.thread.pool
• WebContainer

The maximum size of these thread pools can be configured under Servers > Application Servers > Server Name > Thread Pools > Thread Pool Name

JMS Connection Pool

There are a few ways to access the JMS Connection Factories and JMS Queue Connection Factories from the Admin Console:

Resources > Resource Adapters > J2C Connection Factories > Factory Name

Resources > JMS > Connection Factories > Factory Name

Resources > JMS > Queue Connection Factories > Factory Name

From the admin panel for the connection factory open Additional Properties > Connection Pool Properties. From here you can control the maximum number of connections.

Conclusion

The increased memory support that 64bit architectures deliver alleviates the constraints on Java heap sizes that can become a limiting factor on 32bit systems when performing operations on large data objects. Increasing the size of your JVM Heap can alleviate the majority of cases where JVM Heap exhaustion has been an issue – however, a balance is needed so that side-effects of this change do not inhibit the performance.

Ø  Tune the JVM appropriately to balance GC intervals and GC pause times.

Ø  Consider available design patterns aimed at reducing the stress on the JVM.

Ø  Use a dedicated server for processing large messages.

Ø  Constrain concurrency or single-thread requests through the large message server.

Downloads

File Name	Description	Size	Download
Handling large objects in WESB.pdf	Best practices of using large objects in WESB Document	270 KB	Download

continue reading