Are you familiar with the Oracle Service Bus 12c best practices? In his blog, Digia's expert Jani Hurskainen shares his learnings about topics such as virtualized development environment and custom XPath functions.
In this blog we share some of Oracle Service Bus 12c best practices by Digia OSB 12c experts. The focus on these posts is not how to do things but what things should be done.
The advice here has been collected from OSB 12c in-premise environment with strong emphasis on good old-fashioned SOAP web services. OSB 12c does have a decent support for REST too, but there haven’t been opportunities to use that extensively yet. However, even though some of the advice here might not be applicable as such to your business environment, we believe most of them will be.
Please note that all provided links to Oracle's documentation were valid at the moment of writing but might fail in the future as in the past Oracle has been notorious for reshuffling the documentation every now and then.
Even all the stories are scribed by Jani Hurskainen, the work behind the scenes has been a joint effort of a team of Teemu Hautala, Tero Jauhiainen, Ville Kokkala, Tero Marttinen and Panu Partanen.
If you have a special subject in your mind, please leave a comment and we'll consider sharing our best practices about it too!
Table of contents:
Virtualized Development Environment
Development should take place in a virtualized development environment that is identical to production setup and can be run in a developer machine.
The development environment:
- should have the same first-class environment status than the production and shared test environments
- should share the same governance model so that there is no configuration drift by design
- should use the same automated provisioning process and tools than the production environment
You will gain the following benefits when your development environment is on par with the production:
- Any developer can work on any integration as all developers share the same setup. There is no works on my machine anymore as all the environments are identical to production.
- There are no more bugs that appear only in one specific environment but can't be reproduced in development. And on the other way around if the code works in the development environment it also works in test and production environments.
- New developer on-boarding becomes mostly painless because the development environment setup is a turnkey solution.
Because your production infrastructure is already under automated provisioning process there should be no huge extra cost taking development environment into the loop too. The setting up is one-time cost that will become amortized to zero over the lifetime of your OSB infrastructure 5-10 horizon.
How we have implemented development environment virtualization
We have a golden VirtualBox image containing the basic stuff like:
- Oracle Linux 7 Desktop
- Weblogic Server 12
- Oracle Service Bus 12c
- Oracle Database 12c
- JDeveloper 12c
The image is created semi-manually mainly because most of the stuff from Oracle is not provisioning automation friendly, and it’s only one-time smallish effort. On top of the image we have Ansible based automation for tasks that we do again and again like:
- operating system updates
- Java updates
- Web logic and Oracle Service Bus patching
- Web logic and Oracle Service Bus configuration
- TLS certificate management (including alerting about soon-to-expire certificates)
- installation of other important software like Emacs and a modern version of Python
- installation/update of home-made developer settings and tools (the code is checked out from VCS, compiled and installed)
Most of the automation is shared with production setup so there is not much overlap. Also note that developers are allowed and welcome to add any tool to the image they consider necessary.
Every now and then we refresh the golden image so that newcomers don't have to run a longish patching sequence when booting up the first time.
One bonus benefit of this flexibility is that it was very easy to make our production build and deployment process available also to development image. This gives a developer the one button (the button is imaginary, actually it's a command line alias) installation to non-production environments!
Separate OSB Business Service from the Integration Service
OSB provides a component called business service that is a high-level abstraction of an external (from OSB point of view) service endpoint OSB is calling to. However, we've found it's often useful to abstract that event further.
We usually divide the integration code in two parts that in OSB scope means two different OSB projects:
- an actual web service OSB project
- a business service OSB project
The role of the business service OSB project is mainly to act as a technical glue and extra abstraction layer so that we can hide the possible peculiarities of the external systems (and there's plenty of weird behaving legacy out there).
In our cases this gives us the following benefits:
Code Reuse
The same external service endpoint is often used from more than one web service implementation.
Consistent Logging
One of the golden integration rules is to record the details of the messaging with external systems as detailed as possible. Placing the logging functionality in a separate component guarantees the consistent logging, and off-loads this implementation detail from the web service OSB project.
Consistent Error Handling
All business services implement standardized error response structures. This tremendously simplifies web service project error handling as checking if the call to external service endpoint succeeded or failed is now trivial and always done in the same way.
Single Point of Configuration
Business service OSB project is a single point to control the real address, authentication and other variable parameters of the external service endpoint. Yes, OSB 12c does have configuration files and service accounts but both are fragile features and don't work with fully automated build and deployment process. Hence, we created our own configuration framework with the help of custom XPath functions that plays nice with our automated deployments (DevOps, you see). We don't use static business service configuration but instead we always route dynamically.
JSON - XML conversions
OSB is best when working with XML so how about the external systems using JSON? OSB 12c does have JavaScript action but in our experience when we're providing XML based SOAP web service and need to call REST JSON it's just simpler to translate the JSON to XML and then do all the processing in XML (XML processing is where OSB shines (in the relative terms)).
When business service OSB project makes the JSON - XML conversion under the hood a big burden is removed from the web service OSB project implementation and testing. Instead of relying in this case on Native Format Builder (yet another sub-optimal OSB feature) we have implemented a generic JSON to/from XML conversion that conforms W3C XSLT.3.0 XML Representation of JSON. Implementing the conversion was easier than we thought, see Don't Use NXSD for JSON to XML for the details.
Independent Deployment
Controlling the parameters of an external service endpoint and working with the data model are two very different tasks with different life cycles. Business service OSB project is implemented early and changes rarely, web service OSB project is implemented later and potentially keeps evolving extended time period.
Introspection
Thanks to OSB and Weblogic features and some novel ideas from our side, we can visualize connection attributes, dependencies and clients of all integrations. There is also a build-in health checking so we're on the pulse all the time too.
Separation Illustrated
Summary
Business service project never (or almost never, exceptions apply) touches the logical data model. It just does some technical wizardry (see the list above) so that the web service OSB project doesn't have to. In case of external service is a well-behaving SOAP web service, the business service project is just a simple pass-through from data payload point of view. In other words, the business service project never hides data models and the web service project has to know the external service data model.
Heal the Pain of Split-Join Forever with GenericParallel
I bet that you too hate split-join if you ever had to use it - the developer experience from hell. But thank God for heroic effort of Vladimir Dyuzhev we have GenericParallel that wraps the horrible "interface" of the Oracle implementation so that using this powerful optimization pattern becomes a joy. That's pure magic!
The description of the split-join for uninitiated straight from the horse's mouth:
A split-join is a mediation pattern that can be used in a Service Bus to direct the flow and processing of messages. Split-joins let you split a service payload, such as an order, into individual messages that are sent to multiple services concurrently, as opposed to standard sequential processing. This greatly improves service performance. Split-join achieves this by splitting an input message payload into sub-messages (split), routing them concurrently to their destinations, and aggregating the responses into one overall return message (join). This process of payload splitting and response aggregation is called a split-join pattern.
Split-joins are particularly useful for optimizing overall response times in scenarios where payloads delivered by faster systems are being directed to responding services on slower systems. Without split-join, individual messages in a payload are normally resolved in sequential order by the recipient, which can take a long time if the responding system is slow. With split-joins, multiple messages are processed simultaneously, which reduces burden on the responding system and greatly improves response times. Without a split-join, the overall response time is the sum of the individual response times for each message. With a split-join, the overall response time is roughly that of the longest individual message's response time plus some minor system overhead.
The only drawback about GenericParallel is that we didn't start using it from the day zero, so we still have few split-join implementations in our code base lurking the poor maintainer. We have been running quite a few integrations using it with zero problems.
It's so easy to use that you might want to use it even there is no need to run calls parallel, but the calls are independent, and you can process the combined result set after all the calls.
GenericParallel has enterprise friendly BSD license so you have no excuses not to use it.
Custom XPath Functions
The ability to write custom XPath functions is a powerful feature that enables you to keep simple things simple.
With custom XPath functions you can write an arbitrary complex Java code that you'll be able to call in any context where you can call standard XPath functions. This is also the main benefit when compared to Java callout - you are calling the code in more developer friendly contexts: in XQuery and XSLT transformations and in OSB assign/replace actions. This doesn't render Java callout useless, thought.
Pay attention not to start overusing custom XPath functions. Instead consider them as a small-scale building blocks that enables you to do generic things that otherwise are hard or impossible in OSB. I.e. don't implement your integration/orchestration logic there. Also think twice if your home-made Java code should be a Java callout or custom XPath function.
We have successfully used custom XPath functions for e.g.
*The framework consists of environment specific configuration file structure and a bunch of custom XPath functions used to access the data.
**We're also able to query the details of all deployed OSB projects run-time. The details include all external service back-end properties and the relations between the OSB projects (remember we've separated web service and business service) and the external services.
Don't Use NXSD for JSON to XML
Native format translation (aka NXSD) is a useful feature but we run into serious development issues with it when we used it for JSON to/from XML conversions***. As many aspects in OSB also NXSD is very static and requires exact mapping between the formats. The mapping needs to be created and maintained manually by a developer with the help of a (JDeveloper) wizard.
The problem we run into was that the mapping needs to be fixed manually every time the JSON of the back-end changes. And you have to fix it even your business process is not using the changed part of the JSON (e.g. you only read a subset of the data).
So, every time the backend changed even a single bit in the JSON the NXSD was broken and we had to fix it even we didn't used the changed part of the JSON. This was a big waste of development time and source of frustration.
When we considered how to heal the pain, we noticed XPath 3.1 has a build-in generic conversion to and from JSON. Unfortunately, OSB has only Oracle flavored version of XPath 2.0 so we didn't get that out of the box. Fortunately, it turned out it's not that hard to write the W3C conforming conversion by yourself! Now we have own generic json-to-xml and xml-to-json XPath functions in our toolbox. The only minor drawback here is that the generic XML is a bit more complex than NXSD generated one but that's a minor issue compared to the benefits.
Our previous development workflow with NXSD in case we had to read JSON data:
- Figure out every single detail of the JSON we need to able to read even we only need a subset. In many cases there wasn’t any description like Swagger/OpenAPI but we had to figure out the JSON structure exploratory.
- Make the NXSD mapping with the help of (occasionally unhelpful) wizard.
- Use the transformed XML for business profit!
Creating NXSD mapping first time (steps 1 and 2) could easily took hours and every time the JSON changes you’re forced back to step 1.
With the help of the in-house developed conversion the development workflow is now:
- Create a single assign/replace action using the custom XPath conversion function. This is a simple 15 seconds operation and you can then forget the conversion forever.
- Use the transformed XML for business profit!
Now we're immune to all those changes to the JSON data model we don't care. Of course, if the data we're using changes we need to adapt.
Because we have separated OSB business service from the integration service the conversion is invisible from the integration service (where the business value exists) point of view.
By eliminating the dependency to the fragile technical gimmickry and manual design-time task we've eliminated hundreds of hours of unnecessary work and tons of developer frustration.
Don't get it wrong though, NXSD is a solid solution that has it uses for, but it didn't work for us in scenarios where JSON payload changes constantly.
***Recall we're providing SOAP web services with XML payload.
FI
SV
EN