Heal the Pain of Split-Join Forever with GenericParallel
I bet that you too hate split-join if you ever had to use it - the developer experience from hell. But thank God for heroic effort of Vladimir Dyuzhev we have GenericParallel that wraps the horrible "interface" of the Oracle implementation so that using this powerful optimization pattern becomes a joy. That's pure magic!
The description of the split-join for uninitiated straight from the horse's mouth:
A split-join is a mediation pattern that can be used in a Service Bus to direct the flow and processing of messages. Split-joins let you split a service payload, such as an order, into individual messages that are sent to multiple services concurrently, as opposed to standard sequential processing. This greatly improves service performance. Split-join achieves this by splitting an input message payload into sub-messages (split), routing them concurrently to their destinations, and aggregating the responses into one overall return message (join). This process of payload splitting and response aggregation is called a split-join pattern.
Split-joins are particularly useful for optimizing overall response times in scenarios where payloads delivered by faster systems are being directed to responding services on slower systems. Without split-join, individual messages in a payload are normally resolved in sequential order by the recipient, which can take a long time if the responding system is slow. With split-joins, multiple messages are processed simultaneously, which reduces burden on the responding system and greatly improves response times. Without a split-join, the overall response time is the sum of the individual response times for each message. With a split-join, the overall response time is roughly that of the longest individual message's response time plus some minor system overhead.
The only drawback about GenericParallel is that we didn't start using it from the day zero, so we still have few split-join implementations in our code base lurking the poor maintainer. We have been running quite a few integrations using it with zero problems.
It's so easy to use that you might want to use it even there is no need to run calls parallel, but the calls are independent, and you can process the combined result set after all the calls.
GenericParallel has enterprise friendly BSD license so you have no excuses not to use it.
Custom XPath Functions
The ability to write custom XPath functions is a powerful feature that enables you to keep simple things simple.
With custom XPath functions you can write an arbitrary complex Java code that you'll be able to call in any context where you can call standard XPath functions. This is also the main benefit when compared to Java callout - you are calling the code in more developer friendly contexts: in XQuery and XSLT transformations and in OSB assign/replace actions. This doesn't render Java callout useless, thought.
Pay attention not to start overusing custom XPath functions. Instead consider them as a small-scale building blocks that enables you to do generic things that otherwise are hard or impossible in OSB. I.e. don't implement your integration/orchestration logic there. Also think twice if your home-made Java code should be a Java callout or custom XPath function.
We have successfully used custom XPath functions for e.g.
- base64 encoding
- JSON to/from XML conversion (see also Don't Use NXSD for JSON to XML)
- our own configuration framework* that completely replaces Oracle's half-baked customization files
- run-time introspection**
*The framework consists of environment specific configuration file structure and a bunch of custom XPath functions used to access the data.
**We're also able to query the details of all deployed OSB projects run-time. The details include all external service back-end properties and the relations between the OSB projects (remember we've separated web service and business service) and the external services.
Don't Use NXSD for JSON to XML
Native format translation (aka NXSD) is a useful feature but we run into serious development issues with it when we used it for JSON to/from XML conversions***. As many aspects in OSB also NXSD is very static and requires exact mapping between the formats. The mapping needs to be created and maintained manually by a developer with the help of a (JDeveloper) wizard.
The problem we run into was that the mapping needs to be fixed manually every time the JSON of the back-end changes. And you have to fix it even your business process is not using the changed part of the JSON (e.g. you only read a subset of the data).
So, every time the backend changed even a single bit in the JSON the NXSD was broken and we had to fix it even we didn't used the changed part of the JSON. This was a big waste of development time and source of frustration.
When we considered how to heal the pain, we noticed XPath 3.1 has a build-in generic conversion to and from JSON. Unfortunately, OSB has only Oracle flavored version of XPath 2.0 so we didn't get that out of the box. Fortunately, it turned out it's not that hard to write the W3C conforming conversion by yourself! Now we have own generic json-to-xml and xml-to-json XPath functions in our toolbox. The only minor drawback here is that the generic XML is a bit more complex than NXSD generated one but that's a minor issue compared to the benefits.
Our previous development workflow with NXSD in case we had to read JSON data:
- Figure out every single detail of the JSON we need to able to read even we only need a subset. In many cases there wasn’t any description like Swagger/OpenAPI but we had to figure out the JSON structure exploratory.
- Make the NXSD mapping with the help of (occasionally unhelpful) wizard.
- Use the transformed XML for business profit!
Creating NXSD mapping first time (steps 1 and 2) could easily took hours and every time the JSON changes you’re forced back to step 1.
With the help of the in-house developed conversion the development workflow is now:
- Create a single assign/replace action using the custom XPath conversion function. This is a simple 15 seconds operation and you can then forget the conversion forever.
- Use the transformed XML for business profit!
Now we're immune to all those changes to the JSON data model we don't care. Of course, if the data we're using changes we need to adapt.
Because we have separated OSB business service from the integration service the conversion is invisible from the integration service (where the business value exists) point of view.
By eliminating the dependency to the fragile technical gimmickry and manual design-time task we've eliminated hundreds of hours of unnecessary work and tons of developer frustration.
Don't get it wrong though, NXSD is a solid solution that has it uses for, but it didn't work for us in scenarios where JSON payload changes constantly.
***Recall we're providing SOAP web services with XML payload.