An Earth Shattering Smattering of Scattering

Just when you thought the Aggregator had wrapped things up, it’s time to dive into another essential enterprise integration pattern. Buckle up as we continue our exploration of Gregor Hohpe and Bobby Woolf’s Enterprise Integration Patterns. Building upon what we have learned from the Aggregator, we now move to the dynamic realm of Scatter-Gather (distribution style).

We’re expanding our logistical use-case by sending out requests to our carriers to find the most affordable service, and we’ll demonstrate how to implement this in MuleSoft.

The Scatter-Gather

“How do we combine the results of individual but related messages so they can be processed as a whole?”

Use a Scatter-Gather that broadcasts a message to multiple recipients and re-aggregates the responses back into a single message.

Our Use-Case

Imagine we have split our shipment into individual packages, and for each package, we want to check with multiple carriers to find the most cost-effective service. How do we send those requests to multiple destinations simultaneously and gather their responses efficiently?

A 'Distribution' what-now?

There are two types of Scatter-Gather variants: Auction and Distribution. The Distribution variant allows explicit control over the number of recipients by making the Scatter-Gather aware of each recipient’s message channel.

Key Characteristics of Distribution Style

• Broadcast Mechanism: Messages are sent to all relevant endpoints simultaneously.
• Parallel Processing: Each endpoint processes the message independently and in parallel.
• Aggregation of Responses: The responses are collected and combined to form a single, cohesive output.
• Simplicity: There is no prioritisation or selection among responses; all contributions are equally important.

The Distribution style scatter-gather uses a recipient list pattern to either compute the recipients dynamically, or provide a list of recipients. MuleSoft supports the latter out of the box, where each route of the Scatter-Gather is an entry in the list.

MuleSoft's Scatter-Gather in Action!

Here’s an example of Mule’s Scatter-Gather in action. The Mule event is sent to three routes in parallel. Each route below invokes a subflow that handles the request for a relevant carrier.

Each subflow calls the carrier API service for a different carrier, which may have varied formats and structures in their responses.

Before dealing with the aggregated response (the gather), each subflow harmonises the data by transforming the response into a canonical response model. This simplifies subsequent transformations on what would otherwise be a munging of responses of different shapes and sizes aggregated into some kind of data model abomination. Instead we simply extract the data we want, in the shape we want it to be in. Yay.

Choose cheapest carrier service

Once we have all responses from our carriers (using a wait-for-all strategy), processing exits the Scatter-Gather, adding each response into a single result. This happens implicitly. We then transform the result to select the carrier with the cheapest cost. Here’s an example of the DataWeave used to achieve this:

This yields a payload similar to:

Until Next Time

Next time, we’ll explore the Auction variant of the Scatter-Gather pattern. We’ll see how to incorporate a message broker, like RabbitMQ, to achieve this pattern and ensure we select the most optimal response from multiple bids. Stay tuned to learn how the Auction style brings a competitive edge to message processing. This literally couldn't get more exciting.

By the way...

The Mule source code that accompanies this blog is available here. Happy days!