Integration Patterns V: Callbacks and External Events

Sometimes, business processes take time, and you need to wait for something to happen. It can be tricky to build backend software that reliably handles such cases; thankfully, workflow engines like LittleHorse make it a bit easier.

info

This is the fifth and final part in a five-part blog series on useful Integration Patterns. This blog series will help you build real-time, responsive applications and microservices that produce predictable results and prevent the Grumpy Customer Problem.

1. Saga Transactions
2. The Transactional Outbox Pattern
3. Queuing and Backpressure
4. Retries and Dead-Letter Queues
5. [This Post] Callbacks and External Events

Digital applications must support business processes, and not all business processes execute instantaneously. Sometimes, a process must wait for something to happen: an item needs to get back in stock, a delivery must be completed, or a customer must sign a document. In all three of these cases, the software systems you write need to wait for something out of their control to happen, and then resume taking action once it does happen.

For the rest of this blog, we will use the term External Event to refer to such instants in which our overall process needs to wait for something external to happen.

Case Study: Signing a Document

Let's consider a process in which a fictitious real-estate company sends a closing document to a lawyer, waits for execution, and upon completion then updates the status of a listing from "in escrow" to "sold."

The process might look like the following:

External Event Case Study

In order to make it a little bit more tricky, let's add one wrinkle: if the document is not signed within one week, we "cancel" the offer and roll back the flow.

Let's take a look at how this process would be implemented without an orchestrator and with LittleHorse. In both cases, the process will be kicked off by a POST call to the /sign-document endpoint on a web server.

note

We will assume that, once the document is signed, we receive a webhook including some information about which document was signed.

In the Wild West

In the "wild west" without an orchestrator, we don't have a "leader" in the room that is in charge of orchestrating the process from start to finish. Practically, we as software engineers won't write a single piece of code that represents the whole process.

Let's start with the /sign-document endpoint (our entrypoint). This endpoint needs to do the following:

• Send the document to be signed using some external API (eg. DocuSign).
• Note somewhere that we need to cancel the transaction in seven days time.

The first part is easy—just make an RPC call inside your HTTP handler. For the second part, we have two options: use a delayed queue, or write to a table and use a cronjob.

Using Delayed Queues

We briefly discussed delayed queues in Part IV. Some queueing systems, such as AWS SQS and Rabbit MQ, allow you to write messages that are only delivered after some delay.

Our entrypoint HTTP handler would put the message in the delay queue and send the document to be signed. The messages in the queue would be consumed and processed one week later by (yet another) system which consumes the queue, checks to see whether or not the document was signed, and then optionally cancels the listing (if the document wasn't signed).

We would need one final HTTP handler, which is responsible for listening for completed documents and marking the listing as "sold" when the document is signed.

Pseudocode for our HTTP handler for POST /sign-document might look like the following:

@PostMapping("/sign-document")
public void signDocument(@RequestMapping("document") DocumentDetails document) {
    db.updateListingStatus(document.getListingId(), "SIGNING_DOCUMENT");

    String queueName = "check-expired-docusign";
    queue.scheduleMessage(queueName, document.getListingId(), Duration.ofDays(7));
}

warning

The above code is not robust to a sudden crash while the HTTP Handler is handling the request, which could lead to duplicated or dropped processing.

The callback handler (which listens for completed documents) would update the status of the listing to "SOLD" when handling the document. When consuming the message one week later, our consumer would be responsible for checking to make sure that the listing was not yet in the "SOLD" status before taking compensatory actions.

Note that the responsibility for overall business flow is spread across three components:

1. The entrypoint HTTP handler (POST /send-document)
2. The callback handler
3. The consumer.

A Note on CronJobs

If your message queue doesn't have the ability to schedule delayed writes, you aren't fully out of luck. You can spawn a process which periodically checks for listings that have been in the "signing document" status for over a week. This would likely necessitate a modification to the /sign-document endpoint so that you can keep track of when the document was initially signed, which is still more work.

Furthermore, running cronjobs can be challenging from a Day 2 Operations perspective, and we still have the problem of our business logic being spread across three disparate components (which are most likely deployed separately).

With Workflow

Using a workflow engine to handle your callback-driven workflows simplifies two key aspects of your architecture:

1. The deployment model—no need to manage cronjobs or delayed message queues.
2. The mental model—the entire end-to-end control flow lives in your WfSpec, which sits right in front of you.

Additionally, running your processes in a workflow engine gives you all of the observability and reliability features that we have discussed in the first four episodes.

An executable WfSpec for this use-case might look as follows:

var listingId = wf.declareStr("listing-id");
var customerId = wf.declareStr("customer-id");

wf.execute("send-docusign", listingId, customerId);

// wait for the callback
long sevenDays = 60 * 60 * 24 * 7;
NodeOutput eventResult = wf.waitForEvent("document-signed").timeoutSeconds(sevenDays);

// Fail if it doesn't return in a week
wf.handleException(eventResult, "TIMEOUT", handler -> {
    handler.execute("cancel-listing", listingId);
    handler.fail("signing-timeout", "Closing documents were not signed.");
});

wf.execute(
    "complete-listing",
    listingId,
    // Use the output from the external event in the workflow
    wf.format("Closing documents signed by {0}.", eventResult.jsonPath("$.signer"))
);

Analysis

tip

The key point to note here is that, with LittleHorse you get to write a single piece of code (your WfSpec) that controls the entire process end-to-end.

Our document signing system will necessarily have the following components:

1. The initial REST service must have the POST /sign-document endpoint.
2. There must be some system (Task Worker, Cron Job Poller, or Queue Consumer) that handles expired documents.
3. There must also be an implementation of the callback endpoint.

This is the unfortunate reality of event-driven systems: business flows are spread across multiple components. However, using LittleHorse to orchestrate these processes reduces the complexity of the deployment model and your mental model of how the system works.

note

When business flows are spread across many components, that could be multiple components within the same deployable (a modular monolith), or it could mean multiple separately-deployed microservices. Either way, the problem of wrangling end-to-end flow remains.

Wrapping Up

Congrats on making it through the Integration Patterns series! From the Saga Pattern to queues, retries, and the Outbox Pattern, we've gone through some of the more commonly-used tools and patterns in the belt of software engineers who build reliable and responsive systems.

All of these tools are useful; unfortunately, they are also tricky to get right. Thankfully, LittleHorse makes it easy for you to implement these patterns, taking advantage of the reliability and responsiveness they provide while keeping your architecture simple.

Get In Touch!

Whether or not you end up using LittleHorse for your microservices, I truly hope this series of blogs was useful to you. And if you do want to get involved with LittleHorse:

• Try out our Quickstarts
• Join us on Slack
• Give us a star on GitHub!

What's Next

Over the next few weeks, we will announce three big new releases (two open-source, one product). I also plan to write about the connection between Streams/Tables/Workflows, and how Workflow is similar to (but cooler than!) its cousin Durable Execution.