Tasks and Task Workers

A WfSpec
defines a series of steps in a process to be orchestrated by LittleHorse (in technical terms, each step is a Node
). The most common type of step to execute is a TaskNode
.
Concepts
A task, or unit of work executed by a computer in LittleHorse, is represented by the dual objects TaskDef
and TaskRun
:
- A
TaskDef
is a LittleHorse API Object which defines a certain type of task that can be executed by a computer. You can think of aTaskDef
like a method signature, as it defines the inputs and outputs of a task. - A
TaskRun
is a LittleHorse API Object representing an instance of such a task being executed by a computer as part of aWfRun
.
Tasks are executed by Task Workers, which are software processes written by the users of LittleHorse and represent bite-sized chunks of business logic.
Task Workers
A Task Worker is a program written by a LittleHorse user that connects to the LittleHorse Server, polls for ScheduledTask
s to execute, executes the Tasks, and returns the result to LittleHorse. Task Workers can be easily developed using the Task Worker SDK's in Java, GoLang, and Python according to our documentation here.
When a ThreadRun
arrives at a TaskNode
, the LittleHorse Server puts a ScheduledTask
on a virtual Task Queue. The ThreadRun
blocks at that NodeRun
until a Task Worker pulls the ScheduledTask
off the Task Queue, executes it, and reports the result (or if the configured timeout expires).
Once the TaskRun
is completed, the output from the Task Worker's method invocation can be used by the workflow to mutate variables and control the flow of execution.
Task Workers open a web connection to the LittleHorse Server and do not need to receive connections on any ports. This has several benefits:
- Added security, since your systems do not need to accept any incoming connections.
- Built-in throttling, since the LittleHorse Server only dispatches a
ScheduledTask
to a Task Worker once the worker notifies the LittleHorse Server that it has capacity to perform a task. - Performance, since the protocol is based on grpc bi-directional streaming.
Task Workers can be embedded into existing systems (in Java, for example, all you need to do is add the @LHTaskMethod
annotation), or they can be lightweight standalone wrappers which invoke your existing services via network calls.
Task Logic
The work performed by Task Workers can be incredibly diverse, ranging from charging a customer's credit card to fetching data from an API to deploying infrastructure (as in a devops pipeline). Our Java, Go, and Python SDK's provide utilities that allow you to easily convert a function or method into a Task Worker in five lines of code or less.
In short, a Task Worker can perform any arbitrary function in code. Moreover, a TaskRun can wait on the order of seconds, minutes, or even hours for a Task Worker process to complete, making it adaptable for any type of process that can be executed by a Task Worker (so long as the timeout that you configure on the TaskRun
is long enough).
Deploying a Task Worker
A Task Worker is any program that uses the Task Worker SDK's to execute TaskRun
's. LittleHorse is not opinionated about where or how the Task Worker is deployed: it can be a JVM process running on a bare metal server under a desk; it can be a docker container on ECS, or a Pod in Kubernetes.
Additionally, a single process can run multiple Task Workers for different TaskDef
's at once. This is often useful if you want to take advantage of workflow-driven processes but you have no need for microservices and as such want to avoid managing multiple deployable artifacts.
In real-world scenarios, Task Workers can be deployed in a variety of ways. We have seen successful implementations where a single process manages dozens of task workers for different TaskDef
s, and we have seen other cases wherein a single Task Worker is managed by a dedicated team and deployed on its own process (eg. Kubernetes pod or ECS container).
In Practice
If you want to follow along, all of the code snippets in this section can be copy-n-pasted and executed as standalone files. However, to do this, you'll need access to a LittleHorse Server and lhctl
. You can do that via the instructions in our installation docs or by running the following commands:
brew install littlehorse-enterprises/lh/lhctl
docker run --rm -d --name littlehorse --pull always -p 2023:2023 -p 8080:8080 ghcr.io/littlehorse-enterprises/littlehorse/lh-standalone:latest
To deploy a Task Worker, you have to do two things:
- Register the
TaskDef
. - Create and start the Task Worker, which is the long-lived process that polls the LittleHorse Server for tasks, executes them, and reports the result.
Our SDK's in Java, Go, Python, and DotNet all provide an LHTaskWorker
class/struct/object/utility which makes it easier to do this.
If you change the method signature of your TaskDef
, the rpc PutTaskDef
(called by LHTaskWorker#registerTaskDef()
) will fail with ALREADY_EXISTS
. This is because changing the API of a TaskDef
will break existing WfSpec
s, which is dangerous in production.
If you want to iterate in local development, you can delete the TaskDef
and start again:
lhctl delete taskDef <taskDefName>
Define your Task Method
A Task in LittleHorse is just any other normal function or method. In order to create an LHTaskWorker
, you must pass a pointer to that function or method into the LittleHorse SDK. In this example, we will create a TaskDef
named greet
that takes in a String (representing a "name") and prints it out to stdout
.
In real life, your task method would do something more useful such as query a database, make an API call to an external system, or perform some calculations.
- Java
- Go
- Python
- DotNet
In Java, you can specify a Task Method using the @LHTaskMethod
annotation. That is the only line of LittleHorse-specific code in this file; everything else is Plain Old Java!
package io.littlehorse.quickstart;
import io.littlehorse.sdk.worker.LHTaskMethod;
public class Greeter {
@LHTaskMethod("greet")
public String greeting(String name) {
String result = "Hello there, " + name + "!";
System.out.println(result);
return result;
}
}
TODO
TODO
TODO
Register the TaskDef
and Start Polling
Now that we've written our Task Method, we need to create a Task Worker that connects to the LittleHorse Server and executes our Task Method when appropriate. We can do that with the LHTaskWorker
utility.
- Java
- Go
- Python
- DotNet
package io.littlehorse.quickstart;
import io.littlehorse.sdk.common.config.LHConfig;
import io.littlehorse.sdk.worker.LHTaskMethod;
import io.littlehorse.sdk.worker.LHTaskWorker;
// The `Greeter` class is implicitly imported because it's in the same package.
public class Main {
public static void main(String[] args) {
LHConfig config = new LHConfig();
Greeter greeter = new Greeter();
// Create a Task Worker
LHTaskWorker worker = new LHTaskWorker(greeter, "greet", config);
Runtime.getRuntime().addShutdownHook(new Thread(worker::close));
// Register the TaskDef
worker.registerTaskDef();
// Start the Worker
worker.start();
}
}
TODO
TODO
TODO