Skip to main content

LittleHorse Connect

In this quickstart, we will use the LittleHorse Sink Connectors for Apache Kafka Connect to run a WfSpec. This will enable us to use Apache Kafka's Console Producer CLI tool

Background​

This quickstart makes use of Apache Kafka Connect, Apache Kafka, and the LittleHorse Server.

note

This quickstart builds upon the LittleHorse Kernel quickstart and assumes an understanding of the concepts explored therein.

LittleHorse Connect Concepts​

The quickstart makes use of the following LittleHorse Connect concepts:

  • Kafka Topics, which are ordered logs of records that can be consumed in real time and allow for decoupling event producers from event consumers. Kafka Topics are a core concept of Apache Kafka.
  • WfRunSinkConnector, which is a Sink Connector for Apache Kafka Connect, provided as part of the LittleHorse Platform. The WfRunSinkConnector runs a WfRun for every record in the subscribed Kafka Topic.

Today's Workflow​

We will use the LittleHorse Sink Connectors for Apache Kafka Connect to run the workflow. Recall that the workflow in our quickstart models a "Know-Your-Customer" process. The workflow accepts information about a potential user of a well-regulated enterprise. The workflow will then:

  1. Request a third-party service to perform an identity check on the user.
  2. Wait for the service to respond with an answer. This is asynchronous, so we will use an ExternalEvent.
  3. Either notifies the customer of acceptance or rejection depending on the result of the identity verification step.

In the LH Connect quickstart, we will replace the lhctl run step with the WfRunSinkConnector.

Running the Quickstart​

Let's get this party started! If you haven't done so already, please clone the lh-examples repository:

git clone https://github.com/littlehorse-enterprises/lh-examples.git
cd lh-examples/quickstart
tip

It's highly recommmended that you run through the LH Kernel quickstart first.

Set Up LittleHorse​

important

You might want to remove your lh-standalone container so as to avoid conflicts with ports 2023, 9092, and 8080.

This quickstart uses Docker's --net=host flag, which is sometimes finicky on MacOS.

The Quickstart repo has a convenient ./setup.sh script which:

  1. Starts up the lh-standalone image with Kafka and the LittleHorse Kernel (both the Server and Dashboard).
  2. Downloads and extracts the LittleHorse Sink Connectors for Apache Kafka.
  3. Starts up a Kafka Connect Standalone worker and configures the deployment of WfRunSinkConnector.

If you want to dive deeper into the code, you can do so on github.

Register the Workflow​

We will re-use the same WfSpec from the LittleHorse Kernel quickstart. To refresh your memory, you can speed through the quickstart as follows, for example in Java:

cd ../java
./gradlew run --args register
./gradlew run --args workers

Produce Some Records!​

All that remains to do is kick off our workflow using the Kafka Connectors! As you'll recall, the quickstart WfSpec accepts three input variables:

  • first-name: a STR
  • last-name: a STR
  • ssn: an INT

The WfRunSinkConnector is configured (via wfrun-sink-connector-config.properties) to accept JSON's with each field being a variable passed into the WfRun.

Start up a Kafka Console Producer as follows:

docker run -it --rm --net=host apache/kafka:4.0.0 \
  /opt/kafka/bin/kafka-console-producer.sh \
  --topic new-customers \
  --bootstrap-server localhost:9092 \

And here are some records to produce:

{"first-name":"Obi-Wan", "last-name": "Kenobi", "ssn": 12345}
{"first-name":"Anakin", "last-name": "Skywalker", "ssn": 54321}

The WfRuns should show up in the dashboard now!