Skip to main content

LittleHorse Connect

LittleHorse Connect is a collection of components that allow users to get data into LittleHorse from other systems and to get data from LittleHorse into external systems.

A depiction of LH Connect: Apache Kafka Connect with a LittleHorse Gateway, interfacing between external systems and Apache Kafka and the LittleHorse Server.
LittleHorse Connect Architecture

Architecture and Deployment​

LittleHorse Connect comprises three architectural components to accomplish the task of getting data into and out of LittleHorse:

  1. Apache Kafka Connect.
  2. A Schema Registry compatible with ApiCurio.
  3. The upcoming LittleHorse Gateway (to be released soon) and an OAuth-compatible IdP.

The three components above can be used together to capture events from the outside world, both via polling (Kafka Connect Source Connectors) and listening to webhooks (LittleHorse Gateway), produce them to Kafka with known schemas, and respond to those events inside the LittleHorse Kernel.

Deployment​

LittleHorse Connect (OSS) is flexible and can be deployed a-la-carte: in fact, certain components such as the upcoming LittleHorse Gateway can even be deployed without the LittleHorse Kernel!

To run the dependencies (Apache Kafka Connect, ApiCurio Schema Registry, and an OAuth-compatible IdP), we refer you to relevant open-source projects' documentation:

  1. Apache Kafka Connect
  2. Keycloak
  3. ApiCurio Registry

Use-Cases​

LittleHorse Connect allows multiple use-cases:

  • Workflows which have asynchronous tasks which require waiting for a response from an external system (an ExternalEventNode).
  • Triggering workflows to respond to events in external systems.
  • Triggering workflows in response to events derived from the LittleHorse Output Topic.
  • Exporting data from the LittleHorse Output Topic into external systems (via Apache Kafka Connect).

Components​

LittleHorse Connect extensively leverages the Apache Kafka ecosystem as a platform for event-driven processing. This is a natural fit given that the LittleHorse Kernel itself is a Kafka Streams application and stores all of its state in Kafka.

Kafka Connectors​

LittleHorse Connect provides three Sink Connectors for Apache Kafka:

  1. io.littlehorse.connect.WfRunSinkConnector, which runs a WfRun for each record in Kafka.
  2. io.littlehorse.connect.ExternalEventSinkConnector, which posts an ExternalEvent for each record in Kafka.
  3. io.littlehorse.connect.CorrelatedEventSinkConnector, which posts a CorrelatedEvent for each record in Kafka.

These Kafka Connectors enable your LittleHorse Kernel installation to react to events (technically, records) in Apache Kafka in real time. They are available for download on GitHub.

LittleHorse Gateway​

danger

The LittleHorse Gateway is still under development and is not yet released.

The LittleHorse Gateway is a Quarkus extension which allows you to easily capture REST-based webhooks, authenticate them with OAuth, and convert each webhook request into a record in Apache Kafka with a valid schema (compatible with ApiCurio Registry).

LittleHorse Connect provides the open-source LittleHorse Gateway Quarkus extension and a pre-built docker image with built-in Gateway capabilities.