The BulkJob system is an experimental feature introduced in LittleHorse 1.2. The API and behavior may change in future releases.
Bulk Jobs
What is a BulkJob?
A BulkJob represents a long-running background operation that acts on many objects in LittleHorse — for example, deleting thousands of WfRuns — without blocking Kafka Streams on large, expensive transactions.
Instead of executing as a single atomic operation, a BulkJob is automatically sharded per Kafka partition. Each partition shard processes its portion of the work independently and in parallel, each with its own sub-status. The overall BulkJob status reflects the aggregate of all shards.
Status
A BulkJob moves through the following states:
| Status | Meaning |
|---|---|
BULK_JOB_RUNNING | The job is actively processing across one or more partition shards. |
BULK_JOB_COMPLETED | All partition shards have finished successfully. |
BULK_JOB_FAILED | One or more partition shards encountered an error. |
The subprocesses field on a BulkJob contains one entry per Kafka partition shard, each reporting its own status.
Currently Supported Operation: BulkDeleteWfRun
The only operation currently supported is BulkDeleteWfRun, which bulk-deletes WfRuns matching specified criteria.
Required fields
| Field | Type | Description |
|---|---|---|
wf_spec_name | string | Name of the WfSpec whose WfRuns should be deleted. |
earliest_start | Timestamp | Earliest startTime (inclusive) of WfRuns to delete. |
latest_start | Timestamp | Latest startTime (inclusive) of WfRuns to delete. |
Optional fields
| Field | Type | Description |
|---|---|---|
wf_run_status | LHStatus | If set, only WfRuns in this status will be deleted. |
How to Use
1. Create a BulkJob
Submit a CreateBulkJobRequest via the CreateBulkJob RPC. The request wraps the operation (e.g. BulkDeleteWfRun) and optionally accepts a client-provided ID for idempotency — submitting the same ID twice returns the existing job rather than creating a duplicate.
// Example CreateBulkJobRequest structure
CreateBulkJobRequest {
id: "my-cleanup-job-2026-08-12" // optional, for idempotency
bulk_delete_wf_run: BulkDeleteWfRun {
wf_spec_name: "my-workflow"
earliest_start: "2026-01-01T00:00:00Z"
latest_start: "2026-06-01T00:00:00Z"
wf_run_status: COMPLETED // optional filter
}
}
2. Poll Job Status
Use the GetBulkJob RPC with the job's ID to retrieve the current BulkJob object. Check the top-level status field, and inspect subprocesses for per-shard detail.
3. Search Jobs by Status
Use the SearchBulkJob RPC to list all BulkJobs, optionally filtered by status (e.g. BULK_JOB_RUNNING). This is useful for monitoring active jobs or finding completed ones to clean up.
4. Clean Up Terminal-State Jobs
Once a BulkJob reaches a terminal state (BULK_JOB_COMPLETED or BULK_JOB_FAILED), you can remove it from the system with the DeleteBulkJob RPC. Attempting to delete a job that is still running will be rejected.
API Reference
The BulkJob-related messages and RPCs are defined in the LittleHorse protobuf schema. See the LittleHorse API Reference for the full generated documentation, including:
BulkJobBulkDeleteWfRunCreateBulkJobRequestSearchBulkJobRequestDeleteBulkJobRequest