![\"A](\"https://littlehorse.io/assets/images/2026-01-16-wfspec-a785811d6d2ac4fbfe2cad2ece96cb00.png\")
In the era of microservices, the word \"monolith\" has often been treated as something to decommission. However, as many engineering teams have discovered, decomissioning long existing services can be a non-starter. In addition, prematurely splitting a monolithic application into many or dozens of distributed services can lead to an operational nightmare. Instead, we suggest the Modular Monolith.
\nBut how do you give a modular monolith the resilience, observability, and workflow modularity of a distributed system? The answer lies in orchestration.
\nA monolith is simply a software application where all logical components- interface, business rules, and data access—are combined into a single deployable.
\nA modular monolith takes this a step further. It is still a single deployable unit, but its internal architecture is strictly divided into distinct, loosely coupled modules (often based on Domain-Driven Design). Each module holds its own logic and data, communicating with other modules through well-defined, in-process interfaces rather than tangled function calls.
\nYou might associate orchestration exclusively with microservices. However, the same principals can be applied to monoliths, often with many of the same benifits.
\nDecomposing a traditional \"big ball of hair\" monolith into a modular one requires discipline:
\nWhen building a modular monolith, borrowing concepts from microservices can be highly effective, provided you avoid the pitfalls.
\nImagine an E-commerce Checkout process. It requires:
\nIn a traditional monolith, this might be a single, synchronous HTTP request. If reserving the inventory or shipping the item, times out, fails or takes days there is no graceful failure condition without a complex system of job queus or external systems.
\nIn an orchestrated modular monolith, these four steps are distinct tasks coordinated by an orchestrator. If the inventory module succeeds but the payment module encounters an error, the orchestrator pauses the workflow and retries the payment module later, or fails another way succesfully. The user gets a responsive experience, and your system remains highly reliable without the operational overhead of managing three separate microservices. In addition external systems can add additional context to this entire process by using External Events
\nAs you start treating your monolith's modules as independent actors, you need an orchestration engine that is fast, reliable, and unobtrusive. This is where LittleHorse.io shines as an ideal bridge.
\nLittleHorse was built to handle high-throughput, mission-critical workflows. In the context of a modular monolith, you can easily treat each internal module as a LittleHorse \"Task Worker.\"
\nBecause LittleHorse is highly performant and language-agnostic, integrating it doesn't add the heavy overhead usually associated with external orchestration. It quietly routes tasks between your monolith's modules, managing state, retries, and error handling seamlessly.
\nMore importantly, LittleHorse makes integrating your monolith into a broader microservice environment effortless. Because LittleHorse doesn't care where a Task Worker lives, a single workflow can seamlessly coordinate a task running inside your dependable monolith alongside a task running in a newly deployed microservice. It provides the perfect, low-friction integration point for hybrid architectures, quietly handling the complex state management behind the scenes. This also allows you to branch of features and functionality, avoiding the complex and lengthy development times associated with monoliths.
\nThe choice isn't just between a tangled monolith and a sprawling microservice architecture. By adopting a modular monolith and orchestrating its workflows, you gain the operational simplicity of a single deployment alongside the resilience, visibility, and scalability of distributed systems. With tools like LittleHorse smoothly bridging the gap, you can incrementally modernize your architecture at your own pace, focusing on business value rather than infrastructure complexity.
", "url": "https://littlehorse.io/blog/orchestrating-modular-monoliths", "title": "Orchestrating Modular Monoliths: Benefits, Strategies, and How LittleHorse Can Help", "summary": "In the era of microservices, the word \"monolith\" has often been treated as something to decommission. However, as many engineering teams have discovered, decomissioning long existing services can be a non-starter. In addition, prematurely splitting a monolithic application into many or dozens of distributed services can lead to an operational nightmare. Instead, we suggest the Modular Monolith.", "date_modified": "2026-04-17T00:00:00.000Z", "author": { "name": "Mitchell Henderson" }, "tags": [ "architecture", "monolith", "Microservices", "orchestration", "littlehorse" ] }, { "id": "https://littlehorse.io/blog/have-your-claw", "content_html": "Among leaders and executives, there has been much recent buzz about OpenClaw in the context of enterprise automation. While Claws do give us a glimpse of what Agents can do, there are fundamental limitations to the Claw architecture when used as process orchestrators. Thankfully, there is a safer, cheaper, and more predictable way to automate processes with Agents. (And no, LittleHorse is not a competitor with Claws...as we will see in this post, LittleHorse focuses on enterprise process orchestration while Claws fit best as powerful personal assistants).
\nIn this post, we will:
While I often write specifically for engineers, this post focuses more on business-level challenges and should be relevant to both engineers and executives alike.
To understand how the Claw architecture works at a high level, we will take a scenic (but still pleasant and relaxing!) route:
\nIf you've been hands-on with agents for a while and understand the internals of a Claw, you can skip ahead. The following section assumes no technical background.
I'll write a technical deep dive on gradient descent, LLM's, and agent internals once I finish the Agentic Patterns series.
At the risk of oversimplifying one of the most impactful inventions of the 21st century, an agent is simply a computer program that calls an LLM repeatedly in a fancy loop.
\nThe program (sometimes called an \"Agent Harness\") first defines a set of tools that the LLM can use. For example, tools might allow the LLM to perform such actions as \"search the web,\" \"run a bash (terminal) command,\" or \"read the contents of this file on my laptop.\" Once the tools are defined, the program then:
\n1 through 3 until the LLM says \"I'm done!\"The key insight is that step 2 gives the LLM access to interact with the outside world through tools. This is the source of many security vulnerabilities: the most infamous one is \"prompt injection,\" in which a malicious prompt tricks the LLM into calling a tool with parameters that cause a data leak or which cause inappropriate actions to take place.
The Toolformer Paper from 2023 is a good read to deep dive into exactly what's going on in step 2!
A Claw is a thin (but extremely powerful) layer on top of an Agent that, in most cases, runs locally on your very own laptop. The Claw architecture utilizes three key insights (or are they tricks?):
\nLastly, a tiny amount of plumbing code allows users to prompt their agent through chat interfaces like WhatsApp, turning a Claw into a full-blown digital assistant. In fact, the Claw architecture is so elegant and simple that it can be implemented in just 500 lines of code (see: The NanoClaw Project) on top of a coding agent such as OpenCode.
\nAs we'll see shortly, the danger with the Claw architecture is how free-wheeling it is. Processes aren't coded anywhere: the behavior of the Claw is entirely up to the probabilistic (which means definitely eventually wrong) LLM and the non-binding instructions (or pleas?) that you give the Claw.
\nNVidia recently released NemoClaw, which builds security on top of OpenClaw. NemoClaw is NVidia's enterprise stack for running OpenClaw, allowing you to install and run a Claw in an isolated sandbox (you can think of this as a virtual machine). NemoClaw also has the handy ability to run on cloud hardware, solving the problem wherein closing your laptop stops your agent from making progress.
\nWhile NemoClaw does allow you to mitigate a lot of the security implications of running Claws in the enterprise, the project still does not turn the agent into a reliable, auditable, and efficient orchestrator. As we'll see in the next section, NemoClaw is a useful assistant but is not a panacea for enterprise automation.
\nIn short, NemoClaw removes some of the security considerations that OpenClaw has as a personal assistant but does not transform the Claw architecture into an enterprise process orchestrator.
\nThe Claw architecture is effective and elegantly simple: coding agent harnesses such as Claude Code and OpenCode can use a computer in ways similar to a human, enabling the Claw to accept instructions in natural language.
\nThat simplicity comes at a cost in governance, predictability, and token spend. For mission-critical business flows, we want the agent to participate in the process, not invent the process.
\nOpenClaw (the most famous Claw!) cut its teeth with personal assistant tasks such as:
\nFor such tasks, the Claw architecture is highly effective and elegantly simple: coding agent harnesses such as Claude Code and OpenCode can perform these tasks simply by using bash commands and the Browser (through MCP). Handy interfaces (such as WhatsApp or Signal chats) make interacting with the Claw truly feel like a live personal assistant.
\nThe Claw-style loop which wakes an agent on a schedule and lets it do something on a computer is great for building an assistant that can solve open-ended personal automation tasks. However, this architecture is definitively unfit for enterprise process orchestration for critical processes such as:
\nFirst, a system built on top of natural language instructions parsed by a non-deterministic computer program will never hold up under real-world edge cases such as prompt injection attacks, much less a regulatory audit. Details such as authorization, specific business logic, and when to involve human judgment should be codified rather than left to a probabilistic LLM.
\nSecondly, the Claw architecture is extraordinarily expensive. Token costs can easily exceed hundreds of dollars per day per person, wiping away any efficiency gains. This is because lack of structure forces the agent to spend tokens figuring out what needs to be done, injecting creativity in places where it is potentially unwanted (and weighing down the wallet).
\nLastly, in addition to the challenges of using a Claw to automate back-office processes, OpenClaw's own security docs caution against deploying a Claw in a multi-tenant environment. Rather, users are advised to deploy a single Claw per human user of them in a true assistant-style deployment.
\nClaws spend tokens to figure out what to do. While this is great for open-ended, assistant-style tasks, this architecture is not suitable for business-critical workflow automation. In such cases with well-defined, sensitive processes, Claws are both expensive and error-prone due to the fact that processes are not codified but rather inferred in real-time by the LLM.
\nIf Claws aren't the answer, how can we take advantage of the transformational power of Agents? Simply put, codify the parts of your process that you know must happen and use agents to route individual cases within the larger process. That's what I've been writing about in the Agentic Patterns series, and that is exactly what Business-as-Code excels at.
\nFor business process automation, Business-as-Code gives you a safer and more predictable way to deploy Agents without blowing your budget.
\nThe core premise of Business-as-Code is that you (or Claude) write high-level code which specifies what is supposed to happen (and when) in your business process. These actions take place across myriad systems: microservices, external SaaS platforms, agents, and legacy systems such as mainframes.
\nCrucially, as much as possible of the process is codified in deterministic code: Agents are only responsible for making routing decisions within a larger process, such as classifying a refund request as valid or fraudulent in the context of a user's past interactions. The Agents do not have the capability to reinvent the order return process: that was already codified, reviewed, and accepeted in deterministic Java / Python / Go / C# code which matches your business process and is visualized in the LittleHorse Dashboard (as below):
\n
When you use a Claw to automate enterprise processes, you'll repeatedly pay for tokens used to figure out steps that should have been deterministically codified to save costs and reduce errors. Business-as-Code lets you have your Agent and eat it too: you can reap the benefits of agentic intelligence, save money, and build processes that never break down.
\nIn short, if you haven't codified your process with Business-as-Code, the Agent will invent one for you.
", "url": "https://littlehorse.io/blog/have-your-claw", "title": "Executive POV: Have Your Claw and Eat It Too?", "summary": "Among leaders and executives, there has been much recent buzz about OpenClaw in the context of enterprise automation. While Claws do give us a glimpse of what Agents can do, there are fundamental limitations to the Claw architecture when used as process orchestrators. Thankfully, there is a safer, cheaper, and more predictable way to automate processes with Agents. (And no, LittleHorse is not a competitor with Claws...as we will see in this post, LittleHorse focuses on enterprise process orchestration while Claws fit best as powerful personal assistants).", "date_modified": "2026-04-07T00:00:00.000Z", "author": { "name": "Colt McNealy" }, "tags": [ "AI, LLM's, and Agents", "analysis" ] }, { "id": "https://littlehorse.io/blog/decision-workers", "content_html": "How can we mitigate issues like prompt injection, governance, and plain old integration challenges when using agents for automation? We've helped our customers overcome these problems with the Decision Workers pattern, which separates the concerns of intelligence and orchestration. In this pattern, LittleHorse WfSpecs achieve reliable and secure orchestration while Decision Workers (Agents with read-only tool access) provide the intelligence.
This is the second part in a four-part blog series about building agentic workflows with LittleHorse. This blog series will help you automate internal and customer-facing business processes with guardrails and security.
![\"Example](\"https://littlehorse.io/assets/images/decision-worker-wfspec-4754e9ee5b18636f281d143eb9a9c41e.svg\")
In this pattern, a LittleHorse workflow (WfSpec) consults an agent to extract important information and make a business decision about which path to take in the process. The agent does not orchestrate the process itself but rather serves as a router which decides what path to take. Along the way, deterministic guardrails enforce security and catch erroneous decisions that agents make, and the orchestrator (here, a WfSpec) shepherds the process to completion while collecting an audit trail of everything that happened.
The key to making the agent inside the Decision Worker effective yet safe is to give it tools that have read-only access to sensitive systems. The agent is free to fetch as much context as it needs in order to make a decision; however, it does not directly take action itself but rather outputs a structured decision which is then acted upon by the WfSpec.
Let's analyze two architectures for a system that automatically triages incoming support requests over email. We'll look at the system with two architectures:
\nOur system will use an agent to read an email, fetch context using tools (eg. find info about the order, find info about the user, etc), and then decide how to handle the ticket:
\nA naive approach would be to give an agent multiple tools: some tools to fetch context (such as looking up information about the order and the user), and some tools to take action by hitting other API's. It would be dangerous to give the agent a tool such as:
\nprocessRefundAndCancelOrder(String orderId, BigDecimal refundAmount);
The services and API's involved in processing refunds are decidedly not multi-tenant, so this approach requires the agent to correctly propagate tenancy context. This causes the agent to become vulnerable to prompt injection attacks: a malicious email could manipulate the model into issuing an unauthorized action (such as canceling someone else's order).
\nOur problem is that the agent's decision space is not properly constrained: we're giving the agent the ability to decide to send a refund to the wrong person. In a multi-tenant system, that means the model must correctly preserve tenant context on every tool call. A malicious email can exploit that requirement through prompt injection, causing the system to cancel the wrong order or issue the wrong refund.
\nAgents perform far more reliably when you give them read access to tools, and then parse and act upon a structured output decision. But how can we effectively orchestrate this?
\nAs mentioned earlier, a Decision Worker is a pattern in which you wrap an agent inside a LittleHorse Task Worker. The agent has read-only tool access to sensitive systems, and outputs a decision which is acted upon by the WfSpec. As we'll see later, this pattern vastly mitigates the exposure to prompt injection attacks (or even simple hallucinations).
Here's a LittleHorse Task Worker which implements that:
\npublic enum SupportTicketResponse {
ESCALATE_TO_TEAM,
SEND_ORDER_INFO,
CANCEL_ORDER
}
@LHTaskMethod(\"triage-support-ticket\")
public SupportTicketResponse triageTicket(String emailContent) {
// Not shown here for simplicity: use the LLM in a loop to gather more context
// like a proper agent does.
//
// The agent is ONLY allowed to READ, not write.
String response = anthropic.messages().create(MessageCreateParams.builder()
.model(\"claude-sonnet-4-20250514\")
.maxTokens(50)
.system(\"You are a support ticket classifier. Respond with exactly one of: \" +
\"ESCALATE_TO_TEAM, SEND_ORDER_INFO, CANCEL_ORDER. \" +
\"No other text.\")
.addUserMessage(emailContent)
.build()
).content().get(0).text().trim();
try {
return SupportTicketResponse.valueOf(response);
} catch(Exception exn) {
return SupportTicketResponse.ESCALATE_TO_TEAM;
}
}
Agents can take their time. Best to leave at least 60 seconds task timeout.
Below is a WfSpec which places guardrails around the above Decision Worker, handling all orchestration and enforcing proper security contexts.
@LHWorkflow(\"handle-support-ticket\")
public void supportTicketFlow(WorkflowThread wf) {
WfRunVariable emailBody = wf.declareStr(\"email-body\").required();
WfRunVariable status = wf.declareStr(\"status\").searchable();
WfRunVariable userId = wf.declareStr(\"user-id\").searchable();
WfRunVariable orderId = wf.declareStr(\"order-id\").searchable();
status.assign(\"TRIAGING\");
// Fail and escalate to humans if the agent cannot extract a valid order-id which
// belongs to the user
orderId.assign(wf.execute(\"extract-order-id\", emailBody, userId));
wf.doIf(wf.execute(\"validate-order-and-user\", userId, orderId).not(), handler -> {
handler.assignUserTask(\"review-support-ticket\", null, \"support-team\").withNotes(\"couldn't extract valid orderId\");
});
// Decision Agent here!!!
WfRunVariable decision = wf.declareStr(\"decision\");
decision.assign(wf.execute(\"triage-support-ticket\", emailBody));
wf.doIf(decision.isEqualTo(\"CANCEL_ORDER\"), then -> {
status.assign(\"REFUNDING\");
// Avoid prompt injection: we know the orderId is tied to the user. This is
// controlled by the workflow, not the agent.
then.execute(\"process-refund\", userId, orderId);
then.execute(\"cancel-order\", userId, orderId);
then.execute(\"send-email\", userId, \"Refund Issued\", \"We've issued a refund for your recent order.\");
}).doElseIf(decision.isEqualTo(\"SEND_ORDER_INFO\"), then -> {
status.assign(\"RESPONDING\");
then.execute(\"send-order-info\", orderId, userId);
}).doElseIf(decision.isEqualTo(\"ESCALATE_TO_TEAM\"), then -> {
status.assign(\"ESCALATED\");
then.assignUserTask(\"review-support-ticket\", null, \"support-team\").withNotes(orderId);
});
status.assign(\"DONE\");
}
A WfSpec is the perfect way to capture the power of an agent while reducing the amount of work it must do. It encodes processes which don't change (such as payments, onboarding, orders, etc) while freeing up agents to make decisions. Most importantly, using WfSpecs to harness your agents allows you to deterministically control what your agent does and doesn't touch.
With the Decision Worker pattern, your agent only has to be able to find information rather than take (potentially destructive) actions. For example, our email triage worker might be allowed to look up any information it wants about the user, past orders, etc, but it's not able to (nor does it need to) call any privileged API's with write access.
\nCrucially, the Decision Worker pattern makes it much harder to trick agents into spilling secrets. This is because the agent itself does not have the tools needed to do so: we can write non-agentic (deterministic) code to explicitly validate that the order we are acting on belongs to the user who sent the initial email. Even with malicious emails, the WfSpec deterministically enforces security constraints and takes away the agent's ability to make errors (for example, canceling the wrong user's order).
If you simply pass over the orchestration to the agent itself, you get no such defense.
\nLastly, all of the traditional benefits of LittleHorse still apply here. You can see the execution history of all support tickets. LittleHorse still provides all of the durability and orchestration benefits between steps that it always does:
\nMost importantly, you can clearly see what decisions were made by which agent, and when.
\nDecision Workers are a pragmatic way to put agents into production: let the agent do what it is good at (interpreting messy human input and choosing intent), while the WfSpec does what deterministic software is good at (validation, side effects, retries, and escalation).
In our ticket-triage example, this separation gives you a safer architecture. The agent never gets direct authority to perform privileged write actions. Instead, the WfSpec validates identity and order ownership, then executes the approved path with a full audit trail.
Stay tuned for the next post about using agents to verify other agents!
", "url": "https://littlehorse.io/blog/decision-workers", "title": "Agentic Patterns II: Decision Workers", "summary": "How the Decision Workers pattern separates AI judgment from deterministic workflow orchestration.", "date_modified": "2026-03-07T00:00:00.000Z", "author": { "name": "Colt McNealy" }, "tags": [ "Agentic Patterns", "AI, LLM's, and Agents", "Orchestration" ] }, { "id": "https://littlehorse.io/blog/agents-need-workflow", "content_html": "Just how much decision power should you give your agents in a business process? On one extreme, AI hype-artists claim that agents should stitch together all API's and \"figure out your business process for you\" without any orchestrator. On the other extreme, detractors claim that agents should not be allowed to make decisions without human oversight. In reality, by giving agents proper guardrails you can maximize user experience intelligence while minimizing mistakes (and even save tokens while you're at it!).
\nThis is the first part in a four-part blog series about building agentic workflows with LittleHorse. This blog series will help you automate internal and customer-facing business processes with guardrails and security.
Last week's blog focused on the software development lifecycle: LittleHorse provides a governance harness for AI-written microservices to allow engineering teams to tackle larger problems with more success. In contrast, this new blog series takes a different angle: how should you architect end-user systems that involve agents at runtime (not development timme)? How can we build systems that maximize intelligence while reducing surface area for mistakes made by agents and minimizing token costs?
\nWhile good and useful software can be produced by LLM's, I don't believe in wasting people's time by asking them to read AI slop. This blog was written entirely without the aid of LLM's.
Software systems exist to serve people, free us from repetitive toil, and minimize human error. I do not enjoy writing simple CRUD endpoints on a REST API, nor do I enjoy pulling pieces of info out of an unstructured email and copying them into our CRM. Before the invention of LLM's, there was little hope of building systems which could automate these processes, but now we can!
\nWe've built quite a few agentic workflows at LittleHorse, and today I wanted to share some thoughts about how we have maximized the intelligence of these systems while minimizing error.
\nI'm a big believer that automation will result in more jobs long term. Typewriters and automated elevators put an end to the jobs of scribes and elevator operators, but today we have far fewer unemployed people than we did before those two inventions.
The same will happen for copy-n-paste monkeys.
This blog series will start by examining differences between agents and classical software, then we'll analyze which use-cases are best for which pattern. Finally, we'll take a look at how to marry agents together with classical software to safely handle the most complex use-cases.
\nAgents are probabilistic, whereas classical software is deterministic. At the core of an agentic application is a Large Language Model, which is simply an autoregressive neural network trained to predict the next token (roughly, token == word).
\nWhen agents do things, there is no guarantee that they will get the right answer. It's the same as asking a human to do something: sometimes they do the right thing, sometimes they make mistakes.
\nIn contrast, the software systems we've relied on for the past 40 years always produce the same results when run with the same input. When they make a mistake, the mistakes are predictable and can be fixed (we call them \"bugs\" 😉).
\nImperative classical software produces correct answers for predictable inputs. However, classical software systems struggle to handle dynamic inputs. For example, it's really hard to write a (non-AI) program which reads an email and decides whether to issue a refund, re-send a package, escalate to customer support, or simply respond with order information.
\nHowever, that is now possible today with agentic applications. LLM's excel at extracting structured data from natural prose and making decisions based on that data.
\nDue to their probabilistic flexibility, agents are not suitable to orchestrate business processes where you already know what steps must happen.
\nThe process for transferring money from one account to another at a bank is well-known and unchanging. Sure, an agent could probably figure out calling the tools in the right order 99.9% of the time, but the 0.1% of the time when it forgets to check for balances or to handle import taxes are devastating.
\nAdditionally, agents are typically just long-running software programs which are exceptionally susceptible to crashes and failures (due to their long-running nature and the amount of network calls they make). If the agent crashes during the middle of an important process, you're left with the question of how to identify and recover stuck processes.
\nThis multi-step orchestration where you absolutely cannot tolerate failure is what classic software is for. As I've written before, it's a very difficult problem but is much simpler when you use an orchestrator like LittleHorse. In fact, whether you write the logic yourself or use agents to write your workflows and verify the result, LittleHorse makes it easier to achieve and verify correctness.
\nAs an added benefit, it is far cheaper to decide your critical business process once than to pay Anthropic or OpenAI for tokens to make the same decision over and over again. Trust me, those bills can add up real quick.
\nAgents and workflows are not substitutes but rather complements: LittleHorse provides a harness to integrate and govern agents safely. Many of our customers and open-source users have used LittleHorse with agents in creative ways; I plan to write about three of the most prominent ones in this blog series:
\nWfSpec to take, reserving decisions for AI and actions for the workflow orchestrator.In all three cases, Agents provide intelligence while LittleHorse shepherds processes to completion across multiple steps. The risk of \"rogue agents\" is mitigated because the agent itself doesn't need to be able to take much action: the agent simply decides what action to take, and LittleHorse can then take the action with rigid guardrails. Finally, these patterns allow our users to limit the size of the task given to the agents to be as small as possible.
\nThis is an exciting time for builders everywhere. At small startups and large enterprises alike, agents have given builders the tools to builid much more powerful apps than ever before.
\nLittleHorse already powers many agentic applications by providing a harness for these distributed processes. However, I'm even more excited about our upcoming Agent Builder products in our Saddle Command Center which will make it even safer and easier to build agents. Stay tuned!
\nAgents are not substitues for workflow 🤷♂️. They're complements 🔥. It's a paradigm shift, you're absolutely right! 🚀 💪
As I wrote last month, LittleHorse helps ensure alignment between your business goals and your software, acting as a Command Center for Business-as-Code. AI Agents are starting to unlock the production of vastly greater amounts of code than ever before, making this alignment and governance even more crucial.
\n![\"AI-Generated](\"https://littlehorse.io/assets/images/ai-writing-code-6209fc6df9ec5023543338d0ca47a0f0.png\")
While good and useful software can be produced by LLM's, I don't believe in wasting people's time by asking them to read AI slop. This blog was written entirely without the aid of LLM's.
In December 2021, I set out with my father to build the best harness for microservices that the world had ever seen. Agentic AI, and especially vibe-coding, was a monkey wrench that none of us could have predicted in 2021. The question every architect must face now is, \"how will my architecture change because of agentic development?\" It turns out that we're barreling straight for a world that will be saturated with microservices in need of a Command Center.
\nAfter much study (at the terminal, in the field, reading papers), I believe that:
\nA microservice is a small piece of software that is developed and deployed in-house to provide a small set of functionality. Microservices allow different pieces of software to be deployed independently, but they come at the cost of distributed complexity.
In economics, Jevon's Paradox posits that, when a market becomes able to use a resource more efficiently, the market ends up using more of that resource. This is because greater efficiency makes certain transactions, which were previously unprofitable, profitable.
\nIn the context of agentic coding and software, we will see vastly more code being produced by even more engineers. Commit data from GitHub corroborates this increase in total software written, and we've even seen an uptick in software job postings over the last three months since Opus 4.5 (and now 4.6) was released. (For you Codex fans out there, I'm not trying to argue which model is better. Just don't use GPT 3.5 please.)
\nThe first consequence of the cost of writing code* reaching zero is summarized by Jevon's Paradox: many things which were too costly to automate are now in-play. On the \"hobbyist\" side of things, I've seen business analysts vibe-coding data portals and dashboards, non-technical founders take over the maintenance of their website, and a litany of vibe-coded personal productivity apps (just open Twitter and you'll see dozens).
\nHowever, the productivity gains from agentic coding are not limited to just vibe-coded toys. Many enterprise projects which were previously too ambitious are now in play. In the field, we've seen:
\nWithout microservices and AI, none of these wins would have been possible. This trend will only accelerate.
\nWriting code and building digital solutions to business problems are two distinct problems.
![\"LittleHorse](\"https://littlehorse.io/assets/images/lh-at-the-center-4f03d717b4102bd3e44b2ca7a967762f.svg\")
In a world where Claude writes vast quantities of code at a rate far faster than we can review it, how will we avoid the so-called \"AI Slopocalypse?\" We will create harnesses to give our enthusiastic robotic friends the best chances for success:
\nEven though LLM context windows continue to grow, agents—just like the smartest humans—have a higher chance of success when given a smaller problem to work on. The chances of success skyrocket when you give the agent the ability to verify that it arrived at the correct answer in a loop. I've noticed that our good friend Claude can be quite persistent when you give him Opus 4.6 and a way to check his answer.
\nAlso, the speed at which agents can deliver software scales linearly with the number of tasks that can be worked on in parallel. By using microservices, a single large problem (\"write a custom ERP system for me\") can be broken into manageable and parallelizable tasks, enabling great speedup.
\nLastly, breaking problems down into smaller chunks such as microservices, or better yet Task Workers in LittleHorse, makes it easier to define verification steps for each task given to agents.
\nAs I wrote two years ago, microservices can quickly devolve into an unsupervised and chaotic mess. While it's easier to reason about a single microservice than about an entire monolith, teams are still left with the struggle of piecing the microservices together into a cohesive system.
\nLittleHorse solves this problem by providing a Command Center to compose, connect, and govern your microservice flows. It gives you:
\nFor more info about LittleHorse and microservices, check out my Integration Patterns series.
We'll soon see agents producing code at a rate that far surpasses human review capacity. When this happens, how will we understand our system? More importantly, how will we pass security audits and prevent disastrous business process bugs? Some say \"just ask Claude to write documentation.\" However, docs can drift, and LLM's can still make mistakes.
\nUsing a microservice harness like LittleHorse provides a really elegant solution: the dashboard allows you to deterministically visualize the end-to-end flow of your system simply by projecting the code written by your agents.
\nEven better, every interaction is logged, traced, and indexed in LittleHorse so that you can easily chase down bugs and security incidents, and allow your system to self-heal. Claude is already quite capable at debugging LittleHorse workflows using lhctl and this capability will only get better.
While it might be hard to understand a production-ready order-processing microservices architecture, it's simple to reason about your vibe-coded business logic when it's displayed on the LittleHorse dashhboard as follows:
\n
2026 is quite an amazing time to be alive. I would be lying if I told you that I had vibe-coding and AI Agents in mind when I founded LittleHorse in 2021; it just so happens that a microservice harness is even more crucial in 2026 than it was four years ago.
\nThis post has been very forward-looking, but as we near the release of the Saddle Console and version 1.0 of the LittleHorse Server, I'd like to take a minute to look back. Working on LittleHorse has been a blessing in so many ways, but nothing has been better than getting to know my team members, advisors, and other folks I've met along the way.
I'm deeply proud of my team. They've taught me so much about life, leadership, software, project management, business, and so many other things, and many of them have become my close friends. As we've had our first few big customer success stories with LittleHorse, I'm filled with pride for what we've built, and I'm excited to see what we will do this upcoming year.
\nAs with every seismic shift, there will be winners and losers. The winners of the AI era will be people who can think at a high level of abstraction with agency (pun very much intended). Unfortunately, though, folks who made their career off of solving small, well-written JIRA tickets will need to adapt to working at a much higher level of abstraction. They'll need to begin engaging business stakeholders directly, they'll need to understand the business value of what they're building, and they'll need to become accustomed to managing much larger systems.
\nIn short, I think the era of making 6 figures as a \"CRUD Monkey\" is over.
\nFor those of us who can adapt to the new world—by thinking backwards from business problems while still understanding fundamental architecture, by letting go of the need to micromanage syntax, by inventing better solutions rather than copy-n-pasting from StackOverflow—our window of opportunity is just opening. We now have an army of little personal assistants that can do the un-differientiated stuff for us. Now, software engineers can finally be true \"engineers.\" We are blessed to live in these times.
\nThree papers in particular have been quite formative to writing this essay:
\nThe Infrastructure Software world was revolutionized by Infrastructure-as-Code, which provides observability, governance, and agility to infra teams. Analogous problems in application development are soon to be solved by a new paradigm: Business-as-Code.
\nThis is the first part in a three-part series about what makes LittleHorse unique: we are the only platform that allows you write business-as-code for both fast and slow processes, unifying workflows and events.
Business-as-Code allows platform teams to enjoy:
\nWhile Infrastructure-as-Code has successfully brought governance, predictability, and agility to infrastructure and security, the development of business processes running on distributed applications remains unsolved and is ripe for a new paradigm: Business-as-Code.
\nJust as infrastructure teams once struggled with manual, error-prone deployments and inconsistent environments before IaC, application development teams today face analogous challenges when composing business processes across distributed systems. The same chaos that IaC solved for infrastructure—scattered configurations, lack of visibility, and brittle manual processes—now plagues how we build and maintain business logic in microservice architectures.
\nAs I previously wrote, software engineers at the modern enterprise must build applications to support business processes in a highly distributed technical environment. This makes it difficult to deliver stable, working applications and causes significant technical complexity, resulting in a divide between the code engineering teams write versus what the business needs.
\nIn order to make distributed processes reliable, engineers need to spend time on non-differentiated yet frustrating tasks such as:
\nThese integration challenges burn through IT budget; first, by forcing CTO's to buy expensive observability and APM platforms, and secondly by forcing engineers to spend time on reliability rather than new features. This unfortunate reality often stands in the way of true digital transformation.
\nBeyond the technical challenges posed by building applications in distributed environments—and perhaps because of the distributed nature—, fragmented software architecture leads to a gradual disconnect between what your business needs and what your software does.
\nIn a classic microservice architecture, different teams are in charge of certain microservice bounded contexts, such as user management, order processing, inventory, or loyalty. Many organizations that we have worked with struggle to understand the interactions between these services. For example, one customer says:
\n\n\nWe have many different domains, and each team puts out an API that is called by other teams, but we have no way to understand the dependencies between all of those API's. And no one in the company understands our business processes end-to-end.
\n
Besides simply technical challenges, this fragmentation increases the likelihood that software is developed which doesn't actually drive business value because of a lack of clear understanding between product and engineering. This \"spaghetti architecture\" also prevents opportunities for code reuse and standardization.
\nJust as Infrastructure-as-Code tools such as Terraform de-fragment infrastructure and make systems more reliable, Business-as-Code delivered through the LittleHorse Kernel allows engineers to have a single source of truth for how their microservices interact.
\nThe central concept of the LittleHorse Kernel is a Workflow Specification, known in code as a WfSpec. A WfSpec is a computational graph defining which computational steps are to be executed, and when.
Here's what a complete order processing workflow looks like in Business-as-Code. This single WfSpec handles the entire business process, from price calculation to credit checks to shipping and it's all in one place, readable by any Java developer. The following code snippet is from one of our examples, showing the definition of a WfSpec that runs the process of handling an order:
@LHWorkflow(value = \"process-order\", defaultTaskRetries = \"3\")
public void wfLogic(WorkflowThread wf) {
// Declare a bunch of variables
email = wf.declareStr(\"email\").asPublic().required().searchable();
item = wf.declareStr(\"item\").asPublic().required();
orderPrice = wf.declareDouble(\"order-price\").asPublic();
status = wf.declareStr(\"order-status\").searchable();
creditScore = wf.declareDouble(\"credit-score\");
status.assign(\"PROCESSING\");
// Calculate Price
orderPrice.assign(wf.execute(\"fetch-price\", item));
// If too big, bring in human approval
wf.doIf(orderPrice.isGreaterThan(10_000.0), this::creditCheck);
status.assign(\"SHIPPING\");
wf.execute(\"ship-item\", email, item);
wf.waitForEvent(\"item-arrived\").registeredAs(Boolean.class);
status.assign(\"DONE\");
wf.execute(\"send-email\", email, \"Your Order\", \"Your order was completed\");
}
// You can use lambdas or you can use function pointers to fill out if bodies.
private void creditCheck(WorkflowThread wf) {
creditScore.assign(wf.execute(\"fetch-credit-score\", email));
wf.doIf(creditScore.isLessThan(500.0), ifBody -> {
UserTaskOutput result = ifBody.assignUserTask(\"verify-credit-check\", null, \"billing-admins\");
ifBody.doIf(ifBody.condition(false, Comparator.EQUALS, result.jsonPath(\"$.isApproved\")), subIfBody -> {
status.assign(\"REJECTED\");
subIfBody.execute(\"send-email\", email, \"Your Order\", \"Your order was rejected due to insufficient credit\");
subIfBody.fail(\"insufficient-credit\", \"User does not have sufficient credit\");
});
});
}
This is the entire business process. No hunting through Slack channels to figure out which service calls which. No wondering if the retry logic is in the queue config or the service code. It's all right here, and it compiles into a visual graph that product managers can actually understand.
\nWfSpec is CodeFirst, you'll notice that the WfSpec is written in Java (as of this publishing, we also have clients in Python, Go, and C# and are evaluating demand for more languages). Although the code uses a DSL provided by the WorkflowThread object, it is compiled and run like any other Java application, and the DSL is readable by any Java programmer.
Secondly, if you look at the DSL you'll see all of the features of a normal programming language. Variables, conditional branching, function calls (TaskRun execution via wf.execute), and more. This DSL code gets compiled down to a metadata object (a WfSpec protobuf, which you can see via lhctl get wfSpec <name>). The actual WfSpec is a computational graph containing Nodes and Edges between the Nodes. Each Node is a computational step to be scheduled by the LittleHorse Server; the Edges determine the control flow. This is much like the abstract syntax tree in normal programming languages.
WfSpec is BusinessAlthough the WfSpec is code (which comes with a great many set of benefits, such as testing, version control, etc), it is a much higher layer of abstraction than vanilla Java code. The WfSpec shown above spans the entire business process of handling orders. It deals with many different Bounded Contexts and microservices:
ship-item task).send-email task).fetch-credit-score task).fetch-price task).verify-credit-check UserTaskRun).The WfSpec code compiles into a business process, which can be visualized on the LittleHorse Dashboard:
The open-source LittleHorse Kernel gives engineering teams the ability to write end-to-end business processes in code, while abstracting away the non-differentiated tasks such as error handling, retries, and transactional consistency. This yields several benefits to organizations:
\nI am a developer at heart, which is why I wrote the LittleHorse Kernel and open-sourced it. We want software engineers to be able to use familiar and comfortable primitives to operate at a higher level of business value.
\nTo get started:
\nAs the sophistication of AI systems grows, so has the ability to automate tasks. However, scaling multi-agent systems has been a major bottleneck for many engineers. Frameworks like LangGraph seek to solve this problem, but the framework is not very scalable and the api's don't always fit the needs of developers, can become outdated, and may introduce breaking changes...hence the need for custom agents, and modularity. A2A is an effort by Google to standardize inter-agent communication and allow developers to integrate the capabilities of various frameworks.
\nThe probabalistic origins of machine learning and AI models cause a significant amount of non-determinism in the outputs, on the contrary to traditional algorithms that provide a deterministic output (i.e. given the same input, it always produces the same output). Because of this, we can't always predict the output of a machine learning algorithm. So maybe it is obvious that in order to allow these systems to share information, the outputs must be wrapped in some metadata to provide some scaffolding for a message format, even though the messages themselves are always unique.
\nA2A is a protocol to allow AI agents to communicate autonomously amongst themselves in a standardized way. It was introduced by Google in April 2025 and donated to the Linux Foundation for more widespread adoption.
\nExample Query: What is an AI agent?
\nAnswer 1:
\nAn AI agent is a software system that can perceive its environment, make decisions, and take actions to achieve specific goals with some degree of autonomy. Unlike traditional software that simply follows pre-programmed instructions, AI agents can adapt their behavior based on what they observe and learn.
\nAnswer 2:
\nAn AI agent is a software system that can perceive its environment, make decisions, and take actions to achieve specific goals with some degree of autonomy. Think of it as a digital entity that can operate independently rather than just responding to direct commands.
\nWe can see from this example that for two identical queries, the output can vary but still be useful for describing an AI agent. As the input gets more complex, the more these outputs can diverge.
\nNote: See also MCP, a protocol introduced by Anthropic to provide a standard of communication between AI models and tools, data, prompts, and other resources. While MCP allows the AI \"agent\" external functionality like querying a database, searching the internet, or executing a workflow, A2A allows multiple specialized AI agents to communicate with each other and complete complex tasks by utilizing their variety of skills.
\nSo what is an Agent? While the A2A protocol seems like it has been standardized, the definition of \"agent\" certainly has not. Langchain and Microsoft define an agent as \"an LLM + tools + a prompt\". Google has more broadly stated that an AI agent is a software system that uses AI (not just LLMs) to proactively pursue goals and complete tasks. This allows us to expand the scope of agents to include more specialized machine learning algorithms, although the implementation of Google's agent development kit (ADK) indicates that they really fall into the first camp. For now, I will focus on the \"LLM + tools + prompt\" definition, but stay tuned for more exploration into nuanced algorithms in the future.
\nLet's take a look at our first \"agent\"...
\nPrerequisites:
\nimport os
from dotenv import load_dotenv
load_dotenv()
If you would like to follow along, and don't want to copy and paste code, you can clone this github repo and interact with the jupyter notebook version of this blog!
\nfrom google.adk.agents import Agent
from google.adk.tools import google_search
my_first_agent = Agent(
model=\"gemini-2.5-flash\",
name=\"My_first_agent\",
description=\"A simple agent that can call a google search\",
instruction=\"You are a helpful google search agent. Conduct a search when you determine it is necessary to do so.\",
tools=[google_search]
)
As you can see, there is not much effort to declare your first agent. Google's ADK abstracts away a lot of the headache, it's details are out of scope for this work, but will be covered in a future blog post. Each of the supplied parameters are self-descriptive:
\nNow we have a primitive agent. So how do we handle messages?
\nThe Agent Executor interface is contained in the a2a library, and provides a wrapper for the Agent to handle requests. The interface contains two main methods: execute(), which handles the main execution logic of the Agent's runtime and cancel(), which can be called during the execution of a long-running task.
class AgentExecutor(ABC):
\"\"\"Agent Executor interface.
Implementations of this interface contain the core logic of the agent,
executing tasks based on requests and publishing updates to an event queue.
\"\"\"
@abstractmethod
async def execute(
self, context: RequestContext, event_queue: EventQueue
) -> None:
\"\"\"Execute the agent's logic for a given request context.
The agent should read necessary information from the `context` and
publish `Task` or `Message` events, or `TaskStatusUpdateEvent` /
`TaskArtifactUpdateEvent` to the `event_queue`. This method should
return once the agent's execution for this request is complete or
yields control (e.g., enters an input-required state).
Args:
context: The request context containing the message, task ID, etc.
event_queue: The queue to publish events to.
\"\"\"
@abstractmethod
async def cancel(
self, context: RequestContext, event_queue: EventQueue
) -> None:
\"\"\"Request the agent to cancel an ongoing task.
The agent should attempt to stop the task identified by the task_id
in the context and publish a `TaskStatusUpdateEvent` with state
`TaskState.canceled` to the `event_queue`.
Args:
context: The request context containing the task ID to cancel.
event_queue: The queue to publish the cancellation status update to.
\"\"\"
For now, we will focus only on event execution and leave cancellation to a future blog. Google provides an implementation of execute() for a generic ADK agent in their A2A Samples repo that I will borrow for this example.
The execute() method concurrently processes requests by keeping track of them with an EventQueue. Requests are presented in the form of a RequestContext, which contains the content of the request, along with other metadata. That is enough info for now, we will get into the weeds in future work.
A critical component to the successful operation of an Agent is the session. A session is a stateful container that allows agents to interact asynchronously. It manages the ongoing interaction between the agents and/or users, context such as memory and state, and a request/response loop to handle communication and execution of actions.
from a2a.server.agent_execution import AgentExecutor, RequestContext
from a2a.server.events import EventQueue
from a2a.server.tasks import TaskUpdater
from a2a.utils import new_task, new_agent_text_message
from a2a.types import TaskState, TextPart, Part
from google.adk.runners import Runner
from google.adk.artifacts import InMemoryArtifactService
from google.adk.sessions import InMemorySessionService
from google.adk.memory import InMemoryMemoryService
from google.genai import types
class MyAgentExecutor(AgentExecutor):
def __init__(self, agent: Agent, status_message: str = \"Executing task...\", artifact_name: str = \"response\"):
self.agent = agent
self.status_message = status_message
self.artifact_name = artifact_name
self.runner = Runner(
app_name=agent.name,
agent=agent,
artifact_service=InMemoryArtifactService(),
session_service=InMemorySessionService(),
memory_service=InMemoryMemoryService(),
)
async def execute(self, context: RequestContext, event_queue: EventQueue) -> None:
query = context.get_user_input()
task = context.current_task or new_task(context.message)
await event_queue.enqueue_event(task)
updater = TaskUpdater(event_queue, task.id, task.contextId)
try:
# Update status with custom message
await updater.update_status(
TaskState.working,
new_agent_text_message(self.status_message, task.contextId, task.id)
)
# Process with ADK agent
session = await self.runner.session_service.create_session(
app_name=self.agent.name,
user_id=\"a2a_user\",
state={},
session_id=task.contextId,
)
content = types.Content(
role='user',
parts=[types.Part.from_text(text=query)]
)
response_text = \"\"
async for event in self.runner.run_async(
user_id=\"a2a_user\",
session_id=session.id,
new_message=content
):
if event.is_final_response() and event.content and event.content.parts:
for part in event.content.parts:
if hasattr(part, 'text') and part.text:
response_text += part.text + '\\n'
elif hasattr(part, 'function_call'):
# Log or handle function calls if needed
pass # Function calls are handled internally by ADK
# Add response as artifact with custom name
await updater.add_artifact(
[Part(root=TextPart(text=response_text))],
name=self.artifact_name
)
await updater.complete()
except Exception as e:
await updater.update_status(
TaskState.failed,
new_agent_text_message(f\"Error: {e!s}\", task.contextId, task.id),
final=True
)
def cancel(self, context, event_queue):
pass
Now that we can run an agent, how do we (and more importantly other agents, since this is a blog about A2A after all...) know what its capabilities are?
\nThe agent card is a public-facing JSON schema that exposes information and metadata to clients (users and other agents). It is like a user profile, but for AI agents. To define an agent card properly, you need:
\nnamedescription: description of the agent, its skills, and other useful informationversionurl: the web endpoint where we can find our agentcapabilities: supported A2A features like streaming or push notificationsskills: A pillar of A2A, the skills discovered in the AgentCard come in a list of AgentSkill objects.For public agents, the A2A project reccomends that the agent card be discoverable at a well-known uri. The standard discovery path is: https://{server-url}/.well-known/agent.json This is the discovery method we will use for now, but for more information, see the A2A discovery page.
Lets give our agent an AgentCard:
from a2a.types import AgentCard, AgentCapabilities
agent_card = AgentCard(
name=my_first_agent.name,
description=my_first_agent.description,
url=\"http://localhost:9999/\",
version='1.0',
capabilities=AgentCapabilities(
streaming=True
),
defaultInputModes=[\"text\", \"text/plain\"],
defaultOutputModes=[\"text\", \"text/plain\"],
skills=[]
)
Agent skills describe specific capabilities the agent has, like searching the web, querying a database, executing an algorithmm...etc. Clients can find out what skills an agent has from the AgentCard. It's kind of like the agentic version of a resume. Skills have some attributes to define:
id: a unique idnamedescription: more detailed information about the skill's functionalitytags: keywordsexamples: example usage of the skillinputModes and outputModes: supported modes for input and output, like text or jsonLet's go back and define the Google search AgentSkill for our agent:
from a2a.types import AgentSkill
web_search_skill = AgentSkill(
id='google_search',
name='Google Search',
description='Searches the web using the google_search tool',
tags=['web search', 'google', 'search', 'look up']
)
agent_card.skills.append(web_search_skill)
Now our agent has advertized the ability to search the web on its agent card. We will take a deep dive into AgentSkills, AgentCapabilities, and AgentCards another time, but now lets zoom out a little bit. We have given lots of detail about our agent. Where it lives, what it can do, examples for how to use it... So how do we run it and start testing this stuff out?
A2A follows a client-server architecture, where the client--a user-facing application or agent--initiates a request to other agents acting as servers that handle those requests, similarly to a traditional web browser or API sending requests to a remote web server. They send structured metadata and information over HTTP using JSON-RPC 2.0 as the format.
\nWith all of the agent functionality defined thus far, let's deploy an agent as a server. The A2A library provides an out-of-the-box app using the Starlette framework to implement the server endpoints and route JSON-RPC requests. This is perfect for our use-case.
from a2a.server.request_handlers import DefaultRequestHandler
from a2a.server.tasks import InMemoryTaskStore
from a2a.server.apps import A2AStarletteApplication
def create_server():
request_handler = DefaultRequestHandler(
agent_executor=MyAgentExecutor(my_first_agent),
task_store=InMemoryTaskStore()
)
return A2AStarletteApplication(
agent_card=agent_card,
http_handler=request_handler,
)
The server application takes a DefaultRequestHandler as an argument, which is the engine for all A2A protocol methods, the AgentExecutor, the TaskStore, the QueueManager, and all other JSON-RPC related logic. Lets spin up the server.
Since we are using jupyter, we need some fancy event handling so as to not have the server process cause the cell to run forever. We need to run it in a background process using a thread.
\nimport uvicorn
import asyncio
import nest_asyncio
import threading
import time
nest_asyncio.apply()
def run_agent():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
server = create_server()
config = uvicorn.Config(
app=server.build(),
host=\"127.0.0.1\",
port=9999,
log_level=\"error\"
)
uvicorn_server = uvicorn.Server(config)
loop.run_until_complete(uvicorn_server.serve())
thread = threading.Thread(target=run_agent, daemon=True)
thread.start()
time.sleep(2)
print(\"Server is running at http://127.0.0.1:9999\")
Quick note: for now, if we need to stop the server, we have to restart the jupyter kernel.
\nGreat news! We now have our very first agent up and running! So now what? We want to be able to interact with our AI agent and benefit from all of the cool stuff it can do. As I mentioned, A2A communication occurs over HTTP, so lets send a simple http request to our agent. We can start by fetching the AgentCard to see what our agent can do.
AgentCardimport httpx
import json
async with httpx.AsyncClient() as http_client:
response = await http_client.get(\"http://localhost:9999/.well-known/agent.json\")
agent_card_json = response.json()
print(json.dumps(agent_card_json, indent=2)) #json.dumps is completely optional but makes the output more readable
{
\"capabilities\": {
\"streaming\": true
},
\"defaultInputModes\": [
\"text\",
\"text/plain\"
],
\"defaultOutputModes\": [
\"text\",
\"text/plain\"
],
\"description\": \"A simple agent that can call a google search\",
\"name\": \"My_first_agent\",
\"protocolVersion\": \"0.2.5\",
\"skills\": [
{
\"description\": \"Searches the web using the google_search tool\",
\"id\": \"google_search\",
\"name\": \"Google Search\",
\"tags\": [
\"web search\",
\"google\",
\"search\",
\"look up\"
]
}
],
\"url\": \"http://localhost:9999/\",
\"version\": \"1.0\"
}
Amazing! Just as we defined it earlier, the AgentCard is showing all of the information about our agent. To interact with our A2A server properly, we need an A2A client.
from a2a.client import A2AClient
my_first_agent_card = AgentCard(**agent_card_json)
a2a_client = A2AClient(
httpx_client=http_client,
agent_card=my_first_agent_card,
url=\"http://localhost:9999/\")
agent_card and url parameters are not both required, but one of them is neeeded to fetch the url of the agent's RPC endpoint. With our client, we can send RPC messages to the A2A server (our agent) by sending an HTTP POST request to the RPC endpoint (the URL). So how do we structure this message and what is it going to look like?
Without diving too deep into the weeds of JSON-RPC 2.0, we can analyze the layout of a simple A2A message, encapsulated in a JSONRPCRequest object:
jsonrpc: must be 2.0method: \"message/send\", \"tasks/get\" ...params: typically an object containing the parameters used for invocationid: unique identifier, not always required if the request does not expect a response.More info here. We will use the message/send method, and pass in our first message:
rpc_request = {
'message': {
'role': 'user',
'parts': [
{'kind': 'text', 'text': 'What are some trending topics in AI right now?'}
],
'messageId': uuid.uuid4().hex,
}
}
The Python SDK for A2A provides lots of useful helper methods that help properly structure our input. Here, we'll create a SendMessageRequest and pass in the rpc_request as the params to our JSONRPCRequest object using the MessageSendParams method:
request = SendMessageRequest(
id=str(uuid.uuid4()),
params=MessageSendParams(**rpc_request)
)
import uuid
from a2a.types import SendMessageRequest, MessageSendParams
rpc_request = {
'message': {
'role': 'user',
'parts': [
{'kind': 'text', 'text': 'What are some trending topics in AI right now?'}
],
'messageId': uuid.uuid4().hex,
}
}
request = SendMessageRequest(
id=str(uuid.uuid4()),
params=MessageSendParams(**rpc_request)
)
print(request)
id='70c6ac85-0e2e-44d9-8183-a71ffaa846b9' jsonrpc='2.0' method='message/send' params=MessageSendParams(configuration=None, message=Message(contextId=None, extensions=None, kind='message', messageId='de7bfa7380ed4c909ad2ad0b22d888f3', metadata=None, parts=[Part(root=TextPart(kind='text', metadata=None, text='What are some trending topics in AI right now?'))], referenceTaskIds=None, role=<Role.user: 'user'>, taskId=None), metadata=None)
Tying everything together, we can send a properly formatted JSON RPC request to our server:
\nimport uuid
from a2a.types import SendMessageRequest, MessageSendParams
from a2a.client import A2AClient
timeout_config = httpx.Timeout(
timeout=120.0,
connect=10.0,
read=120.0,
write=10.0,
pool=5.0
)
async with httpx.AsyncClient(timeout=timeout_config) as http_client:
response = await http_client.get(\"http://localhost:9999/.well-known/agent.json\")
agent_card_json = response.json()
my_first_agent_card = AgentCard(**agent_card_json)
a2a_client = A2AClient(
httpx_client=http_client,
agent_card=my_first_agent_card,
)
message = \"What are some trending topics in AI right now?\"
rpc_request = {
'message': {
'role': 'user',
'parts': [
{'kind': 'text', 'text': message}
],
'messageId': uuid.uuid4().hex,
}
}
request = SendMessageRequest(
id=str(uuid.uuid4()),
params=MessageSendParams(**rpc_request)
)
response = await a2a_client.send_message(request)
Just for fun, lets print out the response. We should expect to get back a response from our server in the form of a JSONRPCResponse object.
\nprint(response)
root=SendMessageSuccessResponse(id='9d9dd71f-f5cf-4005-942a-5ecbdd474dbc', jsonrpc='2.0', result=Task(artifacts=[Artifact(artifactId='2c1511f7-200c-470b-8059-f9ca3819c348', description=None, extensions=None, metadata=None, name='response', parts=[Part(root=TextPart(kind='text', metadata=None, text='Several key themes and advancements are dominating the field of Artificial Intelligence (AI) in 2025, reflecting its rapid evolution and increasing integration across various sectors.\\n\\nSome of the most prominent trending topics in AI right now include:\\n\\n* **Generative AI Evolution and Integration** Generative AI continues to be a major force, moving beyond basic content creation to more advanced and integrated applications. This includes the development of more sophisticated generative AI models, often referred to as Generative AI 2.0, that are becoming autonomous creators. There\\'s a strong focus on integrating generative AI into everyday applications, making it an \"obligatory copilot\" for skilled workers across various industries. Multimodal AI, which can process and generate content from various inputs like text, images, and video, is also seeing significant advancements.\\n* **Agentic AI** A significant trend is the rise of \"Agentic AI,\" where AI systems can act autonomously to achieve specific goals, make decisions, and take actions independently without constant human oversight. These AI agents are envisioned as the \"apps of the AI era,\" handling tasks on users\\' behalf and transforming business processes. Examples include AIs that can surf the web, order groceries, predict sales, and manage inventory.\\n* **AI in the Workplace and Productivity** AI\\'s adoption in the workplace is accelerating, with a focus on enhancing productivity, automating repetitive and mundane tasks, and transforming work dynamics. AI is increasingly used for tasks like sifting through emails, taking notes, generating content, and streamlining job processes. While concerns about job displacement exist, AI is also seen as augmenting human roles and creating new job opportunities.\\n* **Specialized AI Applications** AI is being deeply embedded into specific industries, driving innovation and efficiency:\\n * **Healthcare:** AI is accelerating scientific research and improving healthcare outcomes, with applications in diagnosing patients and identifying health problems.\\n * **Finance:** AI is revolutionizing the finance landscape through autonomous finance, including customer service chatbots, automated forecasting, and AI-powered fraud detection.\\n * **Supply Chain:** AI is enhancing efficiency in supply chain management through optimized inventory and logistics, risk prediction, and sustainability drives.\\n * **Education:** AI is being explored for personalized learning experiences, although its implementation can be controversial.\\n* **Ethical AI and Regulation** As AI becomes more powerful and pervasive, there is increased scrutiny on AI ethics, responsible AI, and the development of broader AI regulations. This includes concerns about privacy, data breaches, and ensuring fundamental rights are respected, as seen with initiatives like the European Union\\'s AI Act.\\n* **Technical Advancements** Underlying these applications are continuous technical improvements. This includes more reasonable reasoning models, advancements in smaller language models (SLMs) that offer efficiency benefits, and the emerging field of \"Embodied AI,\" which aims to bring multimodal AI abilities into the physical world through robotics. There\\'s also a focus on improving the cost-efficiency of AI inference and exploring new model architectures beyond traditional transformers.\\n'))])], contextId='508efe3a-41ca-4e0b-9478-ea82dbce2ea6', history=[Message(contextId='508efe3a-41ca-4e0b-9478-ea82dbce2ea6', extensions=None, kind='message', messageId='7f67307cfede45438e5ed867c38bd9bb', metadata=None, parts=[Part(root=TextPart(kind='text', metadata=None, text='What are some trending topics in AI right now?'))], referenceTaskIds=None, role=<Role.user: 'user'>, taskId='fb816920-46e3-4bdf-8889-c7fdab377e7a'), Message(contextId='508efe3a-41ca-4e0b-9478-ea82dbce2ea6', extensions=None, kind='message', messageId='f2f05466-2943-4671-bd83-fe41eb2193d2', metadata=None, parts=[Part(root=TextPart(kind='text', metadata=None, text='Executing task...'))], referenceTaskIds=None, role=<Role.agent: 'agent'>, taskId='fb816920-46e3-4bdf-8889-c7fdab377e7a')], id='fb816920-46e3-4bdf-8889-c7fdab377e7a', kind='task', metadata=None, status=TaskStatus(message=None, state=<TaskState.completed: 'completed'>, timestamp='2025-07-17T00:37:42.277421+00:00')))
This is a bit tricky to parse through, since RPC calls come with a lot of attached metadata. Let's print just the json:
\nresponse_dict = response.model_dump(mode='json', exclude_none=True)
print(json.dumps(response_dict, indent=2))
{
\"id\": \"9d9dd71f-f5cf-4005-942a-5ecbdd474dbc\",
\"jsonrpc\": \"2.0\",
\"result\": {
\"artifacts\": [
{
\"artifactId\": \"2c1511f7-200c-470b-8059-f9ca3819c348\",
\"name\": \"response\",
\"parts\": [
{
\"kind\": \"text\",
\"text\": \"Several key themes and advancements are dominating the field of Artificial Intelligence (AI) in 2025, reflecting its rapid evolution and increasing integration across various sectors.\\n\\nSome of the most prominent trending topics in AI right now include:\\n\\n* **Generative AI Evolution and Integration** Generative AI continues to be a major force, moving beyond basic content creation to more advanced and integrated applications. This includes the development of more sophisticated generative AI models, often referred to as Generative AI 2.0, that are becoming autonomous creators. There's a strong focus on integrating generative AI into everyday applications, making it an \\\"obligatory copilot\\\" for skilled workers across various industries. Multimodal AI, which can process and generate content from various inputs like text, images, and video, is also seeing significant advancements.\\n* **Agentic AI** A significant trend is the rise of \\\"Agentic AI,\\\" where AI systems can act autonomously to achieve specific goals, make decisions, and take actions independently without constant human oversight. These AI agents are envisioned as the \\\"apps of the AI era,\\\" handling tasks on users' behalf and transforming business processes. Examples include AIs that can surf the web, order groceries, predict sales, and manage inventory.\\n* **AI in the Workplace and Productivity** AI's adoption in the workplace is accelerating, with a focus on enhancing productivity, automating repetitive and mundane tasks, and transforming work dynamics. AI is increasingly used for tasks like sifting through emails, taking notes, generating content, and streamlining job processes. While concerns about job displacement exist, AI is also seen as augmenting human roles and creating new job opportunities.\\n* **Specialized AI Applications** AI is being deeply embedded into specific industries, driving innovation and efficiency:\\n * **Healthcare:** AI is accelerating scientific research and improving healthcare outcomes, with applications in diagnosing patients and identifying health problems.\\n * **Finance:** AI is revolutionizing the finance landscape through autonomous finance, including customer service chatbots, automated forecasting, and AI-powered fraud detection.\\n * **Supply Chain:** AI is enhancing efficiency in supply chain management through optimized inventory and logistics, risk prediction, and sustainability drives.\\n * **Education:** AI is being explored for personalized learning experiences, although its implementation can be controversial.\\n* **Ethical AI and Regulation** As AI becomes more powerful and pervasive, there is increased scrutiny on AI ethics, responsible AI, and the development of broader AI regulations. This includes concerns about privacy, data breaches, and ensuring fundamental rights are respected, as seen with initiatives like the European Union's AI Act.\\n* **Technical Advancements** Underlying these applications are continuous technical improvements. This includes more reasonable reasoning models, advancements in smaller language models (SLMs) that offer efficiency benefits, and the emerging field of \\\"Embodied AI,\\\" which aims to bring multimodal AI abilities into the physical world through robotics. There's also a focus on improving the cost-efficiency of AI inference and exploring new model architectures beyond traditional transformers.\\n\"
}
]
}
],
\"contextId\": \"508efe3a-41ca-4e0b-9478-ea82dbce2ea6\",
\"history\": [
{
\"contextId\": \"508efe3a-41ca-4e0b-9478-ea82dbce2ea6\",
\"kind\": \"message\",
\"messageId\": \"7f67307cfede45438e5ed867c38bd9bb\",
\"parts\": [
{
\"kind\": \"text\",
\"text\": \"What are some trending topics in AI right now?\"
}
],
\"role\": \"user\",
\"taskId\": \"fb816920-46e3-4bdf-8889-c7fdab377e7a\"
},
{
\"contextId\": \"508efe3a-41ca-4e0b-9478-ea82dbce2ea6\",
\"kind\": \"message\",
\"messageId\": \"f2f05466-2943-4671-bd83-fe41eb2193d2\",
\"parts\": [
{
\"kind\": \"text\",
\"text\": \"Executing task...\"
}
],
\"role\": \"agent\",
\"taskId\": \"fb816920-46e3-4bdf-8889-c7fdab377e7a\"
}
],
\"id\": \"fb816920-46e3-4bdf-8889-c7fdab377e7a\",
\"kind\": \"task\",
\"status\": {
\"state\": \"completed\",
\"timestamp\": \"2025-07-17T00:37:42.277421+00:00\"
}
}
}
If we want to extract just the text, which is what we will pass in as context to future interactions with our agent, we can do that as well:
\nif 'result' in response_dict and 'artifacts' in response_dict['result']:
artifacts = response_dict['result']['artifacts']
for artifact in artifacts:
if 'parts' in artifact:
for part in artifact['parts']:
if 'text' in part:
print(part['text'])
Extracted text:
\n Several key themes and advancements are dominating the field of Artificial Intelligence (AI) in 2025, reflecting its rapid evolution and increasing integration across various sectors.
Some of the most prominent trending topics in AI right now include:
* **Generative AI Evolution and Integration** Generative AI continues to be a major force, moving beyond basic content creation to more advanced and integrated applications. This includes the development of more sophisticated generative AI models, often referred to as Generative AI 2.0, that are becoming autonomous creators. There's a strong focus on integrating generative AI into everyday applications, making it an \"obligatory copilot\" for skilled workers across various industries. Multimodal AI, which can process and generate content from various inputs like text, images, and video, is also seeing significant advancements.
* **Agentic AI** A significant trend is the rise of \"Agentic AI,\" where AI systems can act autonomously to achieve specific goals, make decisions, and take actions independently without constant human oversight. These AI agents are envisioned as the \"apps of the AI era,\" handling tasks on users' behalf and transforming business processes. Examples include AIs that can surf the web, order groceries, predict sales, and manage inventory.
* **AI in the Workplace and Productivity** AI's adoption in the workplace is accelerating, with a focus on enhancing productivity, automating repetitive and mundane tasks, and transforming work dynamics. AI is increasingly used for tasks like sifting through emails, taking notes, generating content, and streamlining job processes. While concerns about job displacement exist, AI is also seen as augmenting human roles and creating new job opportunities.
* **Specialized AI Applications** AI is being deeply embedded into specific industries, driving innovation and efficiency:
* **Healthcare:** AI is accelerating scientific research and improving healthcare outcomes, with applications in diagnosing patients and identifying health problems.
* **Finance:** AI is revolutionizing the finance landscape through autonomous finance, including customer service chatbots, automated forecasting, and AI-powered fraud detection.
* **Supply Chain:** AI is enhancing efficiency in supply chain management through optimized inventory and logistics, risk prediction, and sustainability drives.
* **Education:** AI is being explored for personalized learning experiences, although its implementation can be controversial.
* **Ethical AI and Regulation** As AI becomes more powerful and pervasive, there is increased scrutiny on AI ethics, responsible AI, and the development of broader AI regulations. This includes concerns about privacy, data breaches, and ensuring fundamental rights are respected, as seen with initiatives like the European Union's AI Act.
* **Technical Advancements** Underlying these applications are continuous technical improvements. This includes more reasonable reasoning models, advancements in smaller language models (SLMs) that offer efficiency benefits, and the emerging field of \"Embodied AI,\" which aims to bring multimodal AI abilities into the physical world through robotics. There's also a focus on improving the cost-efficiency of AI inference and exploring new model architectures beyond traditional transformers.
...and that's it for now, we have covered a lot of ground, yet only brushed the surface of A2A and the power it truly holds.
\nIn the coming weeks, we will be releasing a series of blog posts covering the nuances of A2A in depth:
\nAgentCard, AgentSkills, AgentExecutorWe will also be building complex multi-agent workflows and learning how to compose them with the power of Littlehorse.io! Stay tuned!
", "url": "https://littlehorse.io/blog/a2a-protocol", "title": "Agent 2 Agent Protocol (A2A)", "summary": "Exploring Google's A2A protocol for standardized inter-agent communication and multi-agent systems", "date_modified": "2025-07-18T00:00:00.000Z", "author": { "name": "Jackson Oleson" }, "tags": [ "AI, LLM's, and Agents", "Protocols" ] }, { "id": "https://littlehorse.io/blog/callbacks", "content_html": "Sometimes, business processes take time, and you need to wait for something to happen. It can be tricky to build backend software that reliably handles such cases; thankfully, workflow engines like LittleHorse make it a bit easier.
\nThis is the fifth and final part in a five-part blog series on useful Integration Patterns. This blog series will help you build real-time, responsive applications and microservices that produce predictable results and prevent the Grumpy Customer Problem.
Digital applications must support business processes, and not all business processes execute instantaneously. Sometimes, a process must wait for something to happen: an item needs to get back in stock, a delivery must be completed, or a customer must sign a document. In all three of these cases, the software systems you write need to wait for something out of their control to happen, and then resume taking action once it does happen.
\nFor the rest of this blog, we will use the term External Event to refer to such instants in which our overall process needs to wait for something external to happen.
\nLet's consider a process in which a fictitious real-estate company sends a closing document to a lawyer, waits for execution, and upon completion then updates the status of a listing from \"in escrow\" to \"sold.\"
\nThe process might look like the following:
\n![\"A](\"https://littlehorse.io/img/2025-01-14-signing-process.png\" "\\"External")
In order to make it a little bit more tricky, let's add one wrinkle: if the document is not signed within one week, we \"cancel\" the offer and roll back the flow.
\nLet's take a look at how this process would be implemented without a composer and with LittleHorse. In both cases, the process will be kicked off by a POST call to the /sign-document endpoint on a web server.
We will assume that, once the document is signed, we receive a webhook including some information about which document was signed.
In the \"wild west\" without a composer, we don't have a \"leader\" in the room that is in charge of composing the process from start to finish. Practically, we as software engineers won't write a single piece of code that represents the whole process.
\nLet's start with the /sign-document endpoint (our entrypoint). This endpoint needs to do the following:
The first part is easy—just make an RPC call inside your HTTP handler. For the second part, we have two options: use a delayed queue, or write to a table and use a cronjob.
\nWe briefly discussed delayed queues in Part IV. Some queueing systems, such as AWS SQS and Rabbit MQ, allow you to write messages that are only delivered after some delay.
\nOur entrypoint HTTP handler would put the message in the delay queue and send the document to be signed. The messages in the queue would be consumed and processed one week later by (yet another) system which consumes the queue, checks to see whether or not the document was signed, and then optionally cancels the listing (if the document wasn't signed).
\nWe would need one final HTTP handler, which is responsible for listening for completed documents and marking the listing as \"sold\" when the document is signed.
\nPseudocode for our HTTP handler for POST /sign-document might look like the following:
@PostMapping(\"/sign-document\")
public void signDocument(@RequestMapping(\"document\") DocumentDetails document) {
db.updateListingStatus(document.getListingId(), \"SIGNING_DOCUMENT\");
String queueName = \"check-expired-docusign\";
queue.scheduleMessage(queueName, document.getListingId(), Duration.ofDays(7));
}
The above code is not robust to a sudden crash while the HTTP Handler is handling the request, which could lead to duplicated or dropped processing.
The callback handler (which listens for completed documents) would update the status of the listing to \"SOLD\" when handling the document. When consuming the message one week later, our consumer would be responsible for checking to make sure that the listing was not yet in the \"SOLD\" status before taking compensatory actions.
\nNote that the responsibility for overall business flow is spread across three components:
\nPOST /send-document)If your message queue doesn't have the ability to schedule delayed writes, you aren't fully out of luck. You can spawn a process which periodically checks for listings that have been in the \"signing document\" status for over a week. This would likely necessitate a modification to the /sign-document endpoint so that you can keep track of when the document was initially signed, which is still more work.
Furthermore, running cronjobs can be challenging from a Day 2 Operations perspective, and we still have the problem of our business logic being spread across three disparate components (which are most likely deployed separately).
\nUsing a workflow engine to handle your callback-driven workflows simplifies two key aspects of your architecture:
\nWfSpec, which sits right in front of you.Additionally, running your processes in a workflow engine gives you all of the observability and reliability features that we have discussed in the first four episodes.
\nAn executable WfSpec for this use-case might look as follows:
var listingId = wf.declareStr(\"listing-id\");
var customerId = wf.declareStr(\"customer-id\");
wf.execute(\"send-docusign\", listingId, customerId);
// wait for the callback
long sevenDays = 60 * 60 * 24 * 7;
NodeOutput eventResult = wf.waitForEvent(\"document-signed\").timeoutSeconds(sevenDays);
// Fail if it doesn't return in a week
wf.handleException(eventResult, \"TIMEOUT\", handler -> {
handler.execute(\"cancel-listing\", listingId);
handler.fail(\"signing-timeout\", \"Closing documents were not signed.\");
});
wf.execute(
\"complete-listing\",
listingId,
// Use the output from the external event in the workflow
wf.format(\"Closing documents signed by {0}.\", eventResult.jsonPath(\"$.signer\"))
);
The key point to note here is that, with LittleHorse you get to write a single piece of code (your WfSpec) that controls the entire process end-to-end.
Our document signing system will necessarily have the following components:
\nPOST /sign-document endpoint.This is the unfortunate reality of event-driven systems: business flows are spread across multiple components. However, using LittleHorse to compose these processes reduces the complexity of the deployment model and your mental model of how the system works.
\nWhen business flows are spread across many components, that could be multiple components within the same deployable (a modular monolith), or it could mean multiple separately-deployed microservices. Either way, the problem of wrangling end-to-end flow remains.
Congrats on making it through the Integration Patterns series! From the Saga Pattern to queues, retries, and the Outbox Pattern, we've gone through some of the more commonly-used tools and patterns in the belt of software engineers who build reliable and responsive systems.
\nAll of these tools are useful; unfortunately, they are also tricky to get right. Thankfully, LittleHorse makes it easy for you to implement these patterns, taking advantage of the reliability and responsiveness they provide while keeping your architecture simple.
\nWhether or not you end up using LittleHorse for your microservices, I truly hope this series of blogs was useful to you. And if you do want to get involved with LittleHorse:
\nOver the next few weeks, we will announce three big new releases (two open-source, one product). I also plan to write about the connection between Streams/Tables/Workflows, and how Workflow is similar to (but cooler than!) its cousin Durable Execution.
", "url": "https://littlehorse.io/blog/callbacks", "title": "Integration Patterns V: Callbacks and External Events", "summary": "How to handle callbacks and external events reliably in long-running distributed workflows.", "date_modified": "2025-01-14T00:00:00.000Z", "author": { "name": "Colt McNealy" }, "tags": [ "Integration Patterns", "Orchestration" ] }, { "id": "https://littlehorse.io/blog/retries-and-dlq", "content_html": "Retries and Dead-Letter Queues allow you to make multi-step processes reliable in the face of failure in order to avoid the Grumpy Customer Problem, but implementing them properly is quite a toil. Thankfully, LittleHorse helps keep your engineers happy by simplifying this process.
\n\nThis is the fourth part in a five-part blog series on useful Integration Patterns. This blog series will help you build real-time, responsive applications and microservices that produce predictable results and prevent the Grumpy Customer Problem.
In the previous post, we discussed an application which accepts reviews of a product, and first uses a third-party AI service to detect offensive content before finalizing and accepting the review. The slow response times (and flakiness) of the AI service make asynchronous processing a natural fit for this use-case: once the review is saved into a database or queue, you can immediately respond to the client and analyze the review using the AI service later.
\nWe saw that we can successfully make our API responsive and performant for our users either by using a traditional message queue or event log (such as AWS SQS or Apache Kafka) or by using a composer. However, using a message queue has significant advantages in terms of debugging, monitoring, and long-term maintainability.
\nWe haven't yet addressed what happens when the flakey external service fails. In order to avoid the Grumpy Customer problem, we must periodically retry the request (and subsequent processing) until the service responds successfully.
\nWhen we discussed the classic message queue architecture in the last post, we left off with a consumer that polled records from a queue, then made the external API call and either rejected or approved the review. In pseudo-java, naive code might look like:
\nfor (Review reviewToProcess : queue.iterAllEvents()) {
ReviewAnalysisResponse reviewAnalysis = thirdPartyService.analyzeReview(request);
if (reviewAnalysis.containsOffensiveContent()) {
rejectReview(reviewAnalysis);
} else {
acceptReview(reviewAnalysis);
}
queue.acknowledge(reviewToProcess);
}
But what about when the call to analyzeReview() fails (because the service is flakey)? There are a lot of questions we need to consider.
Do we let the exception bubble up, crash our pod, restart it, and then poll the message again (since we never called acknowledge())? This might work if outages are very short and intermittent, but not for sustained downtime of the queue service. Additionally, what if there's a \"poison pill\" message that, for whichever reason, cannot be successfully processed (eg. a deserialization error)?
Furthermore, if we start spamming the system with retried requests, won't we end up making a bad problem worse? Lastly, when do we \"give up\" for an individual message, and what happens when we do?
\nIn order to harden our app, we will do two things:
\nExponential backoff retries is a strategy where the delay between retry attempts increases exponentially, often with random jitter, to reduce contention and prevent synchronized retries. Read more here.
A wise man once said, \"if at first you don't succeed, try, try again!\" Processing messages from a queue is no different. However, if we naively retry every message immediately upon failure, we risk exacerbating the unavailability of our flakey service by overloading it with many retry requests at once. Exponential Backoff solves this problem by increasing the delay between retry attempts exponentially. This reduces the risk of accidentally DDOS'ing the flakey service. By adding a delay between attempts, we can safely wait until the flakey service comes back to life, and then resume processing.
\nThe first step towards implementing retries with exponential backoff is catching failures and enqueuing them with appropriate delay. Once we catch a failure, our consumer process needs to:
\n\nSome queues have native support for retries; however, implementing exponential backoff is still quite a challenge.
In order to calculate the amount of time to wait for the next retry, we follow the Exponential Backoff formula:
\ntime = base * exponent^n
Here, the n represents the attempt number. Therefore, for a given Review, we need to know the last time that we attempted to process it, which requires adding some metadata to the message. This is not the end of the world, but this piece of extra complexity adds one more opening for bugs to appear.
Once we have calculated the next appropriate delivery time for the retried message, we must schedule it for delivery. Some queues, such as AWS SQS, support this natively with \"delay queues.\" However, other systems such as Apache Kafka and AWS Kinesis do not support delayed messages. For these cases, we would have to persist the messages ourselves and use a cronjob to re-publish the messages to the queue at a later date.
\nEven with retries and exponential backoff, there might be some Reviews that we quite simply cannot process (or the third-party service might be in especially dire straits one day). For these messages, we will use the Dead-Letter Queue Pattern.
A Dead Letter Queue (DLQ) is a secondary queue used to store messages that cannot be processed successfully after a predefined number of retry attempts or due to specific errors. It helps isolate problematic messages for later inspection and debugging without disrupting the processing of valid messages.
Once we have failed a certain number of times to process a Review, we can publish a message to a failed-reviews queue (or topic). DLQ's can be used purely for debugging and manual intervention purposes; however, applications can also consume from a DLQ and take automated actions on the messages in the queue.
For example, our review-processing application might have an additional consumer which notifies the customer (or the on-call team, or both) that the review wasn't processed.
\nIn the previous section, we saw how retries with exponential backoff allow us to we can more reliably process our reviews in the face of a flakey third-party service. However, there was a lot of manual toil and additional infrastructure to set up (potentially involving a database and cronjob depending on whether our queue service supports delayed messages).
\nAs you might have guessed, this is much easier to accomplish with a durable workflow composer. As we discussed in the last post, we can model the entire end-to-end process (analyze the review, and either save or reject it) in a single WfSpec, as shown below:
public void reviewProcessorWf(WorkflowThread wf) {
WfRunVariable userId = wf.declareStr(\"user-id\").searchable();
WfRunVariable review = wf.declareStr(\"review\");
WfRunVariable approved = wf.declareBool(\"approved\");
// Call the flakey API with exponential backoff.
var reviewResult = wf.execute(\"analyze-review\", review)
.withRetries(10)
.withExponentialBackoff(ExponentialBackoffRetryPolicy.newBuilder()
.setBaseIntervalMs(5000)
.setMultiplier(2.0F)
.build());
approved.assign(reviewResult.jsonPath(\"$.okay\"));
wf.doIfElse(approved.isEqualTo(true), handler -> {
handler.execute(\"notify-customer-approved\", userId);
}, handler -> {
handler.execute(\"notify-customer-rejected\", userId);
});
}
With just this block, we defined our retry policy and don't need to worry about calculating the delay, keeping track of how many times a message has been retried, or persisting attempts:
\nvar reviewResult = wf.execute(\"analyze-review\", review)
.withRetries(10)
.withExponentialBackoff(ExponentialBackoffRetryPolicy.newBuilder()
.setBaseIntervalMs(5000)
.setMultiplier(2.0F)
.build());
Since the workflow composer automatically persists the state of all workflows for debugging purposes, you don't even need a DLQ! You can quite simply search for WfRun's with the ERROR status in the dashboard.
If you want to manually save a failed WfRun after the outage is resolved, you can do so with the lhctl rescue command.
In case we must process failed reviews (for example, notify the customer to say that the review was unable to be processed), we can do so using LittleHorse's Failure Handling feature, as follows:
\nwf.handleError(reviewResult, handler -> {
wf.execute(\"notify-customer-review-failed\", userId);
wf.fail(\"review-not-processed\"); // fail the workflow
});
Retries and DLQ's are almost always necessary if you require reliable message processing. While they can be tedious to implement, LittleHorse allows you to enjoy their benefits without the frustrating (and error-prone) toil.
\nStay tuned for the next post, which will cover how you can write an application that must wait for something external to happen! In the meantime:
\nWhen integrating API's, we sometimes have to tie together steps that can take a long time or might not always be available. If we force the callers of our API's to wait for completion, we find ourselves with some grumpy customers. So what can we do about this?
\nThis is the third part in a five-part blog series on useful Integration Patterns. This blog series will help you build real-time, responsive applications and microservices that produce predictable results and prevent the Grumpy Customer Problem.
In software architecture, simple is almost always better. With fewer moving parts, there are less chances for failure, less things to debug, and fewer pieces of infrastructure. So when and why would you introduce queues to your architecture?
\nQueues are useful when building services that need to accept a request from a client and then execute some processing which has any of the following characteristics:
\nCrucially, if your service enqueues requests, you need to make sure that the caller of your API doesn't need to wait for their entire request to be processed: a simple promise that it will get done should be sufficient. As we will see with a practical example, this is feasible in many business cases.
\nConsider a product reviews widget on an e-commerce site. In this application, users can submit reviews of a product. However, before a review can be approved to be displayed, it must first be checked for offensive content by a third-party AI service. Sometimes, this third-party service often has response times of over 10 seconds, and sometimes even goes down and is fully unavailable.
\nA naive web app endpoint to handle this use-case might be:
\n@PostMapping(\"/review\")
public ResponseEntity<String> postReview(@RequestBody PostReviewRequest request) {
// Call the third-party AI service, which takes a long time and is flakey
try {
ReviewAnalysisResponse reviewAnalysis = thirdPartyService.analyzeReview(request);
} catch(OffensiveReviewException exn) {
return ResponseEntity.status(400);
} catch(Exception exn) {
return ResponseEntity.status(500);
}
// If we got here, the review is valid
reviewService.save(request);
return ResponseEntity.status(HttpStatus.CREATED);
}
As promised ( 😉 ), this endpoint implementation has a sub-optimal user experience. Many times, when the flakey third-party AI service is unavailable, users will simply be unable to post reviews. Even when it is up, users will see the spinning waiting wheel for multiple seconds.
\nThe solution? Enqueue the request for processing later by some external system, and then respond immediately to the client's request. That can be done in two ways:
\nOnce the request is enqueued, there will be a system polling the queue to call the third-party analytics API and then either reject or approve the review. This system will be responsible for throttling requests according to the API's service limits, retrying failed messages, and waiting for the API to come back online in the case of an intermittent outage.
\nWorkflow engines internally use message queues on their own! So what's the difference from the user perspective?
\nYou can think of a workflow engine as a super-smart message queue, with certain clear advantages over message queues including advanced monitoring and better support for multi-step processes.
\nThe next post in this series will take a deep-dive into retries, idempotency, and failure handling, which is another area in which workflow engines shine above and beyond Plain Old Queues.
Workflow engines provide more insight and oversight into your processes than do message queues. In our reviews application, if an angry user (anakin@jeditemple.com) calls customer support to complain that his review hadn't been processed in over two days, it would be tricky to find the exact cause with a pure message queue.
However, with LittleHorse, you just search for the WfRun where user-id == anakin@jeditemple.com:
![\"Dashboard](\"https://littlehorse.io/assets/images/2024-10-28-workflow-search-6abd5f2dbb3914b3230502f28afab1f2.png\")
and then look on the dashboard to see what went wrong:
\n![\"Dashboard](\"https://littlehorse.io/assets/images/2024-10-28-task-debugging-dc5d52151429b8428cfa326e57cc53d5.png\")
We are also working on workflow metrics that will allow you to use LittleHorse to answer questions such as:
\nprocess-review workflow take on average?analyze-review task attempt take on average, and what percentage of calls fail (i.e. how responsive is the API)?These will likely not be available until March 2025; however, we have nearly finalized the designs for them and have scheduled the implementation to start in January.
\nSo far, the use-case we've discussed involves only two \"steps\" to be executed:
\nYou could arguably execute both steps at once: the only problem we are trying to solve is that we have a flakey API and we don't want our customers to have to wait for it. In theory, the same consumer which calls the external API could also update the visibility of the review to APPROVED.
But what if the business requirements change, and we need to do some post-processing, such as notify a separately-managed (and also flakey) analytics service of what happened? That would require adding another queue:
\nThis gets especially tricky when we want to handle intermittent availability from the analytics service: we'll have to copy the same boilerplate to handle retries and dead-letter-queues (more on that in the next post).
\nHowever, with the workflow-driven approach, all you need to do is add a single line to your WfSpec:
wf.execute(\"notify-analytics-service\", userId, review, approvalStatus);
Queueing is a great tool to improve the client experience of your API's when you can respond to your callers before all of your processing has been done. Workflow engines like LittleHorse can actually be thought of as a super-smart queueing system, which provides all of the advantages of queueing plus better observability and support for multi-step processes.
\nStay tuned for the next post, which will cover retries and dead-letter queues! In the meantime:
\nLike the Saga Pattern, the Transactional Outbox pattern is tool for defending against data loss in your applications. In this blog we cover how it works and how to do it easier using LittleHorse.
\nThis is the first part in a five-part blog series on useful Integration Patterns. This blog series will help you build real-time, responsive applications and microservices that produce predictable results and prevent the Grumpy Customer Problem.
At the technical level, the Transactional Outbox Pattern allows you to atomically:
\nThe Saga Pattern allows you to make a multi-step business process atomic. However, you can think of the Transactional Outbox pattern as a way to ensure that a process doesn't get dropped halfway through.
\nThe Transactional Outbox Pattern is often useful within a Saga transaction.
However, as we'll see later on in this article, LittleHorse removes the need to worry about such difficult technical details.
As an example, let's consider the following Spring Boot REST endpoint (POST /user), which must:
@PostMapping(\"/user\")
public ResponseEntity<String> createUser(@RequestBody CreateUserRequest request) {
database.createUser(request);
queue.publishUserCreatedEvent(request);
return ResponseEntity.status(HttpStatus.CREATED);
}
A few things can go wrong here which would cause the user to be created in the database but the customer never gets a welcome email, and the account setup fails. First, the queue could be inaccessible (this could be saved at the application layer with an exception handler).
\nHowever, one failure mode which cannot be caught at the application layer is if the Spring Boot app crashes during the process of publishing the record to the queue (on or just before the queue.publishUserCreatedEvent() line).
This would definitely cause another case of the Grumpy Customer Problem!
\nThe core idea of a Transactional Outbox is to make use of transactions within a single database to atomically:
\n![\"Transactional](\"https://littlehorse.io/assets/images/2024-09-30user-workflow-outbox-717d91ae3fad27e1d67957580625ab19.png\")
Since items 1 and 2 happen within a single database, it's trivial to wrap them in a transaction. After the queue event is written to the Outbox Table, a separate process eventually reads the new records in the Outbox Table and pushes them to a queue.
We would rewrite our Spring Boot endpoint to only write a transaction to the database. The SQL for that transaction would look something like:
\nBEGIN TRANSACTION;
INSERT INTO user VALUES ...;
INSERT INTO outbox VALUES ...; # Insert the record for the queue
COMMIT;
The Spring Boot application should have another thread which reads records from the outbox table, publishes them to the queue or streaming system, and updates the record in the database as \"read\".
The topic of Exactly-Once Semantics is complex; we do not have time in this post to discuss the implications of EOS and a Transactional Outbox.
As a hint, you can achieve EOS if you transactionally store the last-written offset inside your message broker. There are many \"gotchas\" to this depending on your message broker; for example, in Apache Kafka you must use read_committed consumers.
The Outbox Pattern is necessary to persist outgoing records in the case that we suffer a crash between writing to the database and writing to the record queue. However, what if we could \"delegate\" persistence and reliability to some other system?
\nEnter LittleHorse! What if we had a WfSpec that defined our process, as follows:
public void wfLogic(WorkflowThread wf) {
var userRequest = wf.addVariable(\"create-user-request\", JSON_OBJ).required();
wf.execute(\"create-user\", userRequst);
wf.execute(\"send-welcome-email\", user);
}
Now, all our REST endpoint has to do is run the worklfow:
\n@PostMapping(\"/user\")
public ResponseEntity<String> createUser(@RequestBody CreateUserRequest request) {
// Just run the workflow
littlehorseClient.runWf(RunWfRequest.newBuilder()
.setWfSpecName(\"user-workflow\")
.putVariables(\"create-user-request\", LHLibUtil.objToVarVal(request))
.build());
return ResponseEntity.status(HttpStatus.CREATED);
}
![\"Transactional](\"https://littlehorse.io/assets/images/2024-09-30-user-workflow-lh-e7a8ae4a642a0a8d2c4ab2635ce7b445.png\")
No outbox table needed! If creating the user in the database fails, or if sending the welcome email fails, LittleHorse will patiently retry (according to your retry backoff policy) the TaskRuns until they succeed. In the event that you exhaust your retries, you still haven't lost data:
rpc RescueThreadRun once the database incident is resolved.You can add retries using the Workflow object:
Workflow wf = Workflow.newWorkflow(\"user-workflow\", this::wfLogic);
wf.setDefaultTaskRetries(10);
wf.setDefaultTaskExponentialBackoffPolicy(ExponentialBackoffRetryPolicy.newBuilder()
.setBaseIntervalMs(1000)
.setMultiplier(3)
.build());
wf.registerWfSpec(littlehorseClient);
The Transactional Outbox Pattern is a useful and often necessary tool for building reliable integrations between systems. However, it takes time, infrastructure, and deep understanding of distributed systems to get it right. So why spend time solving problems that don't differentiate your business?
\nThankfully, LittleHorse offers a workaround to the original problem, removing the need to engage with the complexities of Transactional Outboxes.
\nThe Transactional Outbox pattern is useful anytime you need to update information in a database and also publish a record to a streaming log or a message queue.
\nFor example:
\nAnother solution to this specific problem would be to have the request handler (our Spring Boot endpoint) publish directly to an event log like Apache Kafka. Then, there would be two consumer groups for that topic:
\nuser record in the database.The REST endpoint would return 201 as soon as the record was acknowledged by the streaming platform.
If you squint hard enough, you can see that this is very similar to what happens with LittleHorse; however, using this pattern, you are responsible for wiring together a complex topology of topics and queues, which is much harder than using a workflow!
\nStay tuned for the next post on Queues and Backpressure! In the meantime:
\nThe Saga pattern allows you to defend against data loss, dropped orders, and confused (or grumpy) customers. While useful, the Saga pattern is tricky to get right without a composer.
\nThis is the first part in a five-part blog series on useful Integration Patterns. This blog series will help you build real-time, responsive applications and microservices that produce predictable results and prevent the Grumpy Customer Problem.
At a technical level, the Saga Pattern allows you to perform distributed transactions across multiple disparate systems without 2-phase commit.
\nIn plain English, it is a tool in the belt of a software engineer to prevent half-fulfilled bank transfers, hanging orders, or other failures which would result in a Grumpy Customer.
\nThe \"Saga\" pattern gets its name from literature and film, wherein a \"saga\" is a series of chronologically-ordered related works. For example, the \"Star Wars Saga.\"
Business processes often need to perform actions in two separate systems either all at once or not at all. For example, you may need to charge a customer's credit card, reserve inventory, and ship an item to the customer all at once or not at all. If the payment went through but shipping failed, we would see the Grumpy Customer Problem yet again.
\nThe Saga pattern is appropriate when:
\nIt's also worth noting that a different flavor of the Saga pattern can also be used in long-running business processes. In a past job, for example, I worked on a project that implemented the Saga pattern to handle the scheduling of home inspections. In this case, the task of finding an inspector to show up at the home and confirming a time with the homeowner needed to be performed atomically.
While Saga is very hard to implement, it's simple to describe:
\nThe compensation is simply an action that \"undoes\" the previous action. For example, the compensation for a payment task might be to issue a refund.
\nLet's take a look at a familiar use-case: an order processing workflow involving the inventory service, and the payments service. (The orders service is involved implicitly.) As they would in a real world scenario, all of our services live on separate physical systems and have their own databases.
In this business process, we first reserve inventory for the ordered item. Next, we charge the customer's credit card.
\nIf charging the credit card fails, then we have a problem: we've reserved inventory but not sold it.
\nOur services need the following functionality. In SOA, these would be endpoints; in LittleHorse, they would be TaskDefs:
create-order: creates an order in the PENDING status.reserve-inventory: marks an item as no longer available for sale.charge-payment: charges the customer.release-inventory: marks an item as available for sale again.cancel-order: marks an order as CANCELED.complete-order: marks an order as COMPLETED.Using message queues, the happy path looks like the following:
\n![\"Architecture](\"https://littlehorse.io/assets/images/2024-09-24-choreography-simple-830e1fcf682eb3cde8b40210f92b3dd2.png\")
The above image assumes the choreography pattern, in contrast to the composer pattern. The composer pattern is a ton of work and involves writing something that very much resembles LittleHorse!
createOrder().reserve-inventory queue.reserveInventory().charge-payment queue.complete-order queue.completeOrder().In just the happy path, we have strong coupling already between our services in three places, and we have three message queues to manage.
\nBut now we need to release the inventory and cancel the order when the payment doesn't go through. So the flow looks like this:
\n![\"Architecture](\"https://littlehorse.io/assets/images/2024-09-24-choreography-saga-b400a44518ce7213d491df50bad0bb72.png\")
createOrder().reserve-inventory queue.reserveInventory().charge-payment queue.release-inventory queue.releaseInventory().cancel-order queue.cancelOrder().We still haven't even considered the case when the reserve-inventory step fails and we need to catch that exception and handle the order. For the sake of brevity, we will leave that out.
Now, we have five different message queues that we have to wrangle with. We can also see that the overall business flow has started to leak across all of our different services.
\nOne thing we are ignoring in this blog post is reliability: to make this setup production-ready, we would also have to ensure that updates to the internal databases of the services are atomic along with pushing messages to the message queue. We will cover that in next week's post (along with how LittleHorse takes care of that for you).
Using LittleHorse, in java, this whole workflow could look like the following. This is real code that does indeed compile and replaces the need for all of the complex queueing logic we had above.
\npublic void sagaExample(WorkflowThread wf) {
var item = wf.addVariable(\"item\", STR);
var customer = wf.addVariable(\"customer\", STR);
var price = wf.addVariable(\"price\", DOUBLE);
var orderId = wf.addVariable(\"order-id\", STR);
wf.execute(\"create-order\", orderId);
// Saga Here! (We skipped this part in the previous section due to
// complexity, but LittleHorse makes it simple enough.
NodeOutput inventoryResult = wf.execute(\"reserve-inventory\", item, orderId);
wf.handleException(inventoryResult, \"out-of-stock\", handler -> {
handler.execute(\"cancel-order\", orderId);
handler.fail(\"out-of-stock\", \"Item was out of stock. Order canceled\");
})
NodeOutput paymentResult = wf.execute(\"charge-payment\", customer, price);
// Saga here again!!
wf.handleException(paymentResult, \"credit-card-declined\", handler -> {
handler.execute(\"release-inventory\", item, orderId);
handler.execute(\"cancel-order\", orderId);
handler.fail(\"credit-card-declined\", \"Credit card was declined. Order canceled!\");
});
wf.execute(\"complete-order\", orderId);
}
Instead of managing five message queues and five strongly-coupled integration points between microservices, all we need to do is register the workflow, define truly modular tasks, and let LittleHorse take care of the rest.
\nThe Saga Pattern is one of five tools we will cover in this series on avoiding the Grumpy Customer Problem. It's simple to understand but painfully complex to implement. Fortunately, LittleHorse makes it easier!
\nA careful reader, or anyone who reads my rants on LinkedIn, might note that in order to make the order processing workflow truly reliable, we would also need to do something like the Outbox pattern or Event Sourcing.
That is true, and we'll cover it in the next post (and you'll see how LittleHorse does that for you automatically!).
Stay tuned for the next post on the Transactional Outbox Pattern! In the meantime:
\nLittleHorse Enterprises is a workflow engine company. But what is a workflow engine?
\nIt is a system that allows you to reliably execute a series of steps while being robust to technical failures (network outages, crashes) and business process failures. A step in a workflow can be calling a piece of code on a server, reaching out to an external API, waiting for a callback from a person or external system, or more.
\nA core challenge when automating a business process is Failure and Exception Handling: figuring out what to do when something doesn't happen, happens with an unexpected outcome, or plain simply fails. This is often difficult to reason about, leaving your applications vulnerable to uncaught exceptions, incomplete business workflows, or data loss.
\nA workflow engine standardizes how to throw an exception, where the exception is logged, and the logic around when/how to retry. This gives you peace of mind that once a workflow run is started, it will reliably complete.
\nAny workflow-driven application has three components:
\nA Workflow Specification (or WfSpec in LittleHorse) is the configuration, or metadata object, that tells the engine what Tasks to run,\nwhat order to run the tasks, how to handle exceptions or failures, what variables are to be passed from task to task, and what inputs and outputs are required to run the workflow.
In LittleHorse the WfSpec is submitted to and held by the LittleHorse server. Users of LittleHorse can define a WfSpec in vanilla code (Java/Go/Python) using the LittleHorse SDK. The SDK will compile your vanilla code into a WfSpec that the LittleHorse Server understands and keeps inside its data store.
To learn how to write a WfSpec in LittleHorse, check out our WfSpec Development docs.
In the background LittleHorse server takes the submitted spec from the SDK, and compiles a protobuf object that is submitted to the LittleHorse server.
\nFor example, the following code in Java defines a two-step workflow in which we look up the price of an item, charge a customer's credit card, and then ship an item.
\npublic class ECommerceWorkflow {
public void checkoutWorkflow(WorkflowThread wf) {
// Create some Workflow Variables
var customerId = wf.addVariable(\"customer-id\", VariableType.STR).searchable().required();
var itemId = wf.addVariable(\"item-id\", VariableType.STR).required();
var price = wf.addVariable(\"price\", VariableType.INT);
// Fetch Price and save it into a variable
var priceOutput = wf.execute(\"calculate-price\", itemId);
wf.mutate(price, VariableMutationType.ASSIGN, priceOutput);
// Charge credit card (passing in the output from previous task)
wf.execute(\"charge-credit-card\", customerId, price);
// Ship item
wf.execute(\"ship-item\", customerId, itemId);
}
}
Just by using LittleHorse to define the above workflow, you get reliability, observability, retries, and governance out of the box!
Tasks are the unit of work that can be executed a workflow engine. It's best to think in examples:
\nTask workers are programs that use the LittleHorse SDK, connect to LittleHorse, and execute tasks when the workflow says it's time to do so.
\nTo learn how to write a Task Worker, check out our Task Worker Development Guide.
You can also use External Events or User Tasks to wait for input from a human user or an external system (like a callback or webhook).
\nLastly you need to tell LittleHorse when to run a workflow. You can do it with our CLI (lhctl) but in production you'll need to use the LittleHorse SDK to kick off a workflow. You can do this with our page on Running Workflows using grpc
You'll also need to tell LittleHorse about External Events that happen. You can also do this using lhctl or with our SDK's.
There are many different types of workflow engines, each of which supports different use-cases. For example:
\nHowever, LittleHorse allows you to compose business processes across your software systems. Some use-cases are included below.
\nAll microservice-based applications are inherently distributed systems with the goal of supporting some business process (because no one writes microservices for the sake of writing code, right?). While often necessary, microservices present many challenges to developers due to their distributed nature.
\nOur founder Colt McNealy wrote a detailed blog about how a workflow engine's reliabile state management and oversight can mitigate some of the problems inherent in microservices. Check it out!
\nWorkflows often need to get input from humans:
\nThat's hard to coordinate without a workflow engine. You'd have to build your own state management system that correlates tasks to workflows. LittleHorse User Tasks make this much easier.
\nAI is only useful when you call it at the right time, with the right inputs, and do something with the outputs. That's a workflow. And all sorts of things can go wrong when using LLM's, which is why you need to have a robust workflow engine to provide oversight and exception handling.
\nWhether you are integrating legacy systems that you inherited from the past, or integrating multiple tech stacks accrued through M&A, your customers expect a real-time experience that seamlessly spans all of your systems. Workflow engines are useful for reliably composing actions and moving data across multiple different systems.
\nIf we look at the properties of an API gateway and how they are used, a workflow engine makes sense.
\nThe usage of an API gateway is to have a single layer that abstracts further endpoints.\nIn practice this most often means calling the same API gateway multiple times, receiving the requested data, and doing some date manipulation or calculations at the application layer.\nA workflow engine performs all of the most common actions, and includes things like centralized security, possible data obscurity, failure handling, observability and allows for operators to scale compute.\nAll while still maintaining a central plane that can be shared across an entire orginization.\nAdditionally a workflow engine still allows for the standard CRUD(Create, Read, Update, Delete) operations that an API gateway provides.
", "url": "https://littlehorse.io/blog/basics-of-workflow", "title": "The Basics of Workflow", "summary": "A practical introduction to workflow engines, workflow specifications, and durable execution with LittleHorse.", "date_modified": "2024-09-04T00:00:00.000Z", "author": { "name": "Mitchell Henderson" }, "tags": [ "Orchestration" ] }, { "id": "https://littlehorse.io/blog/microservices-and-workflow", "content_html": "While they are often necessary, microservices are a headache. Fortunately, the right workflow engine (such as LittleHorse) can drastically reduce the difficulty of managing microservices.
\nThis is the third and final part of a 3-part blog series:
If you're just joining for the third blog post, we have so far established that microservices are an effective tool for allowing your engineering team to grow beyond just a handful of people working on an enterprise application. However, microservice systems are by nature Leaderless and Distributed, which yields challenges in:
\nThose challenges inspired me to create LittleHorse in the fall of 2021. LittleHorse provides primitives and guardrails out of the box which make it easier to wrangle with distributed systems and coordinate processes/transactions across multiple microservices.
\nIn this post, we'll discuss:
\nWant to give LittleHorse a try? Get in touch with us!
A workflow is a blueprint that defines a series of tasks to be performed (perhaps conditioned on certain inputs or external events) in order to achieve a business outcome.
\nIf you recall the e-commerce example from the previous blog post, you can think of the abstract checkout process as a workflow. This example is interesting because it demonstrates multiple characteristics of common business processes that make microservice development hard.
\n![\"E-Commerce](\"https://littlehorse.io/assets/images/2024-08-27-complex-checkout-915172cb1dfafc0a3e9a7cc0b042ac3a.png\")
First, a workflow can be mission critical. A customer would be very unhappy if the vendor charged their credit card but failed to ship their order. In technical terms, this means that the state of a workflow needs to be consistent and durable, which is hard to achieve in a distributed system.
\nNext, a workflow can have exceptional cases. Our e-commerce flow has special logic to handle cases when the customer's credit card was invalid or when the ordered item was out of stock.
\nFinally, a workflow can be asynchronous, meaning that it requires waiting for input from the external world in order to complete. For example, our e-commerce workflow sometimes must wait for a customer to update their credit card information before completing.
\nThe mission-critical nature of workflows, combined with asynchronous events and exceptional cases, places a premium on consistency. The results of workflows must be predictable for customers and easy to reason about for business managers and software engineers.
\nA technical or business process does not need to satisfy all three characteristics to be a \"workflow.\" In fact, simple processes with just one or two linear steps can benefit from a workflow engine.
A workflow engine is a software system that makes sure the trains run on time in your processes. To use a workflow engine, you must:
\nWfSpec in LittleHorse) which specifies what tasks to execute and when.![\"LittleHorse](\"https://littlehorse.io/assets/images/2024-08-28-lh-application-8847f083fce679fe41abec88d4375125.png\")
Task Workers are where a workflow can interface with the outside world. Since a Task in a workflow results in the LittleHorse SDK calling a programming function/method of your choosing, Task Workers allow LittleHorse to integrate with any system. Task Workers can make database queries, call external API's, provision infrastructure on AWS, send push notifications to customer mobile apps, perform calculations, call an LLM API, and more.
\nIn LittleHorse, the WfSpec is defined in code in a language of your choice. Because LittleHorse was written with developers in mind, our DSL's have all of the primitives that you'd expect in a programming language: variables, control flow, exception handling, child threads, interrupts, and awaiting for external events. This allows workflows to be:
Once you tell LittleHorse to run an instance of your WfSpec, LittleHorse will oversee the entire process until it completes. Failed tasks will be retried, every step will be journaled, and the state of your processes will be safely and durably persisted while waiting for external triggers.
Microservice applications that are designed as distributed workflows without a workflow engine (like a chain of dominoes falling) present operational challenges because there is no \"leader\" providing oversight over the microservice processes. Thankfully, a developer-focused and horizontally-scalable workflow engine like LittleHorse can fill the \"leader\" role, thus providing oversight and reliability, and taming the complexity of your business processes.
\nAdditionally, using a workflow engine allows you to develop a set of reusable and modular tasks which can be easily dropped into any business workflow with a common API. Rather than accumulating tech debt, workflow engines allow you to accumulate a set of useful lego bricks.
\nIn most existing organizations there's a long list of API calls required to simply run a workflow. Training engineers to use all of the new APIs while securely distributing access and permissions causes confusion and slow development cycles. Workflow engines provide a single API and single system that allows anyone to securely manage, run, and operate complex workflows.
\nMission critical business workflows leave no room for technical failures and outages. However, as we discussed last week, the distributed nature of microservices means that technical failures are not likely but rather certain. LittleHorse provides retries and durable execution capabilities out of the box, removing the need to create complex infrastructure for cross-service transactions (such as dead-letter queues, Outbox tables, and the SAGA pattern).
\nAdditionally, mission-critical processes must be audited and observed in a secure manner with proper access controls. LittleHorse supports this?every step in a workflow is journaled, auditable, and searchable in our dashboard. When humans execute User Tasks, you can view an audit trail of when and to whom it was assigned and executed; you can see when each TaskRun started, completed, and failed (and with what inputs). Our ACL's and Multi-Tenancy capabilities (and \"Masked Data\") ensure that the data remains accessible only to those who must see it.
For microservice developers, handling asynchronous business processes is challenging because it forces you to persist state, correlate events, and wire together callbacks into a non-linear flow. Developers often need to create database tables for ongoing transactions and maintain complex flow diagrams showing how different services integrate with business events.
\nHowever, LittleHorse provides two primitives to simplify this process:
\nTogether, User Tasks and External Events allow developers to transform complex asynchronous flows (such as our e-commerce example when we wait for a customer to provide a new credit card) into a more manageable linear flow.
\nFinally, just as processes can fail at the technical level, they can also fail at the business level. As per our ongoing e-commerce example, cards can run out of funds, items go out of stock, customers can cancel orders while they are being processed.
\nHandling any given exceptional case in a business workflow might involve actions in several different microservices. Without a workflow engine, therefore, each exceptional case results in more and more complex interdependencies in your microservices, creating the notoriously feared \"Distributed Monolith.\"
\nIn contrast, with LittleHorse as your workflow composer, the dependencies between microservices are mitigated and workflow concepts such as Failure Handling allow you to easily define rollbacks, SAGA patterns, and edge cases without introducing further accidental complexity into your microservices. This allows startups and enterprises alike to implement robust, enterprise-grade business applications without accumulating costly technical debt.
\nFor a variety of reasons, startups and enterprises alike may need to work with microservices despite the challenges they bring. Thankfully, workflow engines like LittleHorse can mitigate those problems by providing oversight into your entire process.
\nAt the LittleHorse Council, we are very excited about the upcoming 1.0 release. Over the next few weeks, we will:
\nAnd if you enjoyed this post, give us a star on GitHub and try out our quickstarts to get going with LittleHorse in under 5 minutes.
", "url": "https://littlehorse.io/blog/microservices-and-workflow", "title": "Microservices and Workflow: A Match Made in Heaven", "summary": "How workflow engines like LittleHorse reduce the complexity of orchestrating microservices.", "date_modified": "2024-09-02T00:00:00.000Z", "author": { "name": "Colt McNealy" }, "tags": [ "Microservices and Workflow", "Microservices", "Orchestration" ] }, { "id": "https://littlehorse.io/blog/challenge-of-microservices", "content_html": "Microservices are often necessary, but unfortunately they bring with them some baggage.
\nThis is the second part of a 3-part blog series:
Last week, I blogged about the problems that microservices solve, and why they are not only beneficial but necessary in some cases (a good bellwether is the size of your engineering team: beyond 1 or 2 dozen engineers, you will probably start to feel some problems that can be solved with microservices).
\nWhen done correctly, microservices remove several bottlenecks to scaling your business. However, even well-architected microservices bring significant accidental complexity.
\nIn particular, microservices are:
\nIn this article we will explore how the above problems arise from two key facts:
\nMicroservices bring with them additional challenges around operationalization and deployment. However, those challenges are out-of-scope for this blog post as we instead choose to focus on the challenges faced by application development teams rather than operations teams.
As I described in last week's blog:
\n\n\nThe term \"microservices\" refers to a software architecture wherein an enterprise application comprises a collection of small, loosely coupled, and independently deployable services (these small services are called \"microservices\" in contrast to larger monoliths). Each microservice focuses on a specific business capability and communicates with other services over a network, typically through API's, streaming platforms, or message queues.
\n
Crucially, a single microservice implements technical logic for a specific domain, or bounded context, within the larger company. In contrast, a comprehensive business process requires interacting with technology and people across many business domains. The classic example of microservices architecture, e-commerce checkout, involves at least shipping, billing, notifications, inventory, and orders.
\nIn the rest of this blog post we will examine microservices through the the lense of e-commerce checkout flow. To start with a simple use-case, the logical flow we will consider is:
\norders service.inventory service.payments service.shipping service.notifications service notifies the customer that the parcel is on its way.![\"Simple](\"https://littlehorse.io/assets/images/2024-08-27-simple-checkout-301d7115bb5b77f34c84342998c2921d.png\")
Recall that each service (in the workflow diagram above, each box) is its own deployable artifact. That means that the happy-path business process described above will involve five different software systems from start-to-finish.
\nIn the above workflow diagram, each arrow can be accurately interpreted in two ways:
\nGuess what! This means we have a distributed system by definition. As Splunk writes in a blog post:
\n\n\nA distributed system is simply any environment where multiple computers or devices are working on a variety of tasks and components, all spread across a network.
\n
You need to look no further than the Fallacies of Distributed Computing (written by Sun Microsystems Fellow L. Peter Deutsch in 1994) to see that this means that microservices are no easy task.
\nAs we've seen already, any microservice-based application is a distributed system. Some distributed systems have the concept of a leader, which is a special node in the system that has special responsibilities.
\nApache Kafka is my favorite distributed system. In Apache Kafka, the Controller is a special Kafka server that is responsible for deciding which partition replicas are hosted on (and led by) which brokers. If the broker who was in charge of a partition goes down, then the Controller chooses a new broker from the ISR to take its place.
Therefore, the Controller in Apache Kafka can be thought of as a leader.
While systems like Apache Kafka have clear leaders (for example, the Controller may re-assign partition leadership if the cluster becomes too imbalanced), in a microservice-based system there is no central leader to ensure that the chips fall correctly. This is by necessity, because the separation of development concerns and lifecycles across microservices means that microservices cannot and do not have leaders.
\nYou can think of our e-commerce microservice flow as a line of dominoes falling. Once the process starts, no one entity is responsible for ensuring its completion. The business workflow moves from orders to inventory to payments and so on. If payments fails for some reason (perhaps a network outage makes the Stripe API unavailable), then it's quite possible that the shipping service never finds out about the workflow.
However, in real life such outcomes are not acceptable. This means that every single player in the system must:
\nImplementing the above slows down development, more tightly couples one services to another, increases dependencies, and makes your microservice architecture much more heavyweight.
\nSo far, we have established that there are many players involved in a business process, yet there's no one composer involved in ensuring that an ordered item is delievered to the the correct address. This yields three problems:
\nProcessing orders is a mission-critical use-case. This means that orders should always complete and never be dropped (for example, we should not charge the customer's credit card and not ship the product to them).
\nHowever, asynchronous processing such as that which I outlined above is prone to failures. For example, if you chain microservices together with direct RPC calls, a single network partition can cause an order to get stuck. Even with a reliable message broker such as Apache Kafka or AWS SQS sitting between your microservices, a write to the message broker could fail after the payment went through, still resulting in a stuck order.
\nJust as communication between microservices can fail, the actions performed _by_each microservice can also fail. In many cases actions performed by a microservice depend upon failure-prone external systems and API's. If the Stripe API is down, or if the credit card is invalid, we can't just stop processing the order there! We must notify the customer of what went wrong and also release the inventory that we reserved.
\nThis means that microservice developers spend countless hours building out infrastructure to support:
\nBack to the domino analogy, if one domino misses the next, the entire chain just stops.
\nThe second problem with microservices is that once a process instance has started (i.e. the dominoes are falling), it is very difficult to observe what happens between steps 2 through 10. This means that multi-step processes with performance issues are hard to optimize, as there are many microservices which could be the bottleneck and it's hard to know which. Even worse, when a customer complains about a \"stuck order,\" it is difficult to find the point of failure.
\nAs a result, microservice engineers spend time and money:
\norder) in a DB just for visibility purposes at every stepLastly, because microservices are leaderless, each player in the end-to-end process must have hard-coded integrations with the preceding and subsequent steps. This results in:
\nMicroservices come with the promise of loose coupling; however, the unfortunate reality is that this is often not the case. As a result, teams often do have to coordinate with each other during deployments.
\nTo see an example of the complexity introduced by coupling of microservices, let's consider what happens to our e-commerce checkout workflow when we add a few edge cases to make it more realistic:
\n
In the above diagram, each arrow represents the flow of the business process and information. Each microservice must have custom logic which sends information to the right place. In essence, while we intended to have modular microservices that understand only their own Bounded Context, what we have is tightly-coupled systems which must understand pretty much the entire business workflow.
\nTherefore, when business requirements change, unrelated microservices end up having to change their internal implementation as well.
\nMicroservices have clear and proven benefits, and are often not just advantageous but necessary in some cases. However, as we discussed today, those benefits do not come without a cost. Because microservices are inherently distributed systems, challenges such as reliability, observability, and coordination are exacerbated.
\nWithout spoiling the punchline of the next blog post, these challenges are why I started LittleHorse almost three years ago. Stay tuned for a description of how a workflow composer can alleviate a good portion of the headaches that come along with microservices.
\nAstute readers may notice that when discussing the e-commerce checkout example, we didn't discuss the problem of analytics. We focused exclusively on online transaction processing, or ensuring that the orders are properly fulfilled and processed. However, no attention was paid to business analytics to optimize future sales!
\nThis area is yet another challenge. The LittleHorse Council is working on a major feature (an output Kafka Topic with records for anytime something interesting happens inside a WfRun) for the LittleHorse Server that will address this. Don't worry, we'll blog about it soon 😉.
If microservices add so much complexity, why bother with the hassle?
\nThis is the first part of a 3-part blog series:
We've all heard of microservices, but unless you've read copious amounts of Sam Newman and Adam Bellemare's writings, you might be wondering whether, when, and why you should adopt them. In this blog post, we will examine the halcyon land promised by microservices.
\nMicroservices have been deployed widely across many large enterprises, most notably Netflix, Uber, Shopify, PayPal, and others. As we will discover throughout this blog series, a microservice architecture is mandatory once you reach a certain size of company, and it's probably overkill for a 12-person startup. The gray area inbetween is the interesting part!
\nThe term \"microservices\" refers to a software architecture wherein an enterprise application comprises a collection of small, loosely coupled, and independently deployable services (these small services are called \"microservices\" in contrast to larger monoliths). Each microservice focuses on a specific business capability and communicates with other services over a network, typically through API's, streaming platforms, or message queues.
\nIn practice, this means that a user interaction with an application (such as placing an order) might trigger actions that occur in many small, independently-deployed software systems, such as:
\nFrom the user (client) perspective, one request is made (generally through a Load Balancer, API Gateway, or Ingress Controller) but that request may ping-pong between multiple back-end services and may also result in future actions being scheduled asynchronously:
\n![\"Microservices](\"https://littlehorse.io/assets/images/2024-08-22-microservices-arch-2389e95f878235ea53f0c0afa2eea967.png\")
In contrast to microservices, a monolithic architecture would serve the entire \"place order\" request on a single deployable artifact:
\n![\"Monolithic](\"https://littlehorse.io/assets/images/2024-08-22-monolith-arch-894351f807fd64060d9d05380fc799cf.png\")
In Domain Driven Design, accidental complexity refers to the unintentional complexity that you introduced to your architecture (deployments, service interactions, third-party dependencies, etc.). Rule #1 of maintaining software systems is to avoid introducing accidental complexity as much as possible.
\nSimply by looking at the visuals above, microservices add a significant dose of accidental complexity to your architecture (more on this in next week's post!). Given that, what benefits would make up for the extra complexity introduced by microservices?
\nI would be first to admit that microservices bring with them a series of headaches around cost, observability, maintenance, and ease of evolution (otherwise, I would not have founded LittleHorse Enterprises!). However, microservice architecture plays a vital role in addressing two critical trends reshaping the software development landscape today:
\nThe level of digitization expected of businesses in order to compete in the modern market has drastically increased: IT teams must build software that interfaces with an ever-expanding list of external API's, legacy systems, user interfaces, internal tools, and SaaS providers.
\nFor example: in the early 2000's, it was perfectly acceptable (even expected) for a passenger to book airline tickets over the telephone or through a travel agency. However, such an experience would be unheard of today and would immediately hobble an airline who provided such poor digital services.
\nIn addition to using automation to provide better customer services, companies are generating, processing, and analyzing massive amounts of data. For example, grocery stores with razor-thin margins analyze seasonal consumption patterns in order to optimize inventory and prevent costly food waste.
\nThese trends have coincided with (or caused, I would argue) a proliferation in the number of 1) software developers, and 2) software tools and API's found within companies in all industries, leading to two new problems:
\nAs the importance and quantity of digital software systems exploded over the last two decades, so has the availability of nearly-infinite compute power delivered through cloud infrastructure providers such as AWS.
\nThe promise of elasticity, or the ability to quickly spin compute resources up or down according to load and only pay for what you use, is unique to the cloud: for on-prem datacenters, spinning up new compute means buying new machines from Sun Microsystems (hopefully not Microsoft!), and scaling down compute means trying to sell them off on the secondary market. (Ask my father about how that went for a lot of people in 2001.)
\nBeyond scaling up and down, elasticity enables different deployment patterns that did not exist before. Whereas pre-cloud enterprises had dedicated and centralized data-center teams who were in charge of running applications, the accessibility of cloud computing gave rise to the DevOps movement. This has empowered smaller teams of software developers to take on the task of transferring software from \"it works on my laptop!\" to \"it's now deployed in production!\"
\nDespite the extra complexity it brings, the microservice architecture can more than pay for itself by ensuring organizational alignment and allowing enterprise architectures to take full advantage of the cloud's elasticity.
\nAs discussed earlier, the business problems that software engineering organizations must solve today dwarf those that were solved in the 1990's, and so do the software engineering teams that tackle those problems.
\nI am not belittling the engineers of the 90's; the problems they solved were arguably much harder than the problems we face today, and there were fewer engineers to face those problems. However, it is a fact that users expect more digital-native experiences today than they did twenty years ago.
By breaking applications into smaller services, we can accomplish several important things:
\nEngineering teams of over a few dozen engineers working on the same deployable piece of software is a recipe for inefficiency. Merge conflicts, arguments over tech stack, slow \"release trains,\" and excessive intra-team coordination are just a few problems that arise. However, by breaking your application into smaller microservices, you can also break up your engineering organization into smaller, more efficient teams each in charge of a small number (prefably one!) of microservices.
\nAs an added benefit, properly-designed microservice architectures can follow the principles of Domain Driven Design. Ideally, a single microservice corresponds to a Bounded Context inside the business. This enables a small piece of the technical platform (a microservice) to be managed by a small team of software engineers, who collaborate closely with subject-matter experts and business stakeholders within a very specific domain of the business. Such close collaboration can foster better alignment between business goals and the software produced by engineering teams.
\nMicroservices can allow developers to move faster by enabling continuous delivery and independent deployment of services. In a monolithic architecture, releasing a new feature or fixing a bug typically requires redeploying the entire application. Since microservices allow smaller pieces of your application to be deployed independently, engineering teams can iterate faster and deliver incremental value to business stakeholders.
\nThese positive effects are amplified by the advent of cloud computing. Since deploying a new application no longer requires buying a physical machine and plugging it into your datacenter but rather just applying a new Deployment and Service on a Kubernetes cluster, it is now truly feasible for small teams of software engineers to own their application stack from laptop-to-production (obviously, within the guardrails set by the central platform team). Furthermore, cloud computing is a pay-as-you-go (and often even pay-for-what-you-use) expense rather than an up-front cost. Therefore, the dollar cost of infrastructure required to support microservices is much lower today than it would have been before the advent of cloud computing and kubernetes.
The microservice architecture is not just a Twitter-driven buzzword but rather a way of designing system that has several real advantages. For most organizations with over two dozen software engineers, building applications with microservices is not an option but rather a necessity. However, those advantages come with a cost.
\nWe will discuss those challenges in next week's blog post...in the meantime, though, join our Community Slack to get the latest updates!
", "url": "https://littlehorse.io/blog/promise-of-microservices", "title": "The Promise of Microservices", "summary": "Why teams adopt microservices, where they work well, and what tradeoffs come with the architecture.", "date_modified": "2024-08-22T00:00:00.000Z", "author": { "name": "Colt McNealy" }, "tags": [ "Microservices and Workflow", "Microservices" ] }, { "id": "https://littlehorse.io/blog/helm-and-k8s-operators", "content_html": "About Helm vs Kubernetes Operators.
\nHelm is like brew or npm for Kubernetes. There are repositories containing charts, and each chart allows you to install an application into your K8s cluster.
Under the hood, Helm works by filling out some templated Kubernetes yaml files with user-provided values, drastically reducing boilerplate and allowing you to deploy a reasonably complex application without the user of the helm chart having to understand too much about how to manage such an application.
\nIn addition to that template-engine functionality, Helm also manages versions of your application. You can use Helm to release a new version of your app (for example, updating the docker image tag) and then quickly roll back to a previous version if you discover a bug. This is great for teams deploying stateless applications such as microservices or LittleHorse Task Workers.
\nFirst, it is quite simple to write a Helm chart. This means that most DevOps teams can quickly write a helm chart that can be used by microservice teams across their organization.
\nSecondly, Helm is a client library (well, it has been since the removal of Tiller...but that's another blog post). Therefore, you don't need to run any privileged pods inside the K8s cluster; all you need is a CI server with permission to create the necessary K8s resources.
\nUnfortunately, Helm doesn't do much beyond initial installation and upgrades. Monitoring, self-healing, autoscaling, certificate rotation, and management of non-Kubernetes resources (eg. Kafka Topics, LittleHorse Task Definitions, AWS LoadBalancers, etc) are some exercises left to the reader, to name just a few.
\nKubernetes Operators are a pattern introduced by Red Hat that intends to capture the knowledge of an expert Site Reliability Engineer (or, more punnily, a software operator) into a program that manages (or operates) a complex application.
\nTo accomplish this, a Kubernetes Operator extends the Kubernetes API to introduce a new resource type that is custom-made for your application. The Operator works in tandem with Kubernetes itself to manage applications of a specific type.
\nA Kubernetes Operator has two components:
\nCustomResourceDefinition, which defines the extension to the Kubernetes API (including relevant configurations for your application type).The CustomResourceDefinition can be over-simplified as an Open API (not Open AI) specification for how your custom resource will look. For example, in LittleHorse Platform, the simplest version of a LHCluster resource (which creates a, you guessed it, LittleHorse Cluster) is:
apiVersion: littlehorse.io/v1
kind: LHCluster
metadata:
name: hello-littlehorse
namespace: lh
spec:
server:
version: \"0.2.0\"
listeners:
- name: internal-k8s
type: internal
port: 2023
replicas: 3
storage:
volumeSize: \"10G\"
kafka:
strimziClusterRef:
clusterName: my-strimzi-kafka-cluster
listenerPort: 9093
The CustomResourceDefinition allows you to kubectl apply -f <that file up there>, and then you can kubectl get lhclusters:
Now how does the LittleHorse cluster get created, configured, managed, and monitored? That's where the Controller comes into play. In the Operator pattern, a Controller is a process (normally, it runs as a Pod in a cluster) that watches for all events related to a CustomResourceDefinition and manipulates the external world to match what the Custom Resources specify.
Generally, that means creating a bunch of Kubernetes Services, Deployments, etc. to spin up an instance of an application. For example, the Strimzi Kafka Operator watches Kafka resources and deploys an actual Kafka cluster.
However, a Controller can also manage non-kuberentes resources. For example, many Ingress controllers provision or configure physical load balancers outside of the Kubernetes cluster. As another great example, the Strimzi Kafka Topic Operator watches for KafkaTopic resources and creates (you guessed it) Kafka Topics using the Kafa Admin API.
We at LittleHorse plan to add similar CRD's that are specific to LittleHorse...stay tuned to learn about the LHTaskDef and LHPrincipal CRD's 😉.
Kubernetes Operators are beautiful. Since they were developed by Red Hat, they (along with Strimzi) are the biggest reason why Red Hat is in my top-three favorite software companies of all time.
\nA well-written operator can make it a breeze to manage even the most daunting applications. Since the Controller is code written in a general-purpose language (normally Go or Java), an Operator can do just about anything that can be automated by an SRE. This includes:
\nThe biggest downside to Operators in Kubernetes is that writing a Controller is hard. Additionally, it requires running a Pod with special privileges that allow the Pod to create other K8s resources. Because of this, writing an Operator for something like standardizing your team's blueprints for deploying a microservice just doesn't make sense.
Future blogs will dive into some of the challenges that we had to overcome with LittleHorse Platform, and how we minimized the permissions that our Operator needs to provide a self-driving LittleHorse experience to our customers.
\nWell, I'm a software engineer, so I'm going to say \"it depends.\" However, Kafka legend Gwen Shapira said in a fantastic podcast that some \"it depends\" are more helpful than others. So in an effort to fall in the \"more helpful\" side:
\nLittleHorse Platform is an enterprise-ready distribution of LittleHorse that runs in your own Kubernetes environment. We believe that Helm is fantastic for deploying many stateless applications, and even some stateful applications. However, Helm wouldn't let us go far enough towards providing our customers with a fully self-driving LittleHorse experience. As such, we chose to put in the extra work and build a full Kubernetes Operator. Stay tuned for an extensive list of current and upcoming LittleHorse Platform features, all powered by the Java Operator SDK.
\nTo inquire about LittleHorse Platform, contact sales@littlehorse.io. To get started with LittleHorse Community (free for production use under the SSPL), check out our Installation Docs.