October 21, 2026

How we built the Langfuse Assistant

How we built the Langfuse Assistant on our existing MCP server, why the agent runs outside its sandbox, and how we handle background runs.

Lotte

Ben

In June, we launched Halo, the Langfuse Assistant. You can ask it to interpret your traces, build a dashboard or dataset, or help set up evals without leaving the Langfuse UI.

It's somewhat ironic: as a company focused on helping others build agents, we didn't have an in-product agent of our own. We've invested in building the infrastructure and tools for agent integration, such as the Langfuse skill, MCP server, and CLI, for using Langfuse with agents like Claude Code or Cursor. Now we wanted to bring those capabilities into Langfuse itself.

When we first discussed whether we should write this post, I wasn't sure there was much to write about. The first version was simply a Mastra agent, an MCP server, and some code to glue them together. We didn't invent some super-efficient algorithm for agent execution, and we didn't rewrite anything in Rust (although there might be some other posts here coming soon). Building the in-product agent was mostly about integrating existing pieces and making them work smoothly together while laying the foundations for future agentic features.

From the start, we knew we wanted to reuse as much as possible and share the tools and instructions behind Halo with people using Langfuse through Claude Code or Codex. We didn't want a second set of things to maintain just for the in-product agent. If we improved a tool or fixed some guidance for Halo, that should help people using their own agents too. That meant building on our existing MCP server as well as bringing the shared Langfuse skill into the Assistant.

Langfuse back-end

• API
• MCP
• CLI
• Docs
• Skill

Langfuse UI

Halo

Coding agents

How the Assistant is put together

The first step of building Halo was expanding the MCP server. The May 29 release added tools for observations, metrics, scores, comments, datasets, and annotation queues, making most Langfuse project data available to agents. We needed these tools for Halo, but they were also useful to anyone connecting their own agent.

In June, we launched the first beta version of the Assistant: a Mastra agent that ran behind an HTTP endpoint, streaming responses to the browser. It handled one conversation at a time, with no parallel work or background execution. In early July, we gave it write access with human-in-the-loop approvals, and later that month added a sandbox. We added background execution in August. We announced all these capabilities together in September.

Here's the high-level architecture that we ended up with:

Users can ask Halo to investigate something, close the tab, and come back to its progress and results. The worker keeps going independently of the browser, while the saved event log lets the UI catch up when you return. MCP tools let Halo work with the user's project data. The optional sandbox lets Halo process larger results with scripts instead of reading everything into the model's context. Neither gives it more access than you have, and actions that change data still need approval.

Reusing the MCP server and skill

Halo uses our public MCP servers, the same ones you can connect to. Beyond MCP, Halo gets sandbox tools similar to those in coding agents, plus UI-specific tools such as proposing navigation to another page. All customer-data access goes through MCP. We also give Halo the user's name, time zone, browser languages, and current URL through the system prompt. By relying on MCP and tools, we can control which tools the Assistant gets to call without having to secure arbitrary network requests.

For each run, the worker creates a temporary API key scoped to the project. By default, it can only call read-only MCP tools. We maintain our own mapping of tool permissions in the in-app agent's code rather than trusting the MCP server's read-only annotations. We also filter tools by the signed-in user's role before exposing them to the model. If the user's role doesn't allow an action, we don't expose the corresponding tool to the agent at all.

Tools that change data require approval. Choosing Always approve saves the approval for that conversation only. We still recheck it against the user's current role on each run, so an approval can't give the agent permissions the user doesn't have.

If Halo needs another way to access project data, we have to add an MCP tool first. This might seem restrictive at first, but I actually like that constraint, because the same tool is then available to people using their own agents.

We do the same with instructions. Halo's system prompt explains how to handle approvals, link to Langfuse pages, and keep answers short. For guidance on using Langfuse, such as setting up evals or migrating prompts, we load the open-source Langfuse skill as a Mastra skill and rely on the Docs MCP. You can also use both of these with external agents like Claude Code or Cursor.

Using Langfuse for the Langfuse Assistant

Of course, we're using Langfuse itself to observe and analyze the behavior of the Assistant.

When an organization opts into AI feature telemetry, we can use traces of Assistant runs to see where it struggles. When we see Halo struggle with a tool or give poor guidance, we fix the MCP tool or shared skill, or improve our documentation. This way, improvements driven by Halo's interactions benefit all users.

Some of those fixes can be very straightforward. Agents kept asking to sort metrics by totalCost_sum when the tool expected sum_totalCost, so we made the MCP adapter accept both. When dashboard creation failed validation, the error told the agent which field was wrong but not what it could use instead. We added the allowed values and guidance on looking up the schema, so it had enough information to correct the call.

People have been trying to break or exploit Halo since we announced it, and those attempts still turn up regularly. Security shaped the design of both the agent and the sandbox from the start. These attempts reinforce why we keep strict boundaries around what the agent can access and what sandbox scripts can do. Some of those boundaries limit functionality, but they help keep the system secure and protect user data.

The agent does not run in a sandbox

We added the sandbox because we knew some tasks could be handled more efficiently by writing scripts for processing data than by putting raw tool outputs into the model's context. But giving the agent bash and file system tools didn't mean the agent itself had to move into a sandbox.

For interacting with the sandbox, we give the agent four tools: read, write, edit, and bash. When the agent first uses one of these tools in a conversation, we create a new sandbox instance that is limited to that conversation. The sandbox instance is isolated and ephemeral, with no network egress configured by default. Code the agent writes is executed as an unprivileged user within the sandbox. After some period of inactivity, the sandbox is suspended and eventually terminated and cleaned up.

From within the sandbox, no process can make additional requests to Langfuse or the wider internet directly. The agent can only push data into the sandbox after fetching it through authorized MCP calls. Keeping data access in MCP meant we could add the sandbox without also having to secure against arbitrary network calls made from within it.

Many conversations don't need a sandbox at all. Answering a docs question or looking up yesterday's traces might only take a few MCP-based lookups. Starting a VM for those conversations would consume memory and disk without adding value.

There's also self-hosting. Langfuse is proudly open-source, and we have lots of users self-hosting it. For Langfuse Cloud, we use AWS Lambda MicroVMs for the sandbox, but the sandbox interface the agent interacts with is built to be adaptable to different providers. Ultimately though, self-hosters can also choose to not use a sandbox at all. Without one, Halo still works.

Processing tool results with code

Halo sometimes needs to inspect hundreds of observations containing large JSON structures. We needed a way to process that data without blowing up its context window.

Halo can scan through large amounts of data using the sandbox and tools like grep, head, jq, or a short Python script. Say you ask it to group the errors in a few hundred failed observations. It doesn't need to read every field of every observation to do that. It can inspect the files, group the errors, and read only the output it needs. To allow for this, we add an optional silent parameter to every MCP tool when a sandbox is available. This allows the agent to fetch data without putting the full result into the model's context:

// Simplified: when a sandbox is available, every MCP tool gets an optional `silent` flag.
tool.execute = async ({ silent, ...input }, context) => {
  const result = await execute(input, context);
  persistToolCall({ toolName, toolCallId, result }); // stored with the run

  if (!silent || isError(result)) return result; // failures are never silenced
  return `Output available at /workspace/tool_calls/${toolName}_${toolCallId}.json`;
};

With silent: true, the model gets a file path instead of the full result. Before the next sandbox operation, we then write the saved calls into the sandbox as JSON files containing the request and response.

When looking through runs for this post to find examples, I saw Halo use short Python scripts to count observations, combine results from several files, and count errors by environment or model. Sometimes it just printed part of a file to see what was there. The agent decides what to inspect rather than having to read the whole response.

Background runs

In the initial implementation of Halo, the agent loop ran in an HTTP handler. If a user closed the browser, the network request would be interrupted and the agent would stop processing actions. To enable longer-running operations as well as headless invocation, we needed a way to decouple the agent loop from the HTTP request lifecycle.

Langfuse already had separate web and worker processes, so we moved the agent loop from the HTTP handler to the worker. The web server saves new runs to Postgres and queues them through BullMQ. The worker executes each run and saves its progress and results to Postgres. While a user watches a conversation, the web server polls Postgres for new events and streams them to the browser over Server-Sent Events. This lets the worker keep going when the user closes the tab, and the browser catch up on saved events when they return.

When an approval step is presented to the user, the run completes with a pending approval status. When the user approves or denies a proposed step, it creates a new run that continues from the saved conversation in Postgres.

We decided not to automatically retry failed runs. Most of Halo's runs are currently short, so we chose to keep run management simple rather than add automatic recovery. The trade-off is that a failed run stays failed instead of being retried automatically. If your runs take hours, cost a lot, or have side effects you can't check afterward, I'd probably make a different choice and use a durable workflow engine. Should we see similar requirements in the future, we might revisit this decision.

What's next

We're exploring giving sandbox scripts access to the Langfuse SDK and the project's configured LLM connections. The SDK would unlock new capabilities for Halo such as evaluator calibration: testing an evaluator against human-labeled examples, changing its prompt, and checking whether the results improve. We'd need to restrict scripts to the requests the user is allowed to make and has approved. Given security concerns, this is non-trivial and we're still exploring how to do that.

The Assistant is live on Langfuse Cloud and in public beta for self-hosted deployments. If one of these decisions gets in your way, tell us in the feedback discussion.

If you like working on this kind of problem, we are hiring.