What is Helix?
The mandate for AI dominance in the National Security domain is urgent, but the financial and operational costs of replacing legacy software are prohibitive. Recognizing the need for immediate operational readiness, Two Six Technologies developed Helix, an agentic AI orchestrator that coordinates specialized agents to break down complex goals and manage tasks for the Department of War and Intelligence Community.
Formally launched in May 2026, Helix enables chat-forward user experiences that feel as simple and intuitive as ChatGPT, while bringing agentic AI capabilities directly to existing software and data in complex, security-first environments. Some of its key features include synthesizing data across disparate sources, employing custom RAG knowledge bases, and providing a modular architecture that prevents vendor lock-in. Helix has already been deployed in various intelligence organizations where it has significantly accelerated decision-making, being able to act on data in minutes, rather than weeks or even months.
Problem Statement
When Helix receives a user prompt, it orchestrates a fleet of built-in remote agents, combines their outputs, and produces a response with citations for groundedness. The challenge is knowing whether that response is truly reliable. Did the agents choose the right tools and delegation path, and did those citations actually support the claims being made? Even when the answer looks good on the surface, did performance drift to higher latency or increased token usage?
The Helix team identified a clear need for explainability, visibility, and measurability in model behavior. The core challenge is not whether one response appears correct, but whether performance degrades under changing conditions and whether that degradation is detected before users are affected. Pass/fail checks alone cannot answer that question. Because LLM behavior is non-deterministic, a run may pass while still drifting in latency, token usage, or cost. For this reason, Helix needs both system checks and broader quality metrics to track performance over time.
Our Vision
We aim to make evaluation continuous and automatic. With every major code push into Helix, tests run automatically to produce signals that demonstrate and measure if behavior remains reliable over time, not just whether it succeeded once. Our evaluation framework is designed to assess performance of a system on both low and high side systems, where the orchestrator will have no advance knowledge of the tools and data sources that will be available at time of test.
How the Issue was Addressed
Multiple evaluation frameworks have been developed at Two Six. One framework is a test harness for Sentr that provides a trace-and-check model: each run captures prompts, responses, and tool behavior as a trace, then applies checks for behaviors such as tool use and remote-agent delegation, with optional LLM-judge scoring for semantic quality.
A second framework is an evaluation suite for Helix, designed for domain-specific benchmarking across LLMs, workflow settings, and prompt strategies. Instead of only reporting rule-based pass/fail outcomes, it generates comparative evidence across configurations by collecting outputs, scoring answer-quality signals, and estimating token and cost behavior per model.
To achieve this, we build on existing evaluation work and adapt it to Helix’s operational context. We take the original evaluation design ideas that start in the Sentr test harness, including trace-first evaluation, case-based orchestration, and semantic grading, and shape them around data-agnostic Helix use cases, providing customers with the ability to assess their high side deployment systems with the same rigor we apply to SaaS products hosted in the commercial cloud. This harness is then integrated into the Helix evaluation suite, which allows us to extend the framework with additional telemetry.
When a check run starts, the harness executes selected test cases against a live Helix workflow and records the full interaction context needed for grading. It then applies the configured checks to each case attempt and returns clear pass/fail outcomes, plus reasons for failures. Each test run outputs not just binary verdicts, but also a broader quality picture across performance metrics.
The outputs are both decision-friendly and analysis-friendly. At the per-case level, you get pass/fail status and failure explanations. At the run level, you get aggregated metrics such as answer rate, latency, tool-call behavior, citation-related signals, and token/cost summaries, including judge usage when semantic grading is enabled. This gives a practical view of whether the system passed checks and how it performed operationally.
The result is a framework that provides both regression tests for Helix to ensure its services are working as intended, and telemetry to analyze its performance over time. Instead of asking whether one response looked good, it asks whether behavior is stable across cases, whether checks are meaningful, whether metrics are complete, and whether the evaluation itself is trustworthy. With this, we have added operational confidence to Helix.
CI/CD and Live Monitoring
Our work establishes a strong reliability baseline for Helix, but the broader objective is continuous confidence as the system evolves. Because Helix is used in a high-stakes domain, even infrequent failures can carry outsized impact. As a result, our evaluation strategy prioritizes detecting and resolving issues before deployment. We are expanding the evaluation framework by adding new checks and metrics so evaluation coverage keeps pace with changing workflows and newly observed failure modes. In parallel, these signals will be integrated into CI/CD so checks run automatically on code changes instead of relying on manual validation. We are also operationalizing live monitoring of cost, latency, token usage, and tool behavior so drift can be detected and investigated more efficiently. Langfuse provides an immediate path for trend visibility, while custom metrics can be routed into platforms such as Kibana for more flexible dashboards and deeper analysis. Every Helix change should ship with measurable evidence that behavior remains correct, efficient, and trustworthy in production conditions.
