The Auditable AI Ecosystem

From data anomalies to agent decisions: one research-to-product stack.

This ecosystem covers auditing AI across its full lifecycle: a research method, an over-privilege benchmark (FORTIS), an open-source tool, and a curated knowledge base, each built to cite and reinforce the others. It underpins Auditable AI, the company building the auditable layer for AI agents.

The auditable Stack

Three open artifacts that form the agent-layer arm of the ecosystem. Each serves a distinct role; together they give the research community a method, a benchmark (FORTIS), and a production-grade tool.

Tool
auditable is an open-source system of record for AI-agent decisions: it captures what each decision relied on, replays it against live state, and rolls back the committed action when it no longer holds. Framework-agnostic, dependency-free core; pip install auditable.
GitHub  |  PyPI  |  Docs
Method
The research engine the tool is built on: a typed two-layer graph of agent execution and dependency, used to detect run failure and localize the faulting step in multi-agent runs.
Resource
A curated map of papers, tools, datasets, and standards for reliable, auditable AI agents. The Knowledge cell of the Auditable Matrix: 0 hallucinations across 69 arXiv entries, citation-verified.

Auditing Across the Agent Lifecycle

Auditable Agents is the framework that organizes what agent auditability requires: five dimensions and three mechanism classes (detect, enforce, recover). The research below instantiates them at each point in an agent's life.

PRE · Before Deploy
FORTIS: a benchmark for over-privilege in agent skills.
agent-audit: a static scan of agent code for security and over-privilege issues.
LIVE · While Running
Cross-User Contamination: one user's data leaking into another's in shared-state agents.
Topology Matters: measuring memory leakage in multi-agent LLMs.
Aegis: a pre-execution firewall that gates each tool call.
POST · After a Run
Implicit Execution Tracing: recovering which agent caused a harmful result after the logs are gone.
GRADE: localizing the faulting step in a completed multi-agent run.

The Three-Layer Research Stack

The agent-layer work above rests on two layers that came before. All audit eventually grounds in distinguishing in-distribution behavior from out-of-distribution behavior; that measurement backbone gets re-used at higher layers of the stack.

  1. Agent Layer: Auditability and Control
    Auditability frameworks, runtime control surfaces, and agent-specific failure modes (over-privilege, cross-user contamination). The current frontier is turning this research into deployable open-source infrastructure that audits agents from before deployment to after a run. See the PRE / LIVE / POST grid above.
  2. Foundation-Model Layer: Trust and Robustness
    Behavior under attack, hallucination of fact and reasoning, and adversarial robustness of retrieval, all of which propagate downstream once the model is used by an agent or a user-facing system. Work covers jailbreak detection for vision-language models, causal analysis of hallucination, query-agnostic attacks on RAG, and LLM-as-anomaly-detector benchmarks.
  3. Data Layer: Anomaly and Out-of-Distribution Detection
    The methodological root, begun during my PhD with the PyOD ecosystem. PyOD is the canonical open-source anomaly-detection library: 50M+ downloads, named by OpenAI, used in Amazon and European Space Agency anomaly-detection work, and listed by the U.S. DoD CDAO. Now in its third major release, it provides the measurement backbone that the higher layers re-use.

Get Involved

The ecosystem is open: every artifact is public, citable, and built to be extended. If you are building on any part of this stack, the best starting points are the auditable tool page for the SDK, GRADE (arXiv:2606.22741) for the research method, and awesome-auditable-ai for a map of the broader literature.

For the company building on top of this stack, visit Auditable AI (auditable.run). All open-source work lives under github.com/yzhao062.