The Auditable AI Ecosystem

Open-source infrastructure for auditing AI agents, from before deployment to after a run.

Eleven open artifacts, each answering one question an auditor asks at one point in an agent's life. The record they produce has two uses: it shows what a system did wrong, and it shows what it never needed to do. Every one is public and citable, and the tools install from PyPI.

This page covers the auditing-of-action arm of the lab's work and says where each piece applies. The research argument behind it is on the home page. For the detection-in-data arm, see PyOD. For the sourced adoption record across every project, see open source.

Start with Your Task

  • Review Python agent code or deployment configuration: start with Agent Audit for source-linked security findings.
  • Check a proposed tool call before execution: start with AEGIS for policy checks and approval decisions on the tool path.
  • Evaluate skill and tool selection for least privilege: start with FORTIS for benchmark tasks, scoring, code and citation.
  • Decide what evidence an agent system should retain: start with Auditable Agents and its Auditability Card.
pip install auditable

For dependency records and replay, explore the auditable SDK and its documented examples. For the research framework behind the lifecycle view, read Auditable Agents before choosing an auditing mechanism.

Tool page  |  Docs  |  GitHub  |  PyPI

Where Each Piece Applies

An agent can be checked at three moments, and the evidence available differs at each. Before a run you have the declared configuration and the code. During a run you have the call about to fire. After a run you have whatever the trace preserved. Each artifact below sits at the moment where its evidence exists, and appears once.

Pre · Before Deployment
Does this agent ask for more than it needs? A static scan of agent code for security and over-privilege issues, run before anything ships.
FORTIS Benchmark
Does the agent stay inside the skill it was given? A benchmark for over-privilege in agent skills: whether a model selects the minimally sufficient skill and selects tools within that skill's documented scope. The benchmark evaluates choices without executing tools against live backends.
Live · While Running
Aegis Tool
Can this call be stopped before it fires? A pre-execution firewall that gates each tool call while the agent is running.
Is one user's data reaching another? Measures leakage across users in agents that share state.
How much memory leaks between agents? Memory leakage in multi-agent LLM systems, measured against the topology that connects them.
Post · After a Run
auditable Tool
What did this decision rely on, and can it be undone? A framework-agnostic system of record that captures each decision's dependencies, replays them against live state, and rolls back the committed action when they no longer hold.
GRADE Method
How can agent steps and their dependencies be represented? A typed two-layer graph of agent execution and dependency, with dependency edges graded by how they are known. The representation is reused by auditable.
Who caused this, when the logs are gone? Recovers segment-level attribution in a multi-agent run from the final text alone.

Across the Whole Lifecycle

Three artifacts do not sit at one moment. One says what auditability requires, one measures whether the requirement was met, and one maps the surrounding literature.

Framework
What does auditability actually require? Five dimensions and three mechanism classes: detect, enforce, recover. The grid above is that framework instantiated.
Benchmark
Was the failure catchable at all? Puts one auditor's question to all three information states under a single interface. It scores 72 entrants over 1187 declared configurations and 1162 recorded runs, and publishes the 71 contrasts that fail to separate rather than ranking them.
Reference
Who else is working on this? A curated map of papers, tools, datasets, and standards for reliable, auditable AI agents, with published checks for paper-title and destination consistency.

Build on It

Every artifact here is public, citable, and built to be extended. If you are adding a detector, a policy, or a new lifecycle stage, open an issue on the repository it belongs to. If you are benchmarking against this work, CatchBench is the entrant interface, and it takes submissions that fail to separate as readily as ones that win.

Most of this stack lives under github.com/yzhao062. agent-audit and Aegis are maintained in their authors' own repositories, linked from their pages above.