I value practical and reproducible research. This page highlights open-source libraries, benchmarks, and system tools built by my group and collaborators. Many of these projects support Auditable AI Systems across all four audit questions. Detection in data covers anomalies, outliers, and out-of-distribution inputs. Verification of output covers LLM evaluation, hallucination, and jailbreak detection. Auditing of action covers AI agent security, tool-call and MCP security, and audit trails. Optimization of effort covers agent efficiency, model routing, and the cost of defense training. These methods apply in science and high-stakes domains.
Open-source program participation: Several of these projects have been accepted into the Anthropic Claude for Open Source Program and OpenAI's Codex for Open Source.
Ecosystem stewardship: Two NSF awards fund this work as ecosystem building rather than as method research: Pathways to Enable Open-Source Ecosystems (POSE) Phase I for trustworthy-LLM tooling, and POSE Phase II for the anomaly-detection stack. PyOD is now in its third major release.
For most repositories, see my GitHub profile. agent-audit and Aegis are maintained in their authors' own repositories.
| Type | Evidence | Source |
|---|---|---|
| AI Lab | OpenAI Careers names PyOD as expected operational tooling in the Technical Intelligence Analyst job posting Qualifications block: "Have experience with anomaly detection tools, such as PyOD, and discovery processes for surfacing novel or low-prevalence patterns." | OpenAI · snapshot |
| AI Lab | A second OpenAI Careers posting, Quantitative Threat Forecasting Analyst, lists PyOD by pinned version among the toolchains a hire is expected to know: "Expertise with modern toolchains, NumPyro, TensorFlow Probability, PyMC, Darts, GluonTS/Chronos, sktime, PyOD 2.0, River, scikit-survival" | OpenAI (archived) · snapshot |
| UK Gov | UK Government Algorithmic Transparency Record (London Borough of Sutton, "Access Assure" technology-enabled care) links PyOD KNN documentation in its Model Specification (section 4.2.6); a production deployment record | GOV.UK |
| Space Agency | Selected by ESA for OPS-SAT spacecraft telemetry benchmark (all 30 algorithms) | Nature Sci. Data |
| Space Agency | A second ESA document, authored by ESA/ESOC staff, assesses anomaly-detection algorithms against operational requirements and treats PyOD as the reference library a compliant system would extend: candidate algorithms "would need additional mechanisms or modifications of external libraries (i.e., PyOD) to provide a list of affected channels" (p.31); COPOD is evaluated alongside it | ESA/ESOC report |
| Saudi Gov | Saudi Data & AI Authority (SDAIA) Deepfakes Guidelines names PyOD (p.10) among recommended tools for detecting suspicious activity patterns | SDAIA PDF |
| U.S. DoD | CDAO Generative AI Responsible AI Toolkit lists PyOD as a Production / High-maturity OOD-detection tool (entry p.49, embedded in Stage 3.1.10 assessment workflow) | ai.mil PDF |
| Japan MoD | Japan Ministry of Defense ATLA AI Guideline v01 lists PyOD as entry 14 in its recommended-tooling table, mapped to sections B-5 (Reliability and Validity) and B-6 (Ensuring Safety), linking the repository and docs; the second national defense ministry to name PyOD as recommended tooling | ATLA PDF |
| India Gov | Government of India, Department of Telecommunications, Telecommunication Engineering Centre: the draft Standard on AI Robustness lists PyOD in its recommended-tooling table under Anomaly Detection (p.66), "PyOD – Outlier detection algorithms"; the third national government to name PyOD as recommended tooling | TEC PDF |
| Brazil Congress | Brazil's Chamber of Deputies publishes procurement-overpricing research in its official proceedings that runs PyOD, then ensembles HBOS, ECOD, COPOD, GMM, PCA and IForest through SUOD, reporting the full accuracy, precision and recall table and engaging ADBench's comparative claim; a coauthor directs postgraduate coordination at the Chamber's Cefor | Câmara dos Deputados PDF |
| Central Bank | Bank for International Settlements, Irving Fisher Committee Bulletin 57: Deutsche Bundesbank researchers select PyOD as modelling tooling for outlier detection in official statistics and implement the DCSO algorithm, citing PyOD and LSCP | BIS IFC PDF |
| Central Bank | Bank for International Settlements, Irving Fisher Committee Bulletin 66: Bank of Thailand researchers evaluate ECOD on granular regulatory credit data and state that their goal is to deploy it in production, "implemented on-premises under Apache platform (Spark or Hadoop)" | BIS IFC PDF |
| National Labs | Eight U.S. DOE national-laboratory reports use the tools in their methods: Sandia implements APT anomaly detection on PyOD, SLAC runs PyOD's one-class SVM as the comparison baseline in its instrument anomaly-detection study, Brookhaven ships COPOD in an exascale workflow system, Oak Ridge verifies attack detection with LSCP, and Idaho National Laboratory and LLNL both build on TrustLLM | OSTI · SLAC |
| Brazil | Brazil's National Treasury journal Cadernos de Finanças Públicas publishes a federal audit of discrepant municipal education spending built on PyOD, authored by an official of the Controladoria-Geral da União, Brazil's federal internal-audit and anti-corruption body | Tesouro Nacional |
| EU Project | SEDIMARK Horizon Europe D3.1 (p.18) names PyOD and TODS in the outlier-detection module of the EU data-space toolbox | SEDIMARK D3.1 |
| Gov / Labs | Cited in research papers by authors affiliated with Deutsche Bundesbank, NIH, CDC, RAND, NASA JPL, German DLR and DESY, and the Sandia, Brookhaven, and Argonne national labs, plus multiple Fraunhofer institutes (citing ECOD, COPOD, PyOD, ADBench, TODS, and LSCP) | Audit details |
| Platform | Apache Software Foundation / Apache Beam (8.5K+ stars) ships a first-class PyOD ModelHandler at sdks/python/apache_beam/ml/anomaly/detectors/pyod_adapter.py; Apache Beam underlies Google Cloud Dataflow |
apache/beam |
| Enterprise | PostHog (34K+ stars, YC unicorn product analytics) runs a multi-detector PyOD subsystem at posthog/tasks/alerts/detectors/pyod_detectors/ for live-traffic alerting (eight algorithm wrappers: KNN, IForest, COPOD, ECOD, OCSVM, LOF, PCA, HBOS) |
PostHog/posthog |
| Platform | MLflow (25.8K+ stars) official community-flavor docs list PyOD as the canonical anomaly-detection flavor with worked KNN-detector example via mlflavors |
mlflow/mlflow |
| Pharma | Genentech (Roche) Data Detective embeds PyOD/ADBench in its drug-discovery validator factories (adbench_validator_method_factory.py, adbench_multimodal, adbench_ood_inference) |
Genentech/data-detective |
| Enterprise | Walmart real-time pricing anomaly detection (1M+ daily updates) | KDD 2019 |
| Enterprise | Databricks Kakapo framework for unsupervised outlier detection | Databricks Blog |
| Enterprise | IQVIA healthcare fraud detection (123K+ pharmacy claims) | SUOD Paper |
| Enterprise | Ericsson Anomaly Detection Framework (E-ADF) built on PyOD | Ericsson Blog |
| Patents | 49 patents cite PyOD/COPOD/ECOD/LSCP/SUOD/TODS/XGBOD/ADBench (China x29, US x11, WIPO x3, Korea x2, EU x2, Slovakia x1, Japan x1). Many go beyond citation to implementation: China Mobile adopts the PyOD toolkit and selects twelve of its algorithms, Baidu trains COPOD models on five feature classes for anomalous-access detection, and Atlas Space Operations implements COPOD's empirical-copula scoring. Other assignees include Visa International, Tencent, Ping An Medical, CETC 54, Harbin Institute of Technology, and Ericsson | Visa · China Mobile · Baidu · Atlas Space |
| Journal | Two 2026 Nature Scientific Reports papers implement anomaly detection via PyOD in their Methods (eight PyOD detectors in one; COPOD/ECOD/IForest in the other), each citing the PyOD JMLR paper | s41598-026-45091-2 |
| Journal | Cell Press life-science papers run PyOD in their pipelines: Developmental Cell calls the pyod.models.knn detector to identify outlier cells, iScience lists pyod 1.0.9 with the HBOS detector as a key resource in a Broad-Institute-linked morphological profiling pipeline, and Stem Cell Reports uses it for single-cell outlier removal | Dev. Cell · iScience |
| Encyclopedia | Wikipedia "Anomaly detection" Software section names PyOD; reference list cites Zhao, Nasrullah, Li 2019 JMLR | Wikipedia |
| Education | Featured in 5 books (Manning, O'Reilly, Apress, Routledge, IntechOpen) | Manning |
| Education | DataCamp course with dedicated chapter (19M+ platform learners) | DataCamp |
| Type | Evidence | Source |
|---|---|---|
| Benchmark Baseline | Praetor (arXiv:2604.26274) deploys Aegis as a quantitative baseline behavioral firewall, reporting 12.8% attack success for Aegis versus 2.2% for their method, and cites the Aegis paper by title and arXiv ID | arXiv |
| Vendor Comparison | An independent containment paper places Aegis beside two vendor products rather than beside academic prior art, comparing it against NVIDIA OpenShell and Microsoft's Agent Governance Toolkit in its capability analysis: "AEGIS provides a pre-execution firewall with content scanning and tamper-evident audit trails" | arXiv |
| Media | The Agent Times feature names Yue Zhao and explains all five auditability dimensions of the Auditable Agents framework | Agent Times |
| Industry Blog | WIDTH applies the Auditable Agents five dimensions and the 8.3 ms firewall-overhead result to compliance infrastructure | WIDTH |
| Media | Machine Brief explains the mechanism, results, and limitations of Implicit Execution Tracing (recovering which agent produced a harmful output after logs are stripped) | Machine Brief |
| Downstream Citations | Auditable Agents has seven confirmed third-party scholarly citations within months of posting, each discussing the framework in body text rather than listing it in a bibliography: AgentBound, DEMM, Trace2Policy, OpenClawBench, Proof-Carrying Agent Actions, Operational Identity, and Explanation-Bound Tool Execution. All seven verified by direct PDF scan. | AgentBound · DEMM · Proof-Carrying |
| Framework Priority | A 41-page method specification builds on the framework and cedes priority to it, in a section titled "Auditable Agents and the Three-Layer Reading": "The broad auditability-framework coordinate is owned by Auditable Agents"; it then positions its own contribution on the paper's detect/enforce/recover partition. The same paper cites four of the lab's works, including Aegis as the "AEGIS-NTC tool firewall". | DEMM |
| Prior-Art Baseline | An independent agent-safety paper credits Aegis with establishing the mechanism it builds on: "our contribution is therefore not the enforcement mechanism, which pre-execution systems such as AEGIS and AgentSpec already establish" | arXiv |
| Framework Adoption | A third-party tool-execution paper adopts both frameworks by name: "Auditable Agents frames recoverability, policy checkability, responsibility attribution, and evidence integrity as system properties; Agent Audit analyzes code and deployment artifacts before execution" | arXiv |
| Survey Coverage | agent-audit is listed as representative compliance-evidence tooling in an Agentic Services Computing survey, and cited by two further independent agent-security papers | Survey · arXiv |
| Ecosystem | agent-audit is indexed in the ReputAgent ecosystem directory; awesome-auditable-ai is referenced from a Hacker News thread and indexed by Ecosyste.ms | ReputAgent · HN |
| Type | Evidence | Source |
|---|---|---|
| Consulting | Deloitte Germany cites ADBench in an AIxAML anti-money-laundering transaction-monitoring solution | Deloitte PDF |
| Enterprise | Cited in papers with authors affiliated with Microsoft Research, Tencent, Amazon, BlackRock, Visa, Bosch, Siemens, and Ericsson | Audit details |
| Pharma | Genentech (Roche) Data Detective uses ADBench in adbench_validator_method_factory.py, adbench_multimodal, and adbench_ood_inference validator factories for drug-discovery data validation |
Genentech/data-detective |
| Journal | Nature Communications study links the ADBench repository in its Data availability section for the benchmark anomaly datasets used | Nature Comms |
| Type | Evidence | Source |
|---|---|---|
| U.S. Senate | Cited in HSGAC "Hedge Fund Use of Artificial Intelligence" report (footnote 119) | Senate PDF |
| U.S. DoD | Listed in CDAO Generative AI Responsible AI Toolkit | ai.mil PDF |
| NIST | Named in NIST AI 100-2e2025 Section 3.6 "Benchmarks for AML Vulnerabilities" | NIST PDF |
| Policy | Official benchmark in all 4 editions of the FLI AI Safety Index (2024, 2025 x2, Summer 2026) | FLI Report · Summer 2026 |
| National Lab | Lawrence Livermore National Laboratory feature article; LLNL/DOE SafeAI report cites TrustLLM | LLNL · SafeAI PDF |
| International | Cited in International AI Safety Report 2026 (citation #881; led by Yoshua Bengio, 100+ experts, 30+ countries) | Report |
| International | Also cited in the earlier International AI Safety Report 2025 (reference 1035, p.281), a separate edition of the same series | Report PDF |
| Japan | NICT (National Institute of Information and Communications Technology) cybersecurity deck describes TrustLLM and its six trustworthiness dimensions: truthfulness, safety, fairness, robustness, privacy, and machine ethics. A second NICT deck presents an AI Security Evaluation Platform whose “Select test sets” panel offers three options, two of them built on TrustLLM: AdvGLUE and Jailbreak | NICT PDF · NICT platform deck |
| U.S. DOE | Oak Ridge National Laboratory technical report ORNL/TM-2025/3935, "Scalable Workflow for Evaluating Trustworthiness of Large Language Models," discusses and cites TrustLLM | OSTI |
| G7 / OECD | Salesforce G7 Hiroshima AI Process Transparency Report (OECD-hosted) cites TrustLLM (p.4) among its trust-and-safety evaluation metrics | OECD |
| Industry Lab | NTT Technical Review names TrustLLM as an LLM-safety benchmark and links the benchmark repository (reference [4]) | NTT |
| Media | Featured by 机器之心 (Jiqizhixin) and 澎湃新闻 (The Paper) | 机器之心 · 澎湃 |
| Enterprise Editorial | Samsung SDS Insights treats TrustLLM as a flagship LLM trustworthiness evaluation framework in its Korean enterprise editorial; reference list cites arXiv:2401.05561 | Samsung SDS |
| Type | Evidence | Source |
|---|---|---|
| Institute | Vector Institute highlights TrustGen in its ICLR 2026 research roundup | Vector |
| Industry Lab | Adobe Research lists FigEdit ("Charts Are Not Images") and the benchmark release | Adobe Research |
| Type | Evidence | Source |
|---|---|---|
| Journal | Published in Nature Chemical Biology (2022) | Nature Chem. Bio. |
| AI Lab | Google Research trained Tx-LLM on TDC and says so in the official release post: "We leveraged data from the Therapeutic Data Commons (TDC), a public collection of drug discovery datasets for training ML models, and processed 66 tasks most relevant to drug discovery into instruction-answer formats suitable for LLMs." Google DeepMind's later TxGemma models draw their prediction tasks from TDC as well. TDC is a co-authored project; Yue Zhao is not its lead author | Google Research · TxGemma |
| University | Harvard Medical School feature: "Can AI transform drug discovery?" | HMS News |
| Science Press | Phys.org syndication of Harvard article | Phys.org |
| Industry | Amazon Science feature article | Amazon Science |
| Pharma | Cited by researchers at AstraZeneca, Pfizer, Roche, Novartis, Merck, Sanofi, Eli Lilly | Audit details |
| Labs | Cited in papers by researchers at Los Alamos and Brookhaven national labs (cheminformatics, model uncertainty) and OpenAI (biomedical reasoning) | Audit details |
| Type | Evidence | Source |
|---|---|---|
| Policy | Cited by Privacy International in "Nowhere to Hide? Privacy Risks and Policy Implications of AI Geolocation" (p.28, footnote 56) | Report |
| Chinese Media | 机器之心Pro reporting via Sina names "南加州大学教授赵越（Yue Zhao）团队", paper title "Doxing via the Lens", and the arXiv link | Sina |