Publications / Agent evaluation

FORTIS: Benchmarking Over-Privilege in Agent Skills

Shawn Li, Chenxiao Yu, Han Wang, Wei Yang, Ryan Rossi, Franck Dernoncourt, Xiyang Hu, Philip Yu, Chaowei Xiao, Huan Zhang, and Yue Zhao

arXiv preprint, 2026. First author: Shawn Li. This page describes the FORTIS benchmark; FORTIS Lab is the research group.

Paper (v3) · Full text · Code and data · Citation

Does an LLM agent choose more authority than a task needs? FORTIS evaluates least-privilege behavior when an agent can choose among overlapping skills and tools. It separates choosing a broader skill from selecting tools that exceed an assigned skill's permissions.

What is measured: skill and tool selection in constructed scenarios. The benchmark does not execute the selected tools against live backends. Its scores describe model choices under these conditions and do not certify a deployed agent's safety. See the paper's limitations.

How can you evaluate least-privilege choices?

Task 1, skill selection: given a request and overlapping skill descriptions, select the least-privileged skill sufficient for that request. Task 2, skill-grounded tool selection: given an assigned skill, its documentation and a tool inventory, select a sufficient tool set within the skill's boundary. These tasks isolate two decision stages that can fail separately. Task definitions.

Can ordinary requests lead to permission overreach?

The evaluation includes incomplete requests, convenient broader options, and requests near a skill's documented limits. These settings probe overreach without requiring malicious instructions. For example, a request about two mail folders can tempt a model to select a mailbox-wide search when separate searches within those folders would suffice. Failure examples.

What data and code can you use?

The v3 benchmark covers email, e-commerce and filesystem scenarios: 600 skill-selection queries and 1,543 tool-selection queries. The skill hierarchies are synthetically constructed; results require further validation on natural user traces and other domains.

• Queries and evaluation code for the two tasks.
• Skill documents, tool definitions and prompts for inspecting the privilege boundaries.
• Evaluation runners and the repository instructions for model evaluation.

Start by inspecting the query labels and skill definitions for one domain. Then consult the runners for your evaluation setup. This page links the released materials; it does not report a new reproduction of the paper's results.

Cite the paper

Shawn Li et al. (2026). FORTIS: Benchmarking Over-Privilege in Agent Skills. doi:10.48550/arXiv.2605.09163.

@misc{li2026fortis,
  title={{FORTIS}: Benchmarking Over-Privilege in Agent Skills},
  author={Li, Shawn and Yu, Chenxiao and Wang, Han and Yang, Wei and
          Rossi, Ryan and Dernoncourt, Franck and Hu, Xiyang and Yu, Philip and
          Xiao, Chaowei and Zhang, Huan and Zhao, Yue},
  year={2026},
  eprint={2605.09163},
  archivePrefix={arXiv},
  primaryClass={cs.AI}
}

Page updated September 8, 2026. Paper scope checked against arXiv v3 (June 14, 2026); artifact links checked against repository revision 68945b8. The links above follow the current repository; the pinned revision records the version inspected.