FIDEL Agents v1

FIDEL Agents measures whether tool-using language-model agents remain reliable when the same task is expressed in Amharic and a paired English control. It separates language-linked conditions from obstacle-isolation controls and scores completed actions, answer surfaces, and safe abstention.

FIDEL Agents v1 contains 100 measured pairs and 200 task files: 24 calendar, 24 clock, 22 homophone, 12 kin-term, 4 bilingual-name, and 14 abstention pairs. Eight additional bilingual-name pairs in the private source were withdrawn and are not runnable benchmark tasks.

FIDEL Agents is built by Kaleb Gebresillasie and Abhishek Mynam at Catalode Inc. See CITATION.cff for citation metadata.

Install and validate

python -m pip install -e '.[test]'
python scripts/check_symmetry.py
python scripts/check_aux_parity.py
python scripts/smoke_test.py
python -m pytest -q

Every command above runs without credentials or a network request. For an air-gapped setup, see the offline installation section in evaluation guidance; a wheelhouse needs setuptools and wheel alongside the declared dependencies.

List providers and validate a one-pair selection without credentials or a network request:

python scripts/run_evaluation.py --list-providers
python scripts/run_evaluation.py --provider anthropic \
  --model YOUR_MODEL --max-tasks 1 --dry-run

After installing the selected provider extra and supplying your own configuration:

python scripts/run_evaluation.py --provider openai-chat \
  --model YOUR_MODEL --tasks tasks --output episode_records

Supported selections are anthropic, openai-chat, openai-responses, bedrock, gemini-api, vertex, and azure. See evaluation guidance and the original aggregate results.

The identity environment models core elements of Fayda's public OIDC and eKYC relying-party flow. Its mismatch tasks are synthetic safety stress tests, not documented production Fayda procedures. See the identity environment for which components are documented Fayda behavior, which are compressed abstractions, and which are benchmark-created.

Interpretation and release status

Original reported evaluations were collected before task publication. Post-release scores are open-benchmark results that may reflect contamination or benchmark-specific optimization. V1 has no hidden measured-task split and is not a permanently secure leaderboard.

Modeled services and providers do not sponsor, endorse, or affiliate with this benchmark. Their names identify simulated domains or serving configurations.

License and citation

FIDEL Agents v1 is released under two licenses. The software is licensed under the Apache License, Version 2.0. The task data, the documentation, and the aggregate result files are licensed under the Creative Commons Attribution 4.0 International license. Catalode Inc. is the copyright holder.

NOTICE is the canonical statement of which license covers which paths. The full license texts are in LICENSE and LICENSE-CC-BY-4.0.txt, both unmodified upstream texts. Dependency attributions are in THIRD_PARTY_NOTICES.md.

Using the CC BY 4.0 material requires attribution: "FIDEL Agents v1, Catalode Inc., licensed under CC BY 4.0." If you report a score against this benchmark, cite it using CITATION.cff and state the benchmark version, so the number stays traceable to the corpus and caveats it was measured under.