Valkyrie

Valkyrie orchestrates scalable, reproducible evaluations for AI agents. It supports Vals-hosted infrastructure and self-hosted AWS deployments.

Guides, the CLI reference, the Python SDK reference, and contributor setup are published at docs.valkyrie.vals.ai.

Quickstart

Note: Valkyrie can be invoked using either valkyrie or the alias valk. For example: valkyrie run start or valk run start.

Install the CLI:

uv tool install git+https://github.com/vals-ai/Valkyrie@prod

Configure credentials and choose a hosting mode:

valkyrie config init

Upload an agent with a valid contract.yaml:

valkyrie agent push ./agents/sweagent --name sweagent

Start a benchmark run:

valkyrie run start \
  --agent sweagent \
  --benchmark swebench \
  --model anthropic/claude-sonnet-4-6 \
  --slice "0:10" \
  --connect

Where to go next

Topic	Location
Install and run your first benchmark	get-started/quickstart
Uploading agents and writing a contract	agents/agent-contract
Starting, monitoring, and managing runs	runs/start
Results and output artifacts	runs/results-and-outputs
Hosting your own benchmark service	benchmarks/custom-services
Every CLI command	reference/cli/index
Python SDK	sdk/quickstart
Local development	contributing/local-development

Citation

If you use Valkyrie in your research, cite the paper:

@inproceedings{forzano2026valkyrie,
  author    = {Forzano, Jarett and Almatov, Omar and Nashold, Langston and Ravi, Nikil and Kassian, Orestes},
  title     = {Valkyrie: A Microservice-Based Framework for Scalable Evaluation of AI Agents},
  booktitle = {Proceedings of the 1st ACM Conference on Agentic and AI Systems},
  series    = {CAIS '26},
  year      = {2026},
  month     = {may},
  address   = {San Jose, CA, USA},
  publisher = {ACM},
  location  = {New York, NY, USA},
  numpages  = {5},
  doi       = {10.1145/3786335.3813231},
  url       = {https://doi.org/10.1145/3786335.3813231}
}