The short version: Evalyn is a local-first tool. Your data stays on your machine. We don't collect, transmit, or store any of your evaluation data, prompts, or agent outputs.
Our Commitment to Privacy
Evalyn is built with privacy as a core principle. Unlike cloud-based evaluation tools, Evalyn runs entirely on your local machine. Your prompts, responses, traces, and evaluation results never leave your environment.
What Data Evalyn Collects
None. Evalyn does not collect any data from your usage. Specifically:
- No telemetry or usage analytics
- No crash reporting sent to external servers
- No prompts, responses, or evaluation data transmitted anywhere
- No account registration or authentication required
Local Data Storage
All data generated by Evalyn is stored locally on your machine in SQLite databases. This includes:
- Traced LLM calls and their inputs/outputs
- Evaluation metrics and scores
- Human annotations and calibration data
- Project configurations and settings
You have complete control over this data. You can delete it at any time by removing the SQLite database files.
Third-Party Services
When you use Evalyn, you may connect to third-party LLM providers (OpenAI, Anthropic, etc.) through your own API keys. Evalyn does not intermediate these connections. Your API keys and the data you send to these providers are subject to their respective privacy policies.
Website Analytics
This documentation website may use privacy-respecting analytics to understand aggregate traffic patterns. No personally identifiable information is collected. You can block these analytics with any standard ad blocker.
Open Source Transparency
Evalyn is open source under the MIT License. You can inspect the entire codebase to verify our privacy claims. We welcome security audits and contributions.
Contact
If you have questions about this privacy policy or Evalyn's data practices, please contact us or open an issue on GitHub.