LLM evaluation in 2026: revisit promptfoo, DeepEval and Ragas before you ship another prompt change on gut feel

A prompt change looks harmless in review. One extra sentence in the system prompt, a new model version, a different chunk size in the retriever. Then a week later support tickets say the bot has started refusing simple questions. Most teams find out about quality regressions from users, not from tests.

In normal software we learnt long back that we should not merge code without tests. With LLM features many of us have quietly unlearnt that habit, because the output is text and “it looks fine” feels like a test. This post revisits three open-source tools that help you put that habit back: promptfoo, DeepEval and Ragas. The aim is not to crown a winner. The aim is to help you pick one that fits how your team already works.

What an LLM evaluation tool actually does

Before comparing tools, it helps to be clear about the job. An evaluation setup has four parts:

• A dataset of cases: inputs, and where possible expected outputs or reference context.
• A way to run your application against those cases: a prompt plus a model, a RAG chain, or a full agent.
• Metrics or assertions: simple checks (contains, regex, JSON schema, latency, cost) and model-graded checks (an LLM judge scoring relevance, faithfulness or correctness).
• A report and a gate: a result you can compare between runs, and a pass or fail that CI can act on.

All three tools cover these parts in some form. They differ in which part they treat as the centre.

promptfoo: config-first testing and red teaming

History and licence. The promptfoo repository was created in April 2023. It is written in TypeScript and distributed as a CLI and library, under the MIT licence. On 9 March 2026 OpenAI announced an agreement to acquire Promptfoo, with closing subject to customary conditions. Both OpenAI and the promptfoo team said the project stays open source and MIT licensed and will keep supporting many model providers. The latest release at the time of writing is 0.125.1 (October 2026).

How it works. You describe prompts, providers and test cases in a YAML config, then run an eval from the command line. Assertions range from plain string and JSON checks to model-graded rubrics. Results show up as a matrix in the terminal or a local web viewer, which makes side-by-side comparison of two prompts or two models easy.

Where it is strong.

• Teams that are not mainly Python shops. The config is language neutral and the app under test can be an HTTP endpoint.
• Comparing prompts and models side by side before choosing one.
• Red teaming: adversarial tests for prompt injection, jailbreaks, data leaks and similar risks are a large part of the project today.
• CI use. A failing assertion gives a non-zero exit, so a pipeline can block a merge.

What to watch.

• The acquisition is recent. MIT licence means the code you use today stays usable, but keep an eye on roadmap direction and on which features land in the open-source CLI versus hosted products.
• Large YAML configs can become hard to review. Split them and keep test data in separate files.

DeepEval: pytest-style tests for LLM apps

History and licence. DeepEval is developed by Confident AI. The repository was created in August 2023, it is written in Python and licensed under Apache 2.0. The latest Python release at the time of writing is v4.2.4 (September 2026). Confident AI also offers a hosted platform; the framework itself works without it.

How it works. You write test cases in Python, attach metrics, and run them much like unit tests with pytest. The library ships a wide set of ready metrics, including answer relevancy, faithfulness, hallucination and a G-Eval style custom criteria metric, plus metrics aimed at conversations and agents. Most of these are model-graded, so you choose which model acts as the judge.

Where it is strong.

• Python teams who already think in pytest. The mental model is familiar, and the tests sit next to the code.
• Teams who want many metrics out of the box instead of writing rubrics from scratch.
• Synthetic test data generation when you do not yet have a labelled dataset.

What to watch.

• Model-graded metrics cost money and time on every run. Keep a small fast suite for each pull request and a larger suite for nightly runs.
• Some features nudge you towards the hosted platform for dashboards. Decide early whether you need that, and check what data leaves your network if you enable it.

Ragas: metrics focused on retrieval and RAG

History and licence. Ragas started as a research-driven project for evaluating retrieval-augmented generation. The repository was created in May 2023, is written in Python and is licensed under Apache 2.0. The GitHub repository now lives under the vibrantlabsai organisation (the old explodinggradients path redirects there). The latest tagged release at the time of writing is v0.4.3 (January 2026), which suggests a slower release pace than the other two. Check the commit history yourself before you depend on it.

How it works. Ragas gives you metrics such as faithfulness, context precision, context recall and answer relevancy. They look separately at whether the retriever found the right context and whether the generator stayed true to that context. You feed it samples of question, retrieved contexts, answer and, where you have it, a reference answer.

Where it is strong.

• Debugging a RAG pipeline. When answers are bad, Ragas style metrics help you see whether the problem is retrieval or generation. That is the question you actually need answered before changing chunk size or the embedding model.
• Fitting into an existing Python data workflow or notebook, and working with frameworks such as LangChain and LlamaIndex.

What to watch.

• It is narrower by design. For general prompt testing, red teaming or CI gating you may still want one of the other two around it.
• Metric definitions and APIs have changed across versions. Pin the version and re-baseline scores after an upgrade rather than comparing numbers across versions.

Side by side, without a table

Primary interface

• promptfoo: YAML config and CLI, with a local web viewer.
• DeepEval: Python test code, run like pytest.
• Ragas: Python library calls over a dataset.

Licence

• promptfoo: MIT.
• DeepEval: Apache 2.0.
• Ragas: Apache 2.0.

Best fit

• promptfoo: comparing prompts and models, red teaming, polyglot teams.
• DeepEval: Python teams that want broad metrics and test-style workflow.
• Ragas: diagnosing retrieval versus generation in RAG.

Ownership signal to track

• promptfoo: being acquired by OpenAI; promise to stay MIT and multi-provider.
• DeepEval: backed by Confident AI, which sells a hosted platform.
• Ragas: research origin, slower tagged releases in 2026.

We did not run benchmarks for this post. The tools measure different things, and a speed race between them would not tell you much. The real cost driver is the number of judge model calls your metrics make.

The judge model is the part people forget

Model-graded metrics are only as good as the judge. A few practices that help in daily work:

• Pin the judge model and its version, the same way you pin a library. Changing the judge changes the scores even when your app did not change.
• Spot check the judge. Every few weeks, have a human label a small sample and compare with the judge. If they disagree often, fix the rubric before trusting the trend.
• Prefer deterministic checks first. If a JSON schema, a regex or an exact match can test it, use that. It is free, fast and never drifts.
• Watch data privacy. If the judge is a hosted model, your test inputs and outputs go to that provider. For sensitive data, use a judge model you run yourself, or scrub the dataset.

A practical way to start this week

1. Collect 30 to 50 real questions from logs or support, including a few that previously went wrong.
2. Write the cheap checks first: format, must-contain facts, must-not-contain phrases, latency and cost limits.
3. Add two or three model-graded metrics that match your real worry, for example faithfulness for RAG or a custom rubric for tone.
4. Run it on the current production prompt and save the result as the baseline.
5. Put the cheap suite in CI for every prompt or model change, and the full suite on a nightly schedule.
6. Grow the dataset every time a user reports a bad answer. That case becomes a test, just like a bug becomes a regression test.

How to choose

• If your team is mixed language, compares several models, or needs red teaming, start with promptfoo.
• If your team is Python first and likes tests next to code, start with DeepEval.
• If your main pain is “our RAG answers are wrong and we do not know why”, add Ragas metrics to whichever runner you use.

Many teams end up with two of these together, and that is fine. What matters is that a prompt change goes through the same discipline as a code change.

Revisit checklist

• Do we have a saved dataset of real cases, not only demo questions?
• Do cheap deterministic checks run on every prompt, model or retriever change?
• Is the judge model pinned and spot checked against human labels?
• Do we know which test data leaves our network, and to whom?
• Is the eval tool version pinned, with scores re-baselined after upgrades?
• Does every reported bad answer become a new test case?

LLM features are still software. Revisit the testing habits you already know, apply them to prompts and models, and you will ship changes with evidence instead of gut feel.