Supports both end-to-end and component-level LLM evaluation.
支持端到端和组件级别的 LLM 评估 (LLM evaluation)。
Large variety of ready-to-use LLM evaluation metrics (all with explanations) powered by ANY LLM of your choice, statistical methods, or NLP models that run locally on your machine:
• G-Eval
大量即用型 LLM 评估指标（均带有解释），由您选择的任何 LLM、统计方法或在您机器上本地运行的 NLP 模型提供支持： G-Eval (G-Eval)
DAG (deep acyclic graph)
DAG（深度非循环图）(deep acyclic graph)
RAG metrics:
• Answer Relevancy
RAG 指标： 答案相关性 (Answer Relevancy)
Faithfulness
忠实性 (Faithfulness)
Contextual Recall
上下文召回率 (Contextual Recall)
Contextual Precision
上下文精确率 (Contextual Precision)
Contextual Relevancy
上下文相关性 (Contextual Relevancy)
• RAGAS

🦄 LlamaIndex, to unit test RAG applications in CI/CD
🦄 LlamaIndex，用于在 CI/CD（持续集成/持续交付）中进行 RAG（检索增强生成）应用程序的单元测试。
🤗 Hugging Face, to enable real-time evaluations during LLM fine-tuning
🤗 Hugging Face，以在 LLM（大型语言模型）微调期间启用实时评估。

pip install -U deepeval

deepeval login

touch test_chatbot.py

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        # Replace this with the actual output from your LLM application
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
    )
    assert_test(test_case, [correctness_metric])

export OPENAI_API_KEY="..."

deepeval test run test_chatbot.py

The variable input mimics a user input, and actual_output is a placeholder for what your application's supposed to output based on this input.
变量 input 模仿用户输入，而 actual_output 是应用程序应该基于此输入输出的内容的占位符。
The variable expected_output represents the ideal answer for a given input, and GEval is a research-backed metric provided by deepeval for you to evaluate your LLM output's on any custom with human-like accuracy.
变量 expected_output 表示给定输入的理想答案，而 GEval 是 deepeval 提供的基于研究支持的指标，用于评估您的 LLM（大型语言模型）输出在任何具有类人准确性的自定义上的表现。
In this example, the metric criteria is correctness of the actual_output based on the provided expected_output.
在此示例中，指标标准是基于提供的 expected_output 的 actual_output 的正确性。
All metric scores range from 0 - 1, which the threshold=0.5 threshold ultimately determines if your test have passed or not.
所有指标分数范围从 0 到 1，threshold=0.5 阈值最终决定了您的测试是否通过。

from deepeval.tracing import observe, update_current_span
from deepeval.test_case import LLMTestCase
from deepeval.dataset import Golden
from deepeval.metrics import GEval
from deepeval import evaluate

correctness = GEval(name="Correctness", criteria="Determine if the 'actual output' is correct based on the 'expected output'.", evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT])

@observe(metrics=[correctness])
def inner_component():
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    update_current_span(test_case=LLMTestCase(input="...", actual_output="..."))
    return

@observe
def llm_app(input: str):
    inner_component()
    return

evaluate(observed_callback=llm_app, goldens=[Golden(input="Hi!")])

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the actual output from your LLM application
    actual_output="We offer a 30-day full refund at no extra costs.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)
evaluate([test_case], [answer_relevancy_metric])

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the actual output from your LLM application
    actual_output="We offer a 30-day full refund at no extra costs.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)

answer_relevancy_metric.measure(test_case)
print(answer_relevancy_metric.score)
# All metrics also offer an explanation
print(answer_relevancy_metric.reason)

import pytest
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

dataset = EvaluationDataset(goldens=[Golden(input="What's the weather like today?")])

for golden in dataset.goldens:
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=your_llm_app(golden.input)
    )
    dataset.add_test_case(test_case)

@pytest.mark.parametrize(
    "test_case",
    dataset.test_cases,
)
def test_customer_chatbot(test_case: LLMTestCase):
    answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)
    assert_test(test_case, [answer_relevancy_metric])

# Run this in the CLI, you can also add an optional -n flag to run tests in parallel
deepeval test run test_<filename>.py -n 4

from deepeval import evaluate
...

evaluate(dataset, [answer_relevancy_metric])
# or
dataset.evaluate([answer_relevancy_metric])

cp .env.example .env.local
# then edit .env.local (ignored by git)

Curate/annotate evaluation datasets on the cloud
在云端管理/注释评估数据集
Benchmark LLM app using dataset, and compare with previous iterations to experiment which models/prompts works best
使用数据集对 LLM 应用程序进行基准测试，并与之前的迭代进行比较，以实验哪种模型/提示效果最佳
Fine-tune metrics for custom results
微调指标以获得自定义结果
Debug evaluation results via LLM traces
通过 LLM 跟踪调试评估结果
Monitor & evaluate LLM responses in product to improve datasets with real-world data
监控和评估产品中的 LLM（大型语言模型）响应，以使用真实世界的数据改进数据集。
Repeat until perfection
重复直到完美。

deepeval login

deepeval test run test_chatbot.py

cp .env.example .env.local
# then edit .env.local (ignored by git)

Integration with Confident AI
与Confident AI集成
Implement G-Eval
实现G-Eval
Implement RAG metrics
实现RAG指标
Implement Conversational metrics
实现会话指标 (Conversational metrics)
Evaluation Dataset Creation
评估数据集创建 (Evaluation Dataset Creation)
Red-Teaming
红队测试 (Red-Teaming)
DAG custom metrics
DAG自定义指标
Guardrails
护栏 (Guardrails)