Meta

• License: Apache Software License (Apache 2.0)
• Author: HuggingFace Inc.
• Tags metrics , machine , learning , evaluate , evaluation
• Requires: Python >=3.8.0
• Provides-Extra: dev , docs , evaluator , quality , template , tensorflow , tensorflow-gpu , tests , torch

Classifiers

• Development Status
  • 5 - Production/Stable
• Intended Audience
  • Developers
  • Education
  • Science/Research
• License
  • OSI Approved :: Apache Software License
• Operating System
  • OS Independent
• Programming Language
  • Python :: 3
  • Python :: 3.8
  • Python :: 3.9
  • Python :: 3.10
• Topic
  • Scientific/Engineering :: Artificial Intelligence

👁 Image

👁 Build
👁 GitHub
👁 Documentation
👁 GitHub release
👁 Contributor Covenant

Tip: For more recent evaluation approaches, for example for evaluating LLMs, we recommend our newer and more actively maintained library LightEval.

🤗 Evaluate is a library that makes evaluating and comparing models and reporting their performance easier and more standardized.

It currently contains:

• implementations of dozens of popular metrics: the existing metrics cover a variety of tasks spanning from NLP to Computer Vision, and include dataset-specific metrics for datasets. With a simple command like accuracy = load("accuracy"), get any of these metrics ready to use for evaluating a ML model in any framework (Numpy/Pandas/PyTorch/TensorFlow/JAX).
• comparisons and measurements: comparisons are used to measure the difference between models and measurements are tools to evaluate datasets.
• an easy way of adding new evaluation modules to the 🤗 Hub: you can create new evaluation modules and push them to a dedicated Space in the 🤗 Hub with evaluate-cli create [metric name], which allows you to see easily compare different metrics and their outputs for the same sets of references and predictions.

🎓 Documentation

🔎 Find a metric, comparison, measurement on the Hub

🌟 Add a new evaluation module

🤗 Evaluate also has lots of useful features like:

• Type checking: the input types are checked to make sure that you are using the right input formats for each metric
• Metric cards: each metrics comes with a card that describes the values, limitations and their ranges, as well as providing examples of their usage and usefulness.
• Community metrics: Metrics live on the Hugging Face Hub and you can easily add your own metrics for your project or to collaborate with others.

Installation

With pip

🤗 Evaluate can be installed from PyPi and has to be installed in a virtual environment (venv or conda for instance)

pipinstallevaluate

Usage

🤗 Evaluate's main methods are:

• evaluate.list_evaluation_modules() to list the available metrics, comparisons and measurements
• evaluate.load(module_name, **kwargs) to instantiate an evaluation module
• results = module.compute(*kwargs) to compute the result of an evaluation module

Adding a new evaluation module

First install the necessary dependencies to create a new metric with the following command:

pipinstallevaluate[template]

Then you can get started with the following command which will create a new folder for your metric and display the necessary steps:

evaluate-clicreate"Awesome Metric"

See this step-by-step guide in the documentation for detailed instructions.

Credits

Thanks to @marella for letting us use the evaluate namespace on PyPi previously used by his library.

Verified details

Maintainers

Unverified details

Project links

• Homepage
• Download

• Download URL: evaluate-0.4.6.tar.gz
• Upload date: Sep 18, 2025
• Size: 65.7 kB
• Tags: Source
• Uploaded using Trusted Publishing? No
• Uploaded via: twine/6.2.0 CPython/3.9.23

See more details on using hashes here.

• Download URL: evaluate-0.4.6-py3-none-any.whl
• Upload date: Sep 18, 2025
• Size: 84.1 kB
• Tags: Python 3
• Uploaded using Trusted Publishing? No
• Uploaded via: twine/6.2.0 CPython/3.9.23

See more details on using hashes here.