VOOZH

URL: https://github.com/topics/llms-benchmarking

⇱ llms-benchmarking · GitHub Topics · GitHub

#

llms-benchmarking

Here are 104 public repositories matching this topic...

steel-dev / awesome-web-agents

🔥 A list of tools, frameworks, and resources for building AI web agents

awesome ai awesome-list browser-automation ai-agents llms llms-benchmarking

Updated
Python

karthikv792 / LLMs-Planning

An extensible benchmark for evaluating large language models on planning

planning pddl benchmark-suite llms llms-reasoning llms-benchmarking llms-planning

Updated
PDDL

JonathanChavezTamales / llm-leaderboard

A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)

llm llmops llm-evaluation llm-agents llms-benchmarking

Updated
JavaScript

bboylyg / BackdoorLLM

[NeurIPS 2025] BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models

attack backdoor defense llms llms-benchmarking

Updated
Python

lechmazur / nyt-connections

Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words

testing benchmark evaluation puzzles reasoning claude llm gpt-5 llms-benchmarking gemini-pro grok4

Updated
Python

ChemFoundationModels / ChemLLMBench

Official Code for What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks (In NeurIPS 2023)

nlp benchmark chemistry ai4science large-language-models llm llms-benchmarking

Updated
Jupyter Notebook

lamalab-org / chembench

How good are LLMs at chemistry?

benchmark machine-learning chemistry safety materials-science llm llms llms-benchmarking

Updated
Python

lerogo / MMGenBench

Official repository of MMGenBench

mllm llms-benchmarking mmgenbench

Updated
Python

parea-ai / parea-sdk-py

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework

Updated
Python

Portkey-AI / models

This repository contains comprehensive pricing and configuration data for LLMs. It powers cost attribution for 200+ enterprises running 400B+ tokens through Portkey AI Gateway every day.

ai models llms llms-benchmarking

Updated
JavaScript

FSoft-AI4Code / XMainframe

Language Model for Mainframe Modernization

migration cobol mainframe code-summarization codellm llms-benchmarking

Updated
Python

lechmazur / generalization

Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.

benchmark evaluation generalization llm llms llms-benchmarking sonnet3-7 gpt-4-5

Updated

SuperBruceJia / Awesome-Mixture-of-Experts

Awesome Mixture of Experts (MoE): A Curated List of Mixture of Experts (MoE) and Mixture of Multimodal Experts (MoME)

artificial-intelligence sparse moe load-balancing multimodal-learning mixture-of-experts mome gating-network foundation-models large-language-models llms large-language-model large-vision-language-models expert-network llms-reasoning llms-benchmarking mixtrure-of-multimodal-experts sparse-moe sparse-mixture-of-experts sparse-mixture-of-multimodal-experts

Updated

rajpurkarlab / craft-md

conversational-ai llms-benchmarking clinical-llm multiturn-conversations

Updated
Python

multinear / multinear

Develop reliable AI apps

reliability evaluation llm llms llm-eval llm-evaluation llms-benchmarking llm-evaluation-framework

Updated
Python

eth-sri / constrained-diffusion

Constrained Decoding of Diffusion LLMs with Context-Free Grammars.

diffusion diffusion-model constrained-decoding llm llms fill-in-the-middle llms-benchmarking multi-region-infilling

Updated
Rust

RaptorMai / MLLM-CompBench

[NeurIPS'25] MLLM-CompBench evaluates the comparative reasoning of MLLMs with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes

benchmark reasoning vision-and-language multimodal-deep-learning human-annotation foundation-models large-language-models llms vision-language-model multimodal-large-language-models evaluation-llms llms-benchmarking neurips-2024

Updated
Jupyter Notebook

amazon-science / llm-code-preference

Training and Benchmarking LLMs for Code Preference.

code-generation llm-training llm-evaluation llms-benchmarking

Updated
Python

mbzuai-oryx / Agent-X

ICLR 2026: Agent-X Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

benchmarking vision reasoning vfm foundation-models llms-benchmarking agentic-ai

Updated
Jupyter Notebook

declare-lab / resta

Restore safety in fine-tuned language models through task arithmetic

alignment safety alignment-algorithm llm llms llm-safety llms-benchmarking llm-safety-benchmark

Updated
Python

Improve this page

Add a description, image, and links to the llms-benchmarking topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llms-benchmarking topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.