Dataset Card for "ImageNet-Hard"

Project Page - ArXiv - Paper - Github - Image Browser

Dataset Summary

ImageNet-Hard is a new benchmark that comprises 10,980 images collected from various existing ImageNet-scale benchmarks (ImageNet, ImageNet-V2, ImageNet-Sketch, ImageNet-C, ImageNet-R, ImageNet-ReaL, ImageNet-A, and ObjectNet). This dataset poses a significant challenge to state-of-the-art vision models as merely zooming in often fails to improve their ability to classify images correctly. As a result, even the most advanced models, such as CLIP-ViT-L/14@336px, struggle to perform well on this dataset, achieving a mere 2.02% accuracy.

ImageNet-Hard-4K: For the 4K version please refere to this dataset.

Dataset Distribution

👁 Dataset Distribution

Classifiers Performance

Evaluation Code

• CLIP 👁 Open In Colab
  
• OpenCLIP
• Other models 👁 Open In Colab
  

Supported Tasks

• image-classification: The objective of this task is to classify an image into one or more classes, selected from 1000 ImageNet categories (allowing for multiple ground-truth labels per image).

Languages

The english_label field in the dataset are in English.

Dataset Structure

Data Instances

An example looks like this:

{
'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=575x409 at 0x7F09456B53A0>,
 'label': [0],
 'origin': 'imagenet_sketch',
 'english_label': ['tench']
}

Data Fields

The data instances have the following fields:

• image: A PIL.Image.Image object containing the image. Note that when accessing the image column: dataset[0]["image"] the image file is automatically decoded. Decoding of a large number of image files might take a significant amount of time. Thus it is important to first query the sample index before the "image" column, i.e. dataset[0]["image"] should always be preferred over dataset["image"][0].
• label: A List[int] collection containing the ground-truth ids.
• origin: A string containing source dataset.
• english_label: A List[str] collection containg the english labels for the ground-truth classes.

Data Splits

This dataset is a validation-only set.

Dataset Creation

Source Data

This dataset is sourced from ImageNet, ImageNet-ReaL, ImageNet-V2, ImageNet-A, ImageNet-C, ImageNet-R, ImageNet-Sketch, and ObjectNet.

Citation Information

@article{taesiri2023zoom,
 title={ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification},
 author={Taesiri, Mohammad Reza and Nguyen, Giang and Habchi, Sarra and Bezemer, Cor-Paul and Nguyen, Anh},
 journal={arXiv preprint arXiv:2304.05538},
 year={2023}
}

The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification • 4 items • Updated Sep 12, 2025