Image Classification in the Art Domain

Abstract

Digitizing art collections is a major challenge for many museums and art galleries. To facilitate and accelerate cataloging photos of artworks, we tackle two classification tasks from the art domain using deep learning: type classification and genre classification of artworks. To train our models, we use the popular transfer learning approach. Since our training dataset is highly imbalanced, our work focuses on coping with imbalanced training data. Our results show that the transfer learning approach can produce very good results even for small and highly imbalanced training datasets. We observed that acquiring or generating additional training data as well as certain data augmentation methods can slightly improve training results. Over- and undersampling techniques, on the other hand, do not seem to be necessary and did not provide a substantial benefit. To optimize performance in both classification tasks, we experiment with multiple training methods and model architectures. In this way, we obtain good results in both tasks: In the type classification task, we achieve an accuracy of over 99% and an F1-score above 97% for both the minority and majority class. In the genre classification task, we achieve an accuracy of over 96% and F1-scores ranging from 88% to 99% for the respective classes.

Konstantin Dobler

PhD Student & Graduate Researcher

I’m a PhD student at Hasso Plattner Institute researching transfer learning of large language models.