Pattern Networks in Art History

Introduction

The foundation for this project begins with EPFL’s Replica Project (2015-2019) spearheaded primarily by Isabella di Lenardo, Benoit Seguin, and Frederic Kaplan. Essentially, the Replica Project aimed to create a searchable digital collection of artworks leveraging a large dataset of artworks (CINI dataset). Leveraging both the CINI dataset and a CNN-based architecture, a system of links was developed between artworks based on their morphology. Pre-trained with historian corrections, these morphological links aimed to identify whether there existed unquestionable influence from one artwork to another. For example, Image 1 shows a series artworks with a “positive” morphological link; in other words, there is no question the artworks was referencing or was influenced by the other.

From left to right: Sleeping Venus by GIORGIONE (c.1510) (landscape often attributed to TITIAN ) ; Venus of Urbino by T ITIAN (1538); Olympia by E DOUARD M ANET (1863). The figures, while not perfectly the same, are practically identical in morphology (the posture, the shape, etc.)

However, compared to the capabilities of CNN-based image pipelines, transformer-based architectures have emerged with vastly greater capabilities for image-segmentation and analysis. Therefore, our group’s overarching goal has been to develop and understand the power of implementing a modern image processing pipeline for identifying morphological links between artworks. This has been explored in three steps:

1. Using a segmentation model to extract the different elements of paintings.
2. Matching the shape of segmented regions to find plausible correspondences.
3. Confirming the correspondences based on the morphology of the masked elements.

Over the course of the semester, our group has explored a range of visual-image-transformer (ViT) models to perform segmentation on the dataset. Primarily we aimed to exploit the dataset of positive morphological links to test if the models will accurately mask the shape of the necessary features. The main model our group pivoted towards was the Segment-Anything Model (SAM) by Meta. Specifically, we tested the SAM2 model: a transformer-powered vision model for promptable and automatic segmentation. Ironically however, while testing the model, Meta released a new version of the SAM model (SAM3). Our methodological processes and assessment in the sections below go into greater detail into our observations. At a high-level glance, we outline a table below highlighting the milestones and weekly evolution of our project.

Motivation

As mentioned previously, our group’s goal has been to develop and understand the power of implementing a modern image processing pipeline for identifying morphological links between artworks. By testing this pipeline, we hope to open the door for future academics to update the Replica Projects’s CNN-based architecture with a modern transformer-based version. With these updates, there is a potential to discover hidden morphologies previously undetectable.

Original planing

Weekly project milestones

Week	Milestones
1	Explored Cini dataset, relevant repositories, and research papers to better understand the scope of the project.
2	Explored image-segmentation models and research papers on segmentation and related work, including SAM2, panoptic segmentation, and transformer-based image processing.
3	Used segmentation models to test mask generation on pairs of positively linked images to assess whether masks accurately captured morphology.
4	Identified limitations with automatically generated segmentation masks and explored alternative or more fine-tuned segmentation approaches.
5	Investigated the release of the SAM 3 model and explored ways to leverage promptable segmentation within the experimental pipeline.
6	Continued developing the pipeline and identified solutions addressing prompting and contouring limitations in segmentation.
7	Finalized documentation and deliverables within the codebase and presented findings.

• Week 9: Continue exploring models/pipelines
• Week 10: Finish implementing a segmenter + decide on shape comparison method
• Week 11: Exam prep
• Week 12 Finish implementing shape comparator + start of image matching
• Week 13: Writing Wiki, report + catching up on unfinished tasks
• Week 14: Finalizing everything for presentation and submission