Pattern Networks in Art History: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
<h1>Introduction</h1> | <h1>Introduction</h1> | ||
The foundation for this project begins with EPFL’s Replica Project (2015-2019) spearheaded primarily by Isabella di Lenardo, Benoit Seguin, and Frederic Kaplan. Essentially, the Replica Project aimed to create a searchable digital collection of artworks leveraging a large dataset of artworks (CINI dataset). Leveraging both the CINI dataset and a CNN-based architecture, a system of links were developed between artworks based on their morphology. Pre-trained with historian corrections, these morphological links aimed to identify whether there existed unquestionable influence from one artwork to another. For example, Image 1 shows two artworks with a “positive” morphological link; in other words, there is no question that one of the artworks was referencing or was influenced by the other. | |||
[Image1] | |||
[Image 1 Caption]: Two paintings with the same figure of a mother holding a baby. The figures, while not perfectly the same, are practically identical in morphology (the posture, the shape, etc.) | |||
However, compared to the capabilities of CNN-based image pipelines, transformer-based architectures have emerged with vastly greater capabilities for image-segmentation and analysis. Therefore, our group’s overarching goal has been to develop and understand the power of implementing a modern image processing pipeline for identifying morphological links between artworks. This has been explored in three steps: | |||
1. Using a segmentation model to extract the different elements of paintings. | |||
2. Matching the shape of segmented regions to find plausible correspondences. | |||
3. Confirming the correspondences based on the morphology of the masked elements. | |||
Over the course of the semester, our group has explored a range of visual-image-transformer (ViT) models to perform segmentation on the dataset. Primarily we aimed to exploit the dataset of positive morphological links to test if the models will accurately mask the shape of the necessary features. The main model our group pivoted towards was the Segment-Anything Model (SAM) by Meta. Specifically, we tested the SAM2 model: a transformer-powered vision model for promptable and automatic segmentation. Ironically however, while testing the model, Meta released a new version of the SAM model (SAM3). Our methodological processes and assessment in the sections below go into greater detail into our observations. At a high-level glance, we outline a table below highlighting the milestones and weekly evolution of our project. | |||
<h2>Motivation</h2> | <h2>Motivation</h2> | ||
Revision as of 00:49, 17 December 2025
Introduction
The foundation for this project begins with EPFL’s Replica Project (2015-2019) spearheaded primarily by Isabella di Lenardo, Benoit Seguin, and Frederic Kaplan. Essentially, the Replica Project aimed to create a searchable digital collection of artworks leveraging a large dataset of artworks (CINI dataset). Leveraging both the CINI dataset and a CNN-based architecture, a system of links were developed between artworks based on their morphology. Pre-trained with historian corrections, these morphological links aimed to identify whether there existed unquestionable influence from one artwork to another. For example, Image 1 shows two artworks with a “positive” morphological link; in other words, there is no question that one of the artworks was referencing or was influenced by the other.
[Image1]
[Image 1 Caption]: Two paintings with the same figure of a mother holding a baby. The figures, while not perfectly the same, are practically identical in morphology (the posture, the shape, etc.)
However, compared to the capabilities of CNN-based image pipelines, transformer-based architectures have emerged with vastly greater capabilities for image-segmentation and analysis. Therefore, our group’s overarching goal has been to develop and understand the power of implementing a modern image processing pipeline for identifying morphological links between artworks. This has been explored in three steps:
1. Using a segmentation model to extract the different elements of paintings. 2. Matching the shape of segmented regions to find plausible correspondences. 3. Confirming the correspondences based on the morphology of the masked elements.
Over the course of the semester, our group has explored a range of visual-image-transformer (ViT) models to perform segmentation on the dataset. Primarily we aimed to exploit the dataset of positive morphological links to test if the models will accurately mask the shape of the necessary features. The main model our group pivoted towards was the Segment-Anything Model (SAM) by Meta. Specifically, we tested the SAM2 model: a transformer-powered vision model for promptable and automatic segmentation. Ironically however, while testing the model, Meta released a new version of the SAM model (SAM3). Our methodological processes and assessment in the sections below go into greater detail into our observations. At a high-level glance, we outline a table below highlighting the milestones and weekly evolution of our project.
Motivation
Related work
Expected results
Original planing
- Week 9: Continue exploring models/pipelines
- Week 10: Finish implementing a segmenter + decide on shape comparison method
- Week 11: Exam prep
- Week 12 Finish implementing shape comparator + start of image matching
- Week 13: Writing Wiki, report + catching up on unfinished tasks
- Week 14: Finalizing everything for presentation and submission
Our pipeline