Humans of Paris 1900
Motivation
Our project takes inspiration from the famous Instagram page, Humans of New York, which features pictures and stories of people living in current day New York. In similar fashion, our project, Humans of Paris, has the aim to be a platform to connect us to the people of 19th century Paris. Photography was still in its early stages when Nadar took up the craft in his atelier in Paris. Through the thousands of pictures taken by him and his son we can get a glimpse of who lived at the time. We explore the use of deep learning models to cluster similar faces to get an alternative, innovative view of the collection and allowing for serendipitous discovery of patterns and people. There is a story behind every person, and our interface highlights this by association people’s story with their picture.
Historical Background
The collection at the heart of our project is comprised of 23000 photographs (positives) and caricatures by Atelier Nadar. We focus on individual portraits of people for our interface.
The portrait studio Atelier Nadar was founded by Nadar, otherwise known by his birth name Felix Tournachon (1820-1910). Nadar first established himself as a known caricaturist after working in journalism. Then, having been introduced to photography by his Brother, Adrien, he soon realized the potential of this new arform. With the support of a banker friend he went on to create the Studio in Paris in 1854. [1] [1]
Photography had started to become a viable business as new collioduim-on-glass negatives, invented in 1851, allowed for the creation of copies from one negative while being cheaper than previous methods due to the use of glass all while having short exposure times. [2]
The studio was frequented by anyone in the public eye, be it politicians, famous artists and writers of the time. Famous individuals of the time, that are still of large cultural importance today, captured in the photos include Liszt, Victor Hugo and Baudelaire. His minimalistic, no-frills style as well as lack of editing allows the photos highlight the expressions, and as such the personalities of his subjects.
Nadar himself believed that what set him apart from other photographers was his feel for light and how he connected to the subjects of his photographs.s His talent in engaging with people was especially important as in the mid 19th century the camera was a bulky box under which the photographer would disappear here
Later, starting 1860, as Nadar himself moves on the other endeavors, his son, Paul, took over the studio. Paul moved the studio into a more commercial direction, in part due to the increasing competition. A different style of photography started to emerge from the studio, such as theater photography as well as miniature portrait photography.
The collection held by the french state was acquired in 1949, since 1965 the positives of the Photos are stored with the BnF, while the negatives are held at Fort de Saint-Cyr.
Implementation
Website Description
In more concrete terms, our project involves four core interfaces.
- A home page highlighting the most known individuals according with the number of pictures or the amount of text written about them in wikipedia
- A page (FaceMap) that highlight similarities in differences in the faces of the people in the dataset.
- A page to find your 19th century doppelgänger, for fun and to gather interest in people the user my otherwise would never have known.
- A way to search using tags, to allow users to find individuals of interest.
To each person in the pictures we associate background information crawled from wikipedia.
All the code is available at github and the detailed instructions to run the application can be found in the READ.md file.
Methods & Evaluation
Getting & Processing Metadata
As a first step in the processing pipeline, we use the Gallica Wrapper provided by Raphael to get a list of all the photos in the collection of the foundation Nadar on Gallica. Nadar's collection contains a variety of genres: portraits, comics, caricatures, paysage, sculptures, etc. In order to stick to our emphasis on ‘people of 19th century Paris’, we filtered out photographs that are not directly relevant to people of that time.
Getting individual portraits
We used metadata of Gallica collection to filter irrelevant photographs. Among a number of attribute objects in the metadata, we concentrated on the ‘dc:subject’ attribute. This field contains a list that has detailed information about the photograph and the entity in that photo: [Names of individuals],(year of birth - year of death) -- [genre of the photograph]. For each row, We ignored subjects that do not have the substring “-- Portraits” and returned the new list of subjects. This way, we can discard the landscape, comics, caricatures, and sculptures. After filtering only ‘Portraits’, we had column of list that varied in length: that is, the number of people featured in photographs differed. Since our intention is to connect ourselves to people of 19th century Paris by presenting story of each Parisien-ne in the photograph, we filtered out the photographs that features more than one person. In order to filter and get insights of people who work in the same field or had the same role we created the concept of tags that helps us to access and query groups of people.
First attempt
Gallica ‘dc:subject’ metadata had very brief information on each person - only name, year of birth, and year of death, which was not enough information to sort categorize each person. Thus, for our first attempt, we tried to use ‘dc:title’ metadata (that gives title of each photograph) to get additional information on people. However, titles were so badly-formulated, making it hard to preprocess them. Moreover, most of the time, titles of images contained only 1) the name of people in the photograph, or, 2) the fictional characters the people(especially performers) played. Since this did not give us useful information to categorize people, we moved onto our second attempt to query the names we have to get well-formulated information associated with the names and the personages.
Second attempt
Data.bnf.fr is the project driven by the BnF in order to make the data produced by the BnF more visible on the Web, and federate them within and outside the catalogues. Since Gallica is one of the projects of the BnF, it is reasonable to assume that a person who has their name in Gallica metadata will have some document or page in data.bnf.fr semantic web. In order to get corresponding pages, we queried metadata though python SPARQL API. To do so, we must understand the XML schema that is returned by the API. The Data.bnf XML schema has three name attributes: foaf:name, foaf:givenName, foaf:famliyName. We checked how names in <dc:subject> is arranged, compare the arrangement with some names in Data.bnf, and rearranged names accordingly. Finally, we queried them.
Sometimes query result contained different entities that have the same name. For this case, we exploited the fact that we are handling 19th century data, taken by one author, Nadar. Among the namesakes, we chose the one whose living period overlaps the most with that of Nadar.
Creating Tags out of metadata
The table from the data.bnf html page contains a variety of useful information, such as: name, nationality, language, gender, short description. We used the description and nationality to create tags. The method used to extract tags from the descriptions is to use the most frequent words that appear across different people, excluding stopwords. The problem with this is that some important keywords and entities are broken up under word count. For example, “Legion d’honneur”, is split into “legion” and “d’honneur” and the original meaning is lost. We handle this my manually checking. First, we checked the word count without stopwords. Then we got frequent words that are not related to occupations. We checked notes that contain those words and examined whether they are part of important (and frequent) phrases. For the phrases that are found, we concatenated them into one word, so that they can be used as tag. After this step, we fetched 300 major keywords, some being feminin or plural conjugation of another. Cleaning those semantic duplicates was done manually. For future improvement, this job could be done by using proper french NLP library. Once this list of possible tags is created, we assigned tags for each picture by having intersection between the list of all tags and the list of words in each description. Finally, nationality was added in the list of tags.
Using Wikipedia as criteria of importance
Data.bnf contains useful information on each person, but it is hard to infer their historic influence only by looking at the table. So we used wikipedia as a criteria that define people’s ‘importance’ - more images they have in their page, more important. We used Wikipedia API library that is available in pypy wikipedia The library instance contains Wikipedia page, number of images and reference links they have. Moreover, it has auto-suggestion, so the query string does not have to be an exact match with the article we are looking for. Sometimes auto-suggestion fails. To handle these cases, we verified if the content of Wikipedia page contains the name and the birth/death year of the person.The wiki content that does not have enough match were considered false positives and neglected. The number of images in each Wikipedia page defines ‘weight’ of the person. We got this for English and French Wikipedia in order to represent the importance outside francophone culture as well. The final weight (or importance) is the average between two, so the ones that have Wikipedia pages for both languages are boosted in importance.
Evaluation
To check how well the metadata is processed, we randomly sampled metadata 50 metadata, each for data.bnf.fr page, English Wikipedia, and French Wikipedia. It is considered correct if : 1) the name 2) the face 3) the birth/death year matches.
| Metadata | Accuracy (%) |
|---|---|
| Data.bnf.fr page | 94 |
| English Wikipedia | 100 |
| French Wikipedia | 98 |
Mismatches in data.bnf.fr happened for the ones who have namesakes (the person who bears the identical name) in the database and the ones whose name or birth/death year written in <dc:subject> attribute are ambiguous. English Wikipedia has shown surprisingly good accuracy for this sample. Mismatch in French Wikipedia happened for the one that does not have a Wikipedia page, thus redirected to some random page by auto-suggestion algorithm. Overall, the error in metadata processing occurred for the people with limited information or record. One way to improve this is to identify the obvious disambiguation pages and neglect them. But still, it is showing a fair result.
Image processing and cropping
To give users a better aesthetic experience on our webpage, we decided to crop the background on which the photograph is placed. Cropping the background was done based on edge detection. We based our code off of (https://github.com/polm/ndl-crop) but made modifications to adapt it to use on sepia photography on blue backgrounds. This involves adding an additional preprocessing step that sets the blue background to white. We determine the background using the RGB values. It uses image processing techniques provided by OpenCV to find the contours in the images, from which we infer boundaries. Setting the blue background to white gives us better contour detection. Using heuristics about the size of the photo against the background we then choose the most appropriate boundary. See figure x to see an example of the process.
Cropping the face is done using the facial rectangle returned by the py-agender api (see below) and MTCNN. MTCNN (https://github.com/ipazc/mtcnn) is an other python library implementing a Deep Neural Network as proposed by FaceNet (also used to find the facial vectors, see openface). Py-agender is much quicker, so we use it for the initial cropping.
Evaluation
To check how well the background is cropped, we randomly sampled 100 images and check for the cropping. 91% of images are correctly cropped. The incorrectly cropped images contain some border. None of them are overcropped. We might be able to do better by tuning the assumptions we make to choose the most appropriate boundary, as we are currently missing very large pictures and very small ones. But for the current stage of the project we find the accuracy to be sufficient.
Looking at the face cropping done with py-agender, we find a total of 52 incorrectly cropped images. We also find 71 images where no face could be found, despite there being one in the image. We find this number by looking through all 1874 cropped images. We rectify that badly cropped images using the MTCNN library, which correctly identifies that cropped images that do not contain a face and correctly crops those images given the original image. In the end we get 5 incorrectly cropped portraits. They are all from very washed out and damaged photos. We replace those images with manually cropped faces. The accuracy of the method could be improved by re-training each model with our data, to better match our use-case.
Age and Gender
The data from the Gallica API wrapper contains neither information about gender nor about the age of the person pictured at the time the photograph was taken. Therefore had to use some other method to guess the age and gender on a individual in the image. We later find that data.Bnf has ground truth labels for the gender for many individuals in the database. We used this for the evaluation of the gender labels, but note that we used the ground truth labels in our final product. Given the data at hand, we could use the name to guess the gender. However, a quick survey of available methods to detect gender from names shows that no database adapted to the task of classifying 19th century french names exists. Not having the means to create such a database from scratch, we focus on the photo and faces in them. Detecting age and gender from photos has long been a topic of Deep Learning research. Many pre-trained state of the art models trained on modern data exist. People presumably have not changed much in 100 years, so we can expect methods to work well.
We use the python library py-agender which implements a state of the art model proposed by Rasmus Rothe et al., the implementation can be found here and it was trained on the IMBD-Wiki dataset.
The library endpoint returns a list of dictionary objects containing:
- the facial rectangle
- a gender estimate as a value between [0, 1]; 1 indicating the individual is male, 0 indicating the individual is female
- an age estimate, as a floating point number.
We get estimates for all individuals in the dataset, based on one image. We first try for one random image per individual, and if the model can not detect a face, we try another image, until we get a result or no more images are left.
Evalutation
The underlying model itself has limitations. As can be seen in figure of validation loss, both age and gender are still high, despite being the state of the art. Concerning the application of the model to our data, we create a histogram of all the estimates we get. The histogram for gender is strongly skewed towards male values.
We also see that there is a large overlap in values assigned to female and male subjects. Clearly, the model is not working well. To give exact values to each issue, consider the following confusion matrix:
| True Male | True Female | |
|---|---|---|
| Labeled Male | 1393 | 114 |
| Labeled Female | 237 | 130 |
Here we can clearly see that the biggest issue is that the model classifies women as men. Doing a qualitative analysis of the 100 images, we formulate the following hypothesis why the gender estimation model creates so many errors. While people’s faces have not changed, these days women tend to have long flowing hair and wear more visible makeup. This is not the case in our dataset where women appear to mostly have updos and wear barely visible makeup. An additional issue might be the angle at which people posed. To solve this problem we would have to retrain the model on a labeled sample of out images.
The age histogram follows a gaussian like distribution centered around 40. It is much harder to estimate whether this is an accurate assessment. The metadata does not give us any information on when the photo was taken. For this reason we just use random sampling to get a subsample of 95 images on which we do a qualitative evaluation. We annotate the samples by the age, but only giving the decade. Even for humans guessing the age is difficult. We are only interested in an estimate - decades are enough. For 72 out of 95 we get the decade correct. The age estimation algorithm works reasonably well. For the ones that we do miss, the difference between the exact age estimation on the estimated decade is less than a full decade.
Getting the face vector for facemap and doppelganger search
To represent a complex object such as the image of a face in two dimensional space and examine similarities between faces we need to convert it into a vector space. This can also be done using a Deep Learning model. We use the openface, an implementation based on the CVPR 2015 paper FaceNet: A Unified Embedding for Face Recognition and Clustering by Florian Schroff, Dmitry Kalenichenko, and James Philbin at Google.
The method first finds a bounding box and important facial landmarks, which are then used to normalize and crop the photo. This cropped photo is then fed to the neural network, which returns points lying on a 128D hypersphere. Issues can arise at each step of this process. The special property of the resulting vector is that euclidean distance in the feature space represents closeness in facial features.
To find a person's doppelganger we can therefore use cosine similarity between the vectors of the people in our dataset and the vector of the person who uploads their image.
To display the facemap we need 2D vectors. Going from a higher dimensional space (in our case 128) to a lower dimensional space - in our case 2 - requires the use of dimensionality reduction algorithm to project the higher space vector into a lower dimensional space. We tried a total of two dimensionality reduction algorithms, the linear method PCA and the non-linear method T-SNE.
Evaluation
Creation of face vector: To estimate the quality of the resulting vector we again have the accuracy of the underlying method, and how it applies to our case. The deep learning method used to generate the vector is trained on a face classification task. On this task the model has accuracy of 0.9292 ± 0.0134 and AUC (area under curve) of 0.973. Human annotators can achieve AUC of .995 on the task. The papers also discusses the application of the face vectors to clustering. They note that the method is invariant to occlusion, lighting, pose and even age.
On our task, we check whether the face is correctly detected and how well the resulting vector works for the tasks at hand - clustering and finding doppelgängers. Due to the style of photography, we can not produce a face vector as they show only their profile in the images. The models we use are not adapted to such a strong variation from frontal pictures. In total, we have 71 people for which we can not find vectors.
Dimensionality reductions: A qualitative evaluation of both reduction algorithm was done by checking the face-map clusters. We found that T_SNE better approximates gender boundaries and similarities in faces. For this reason we use T_SNE in our final implementation.
Frontend
Use adobe xd to do initial prototype to explore how to design frontend and then use bootstrap templates to match our prototypes. We us D3.js to make the facemap interactive.
D3 (Data-Driven Documents) is the most flexible javascript library to produce interactive data visualizations. Even though it has a lot of built-in functions, it also gives total control to the programmer customize its own visualizations. In order to adapt D3 to have a better zooming of the images we used tsne visualization which uses perperxity to calculate the distance between images based on the predefined clusters (see discussion of clustering).
Backend
To render images and text to a web interface we use Django 2.2.7 which is a powerful web framework written in python. Combined with scripts written Javascript, they give the frontend behavior that we see and interact on the site. Django also contains its own ORM (object-relational mapping) layer which turns easy to store data, update and query it to be handled on the interface.
![[1]](https://raw.githubusercontent.com/cmusatyalab/openface/master/images/summary.jpg){kind=link}
Underneath we are using Sqlite3 to store our data however due to the ORM properly that turn the storage option agnostic with respect to the code and framework implementation.
Unfortunately openface is just available in python2.7 which will be deprecated in January 2020 and our project was written in python3. In order to run both versions smoothly when a new image is uploaded we decided to spin openface separately as another service that receives a post request with a new image and returns the encoded vector to the original website. The openface project was forked in github and on top of that we implemented a Flask application who is ready to receive post request at the endpoint \get_vector and returns json response which will be consumed by our application and find the doppelganger.
Project execution plan
The milestones and the project schedule is bellow:
| Timeframe | Task | Completion |
|---|---|---|
| Week 4 | ||
| 07.11 | Understanding Gallica Query Gallica API | ✓ |
| Query Gallica API | ||
| Week 5 | ||
| 14.10 | Start preprocessing images | ✓ |
| Choose suitable Wikipedia API | ||
| Week 6 | ||
| 21.10 | Choose face recognition library | ✓ |
| Get facial vectors | ||
| Try database design with Docker & Django | ||
| Week 7 | ||
| 28.10 | Remove irrelevant backgrounds of images | ✓ |
| Extract age and gender from images | ||
| Design data model | ||
| Extract tags, names, birth and death years out of metadata | ||
| Week 8 | ||
| 04.11 | Set up database environment | ✓ |
| Set up mockup user-interface | ||
| Prepare midterm presentation | ||
| Week 9 | ||
| 11.11 | Get tags, names, birth and death years in ready-to-use format | ✓ |
| Handle Wikipedia false positives | ||
| Integrate face recognition functionalities into database | ||
| Week 10 | ||
| 18.11 | Create draft of the website (frontend) | ✓ |
| Create FaceMap using D3 | ||
| Week 11 | ||
| 25.11 | Integrate all functionalities | ✓ |
| Finalize project website | ||
| Week 12 | ||
| 02.12 | Write Project report | ✓ |
References
- ↑ title=MetMuseum|url=https://www.metmuseum.org/toah/hd/nadr/hd_nadr.htm%7Cwebsite=https://www.metmuseum.org/%7Caccessdate=24 November 2019
- ↑