Paris: address book of the past

Introduction

This project works with around 4.4 million datapoints which have been extracted from address books in Paris (Bottin Data). The address books date from the period 1839-1922 and contain the name, profession and place of residence of Parisian citizens. In a first step, we align this data with geodata of Paris’s city network, in order to be able to conduct a geospatial analysis on the resulting data in a second step.

Motivation

In the 19th century, Paris was a place of great transformations. Like in many other European cities, the industrialization led to radical changes in people’s way of life, completely reordering the workings of both economy and society. At the same time, the city grew rapidly. While only half a million people lived there in 1800, the number of inhabitants increased by the factor of nearly 7 within one century. To get control over the expansion of the city and to improve people’s living conditions, the city underwent constructional changes during the Haussmann Period (1853-1870), leading to the grand boulevards and general cityscape that we know today.

While all those circumstances have been well documented and studied extensively, they could mainly be described in a qualitative manner, e. g. by looking at the development of certain streets. The Bottin dataset, providing information on persons, their professions and locations during exactly the time of change described above, will be able to open new perspectives on the research, as it permits to analyze the economic and social transformation on a grander scale. With this data, it will be possible to follow the development of a chosen profession over the whole city, or to look at the economic transformation of an Arrondissement throughout the century.

This project will give an idea of the potential which lies in the Bottin Data to contribute to the research on Paris’s development during the 19th century.

Deliverables

Github

The Github contains the code and data for all steps of the project. The following are the jupyter notebooks which were used for the alignment and analysis of the data. They should be executed in the order they are presented hereunder. For the sake of clarity and reusability, we collected the function of each notebook in seperate python files. The files used for a notebook will be given in brackets.

• Preprocessing.ipnyb (preprocessing.py): Preprocessing of the street names in the Bottin Data, including substitution of abbreviations.
• Street_processing.ipnyb (preprocessing.py, paris_methods.py): Aligning the two street network datasets "Open Data" (2022) and "Vasserot" (1836), solving conflicts for non-unique entries
• Alignment.ipnyb (preprocessing.py, alignment.py): Aligning bottin streets with the streets of the street data computed in street_processing.ipnyb
• Analysis.ipynb (analysis.py): Analysis on the aligned data

For an overview on the data files used and produced during the project, see the section "Data" in the Readme in Github.

Google Drive

The Google Drive contains all files which were too large to upload on Github. They should be put in the "data" folder on Github in order for the code to work without error.

Organisation

Milestones

Milestone 1

• Getting familiar with Qgis, Geopandas, Shapely and the “Bottin” and both Street datasets.
• Developing a way to process Bottin data to be able to do perfect matching on a “dummy” street-dataset

Milestone 2

• Improve perfect matching by constructing dictionary of abbreviations and manually fix most common OCR-Mistakes.
• Find a way to deal with streets that matched on a short-streetname, which has multiple possible long-streetnames. For example Bottin data matched on “Voltaire” but we don’t know if “Rue Voltaire” or “Boulevard Voltaire”.

Milestone 3

• Combining the “Openstreet” and “Vasserot” datasets into one dataset. Implementing a way to automatically merge streets with same name.
• Process the “Bottin” professions with Ravi’s code to get tags of professions.

Milestone 4

• Implement fuzzy-matching to align remaining “Bottin” data to streets.

Timetable

	Part	Street Data	Bottin Data	Analysis
Week 4	Alignment	First look at Datasets, learn geopandas	Preprocessing	-
Week 5		Street visualisation	Perfect Matching with first Street Data
Week 6		Develop & implement strategy to work with duplicates
Week 7			Substituting abbreviations, alignment without spaces
Week 8
Week 9		Have one single street dataset
Week 10			Updating Street Data
Week 11		Update methods for 1836 dataset (before 1791 dataset)	Fuzzy Matching, Updating Street Data, Including profession tags
Week 12	Analysis			General Analysis
Week 13		Assign streets to geographical regions	-	Specific Points
Week 14

Alignment

The Pipeline of the Alignment

The alignment happened in two steps: First, two street network datasets of the years 1836 and 2022 were combined to create one dataset which incorporates all available geodata. Second, this dataset was used to align the Bottin datapoints with a matching street and thus georeferenced.

Aligning Street Datasets

The Street Datasets

Two geolocated street datasets were used in this project. The "Vasserot" dataset 1836 provides a baseline how the city looked like in the beginning of the dataset. The "Open Data" dataset gives an accurate depiction of the street network of today. Both datasets were combined into one dataset in order to be able to geolocate the Bottin datapoints.

Preprocessing

In order to be able to compare the streetnames of both datasets, all streetnames were converted into lowercase and accents were substituted by their counterpart without accent. The signs "-_" as well as double spaces were turned into one space. The "Vasserot" dataset contained empty streetnames which were removed from the dataset.

Vasserot duplicate streets

In the "Vasserot" dataset streets are often named more than once. This is because the authors of the dataset treated segments of the same street as a unique entity and because in Paris some streetnames were not unique during that time. For our analysis we needed to differentiate between those two cases and combine the geolocated data of all street segments.

To combine street segments we used the shapely package to see if streets with the same name are overlapping or are closer than a given distance. In the end we decided on a distance of 200meters, after checking the results manually.

With this approach we could reduce the number of duplicate streetnames from 729 to 88. We excluded those duplicate streets from further analysis and focused on the 1202 unique streetnames.

Comparing Opendata and Vasserot streets

To be able to assign each streetname to one geolocation we had to merge the "Opendata" and the processed "Vasserot" dataset. For this we used the same approach as above and compared if streets that share the same name are closer than 200 meters from another. 50 Streets that appeared in the "Opendata" and the "Vasserot" dataset and were further than 200meters appart were excluded from further analysis.

Our final dataset includes 7204 unique streetnames. During the processing of the streets we excluded 138 streetnames from our dataset.

Duplicate streets in Vasserot dataset after processing, excluded in analysis

Duplicate streets between both datasets after processing, excluded in analysis

Streets used in final analysis

Aligning Bottin Data with Street Data

After having constructed one dataset of past and present streets in Paris, it will be used to align each datapoint of the Bottin Data with a geolocation. The alignment process is decribed in detail in the following chapter. For this, we first introduce the Bottin dataset, then elaborate on the methods of the alignment process and finally estimate the quality of the alignment.

Bottin Dataset

The dataset referred to as “Bottin Data” consists of slightly over 4.4 million datapoints which have been extracted via Optical Character Recognition (OCR) out of address books named “Didot-Bottin” from Paris. The address books are from 55 different years within the time span of 1839 and 1922, with at least 37’177 entries (1839) up to 130’958 entries (1922) per given year. The address books can be examined at the Gallica portal. Each entry consists of a person’s name, their profession or activity, their address and the year of the address book it was published in.

As the extraction process from the scanned books was done automatically by OCR, the data is not without errors. Di Lenardo et al., who were responsible for the extraction process, estimated that the character error amounts to 2.6% with a standard deviation of 0.1%, while the error per line is 21% with a standard deviation of 2.9%.

Tagged Professions

This project was able to build upon previous work by Ravinithesh Annapureddy, who wrote a pipeline to clean and tag the profession of each entry in the data. As the code was not necessarily written for the exact dataset we worked with, the notebook “cleaning_special_characters.ipynb” could not be used, as it referenced the entries by row number in the dataframe. However, the notebooks “creating_french_dictionary_words_set.ipynb” and “cleaning_and_creating_tags.ipynb” were successfully run, adding to each entry in the Bottin Data a list of tags. Those tags were the result of a series of steps including the removal of stopwords (such as “le” or “d’”), the combination of broken words, spell correction (using frequently appearing words in the profession data and two external dictionaries) and the extension of abbreviations.

We faced two challenges using the pipeline by Ravinithesh. First, the pipeline was not optimized for large datasets, so it took a long time to run. Second, the created tags were stored in a list and there has not been any work done to cluster them to profession sectors or fields, which made the usage difficult during the analysis. We chose to convert the lists (e.g. [‘peints’, ‘papier’]) to a string where the tags were comma-separated (“peints, papier”). Effectively, this meant that we corrected misspelled professions, which for example led to almost 1000 more entries having the profession entry “vins”. But at the same time, it did not take an entry like “peints, papier” into account when selecting the profession “peints” in the analysis.

Alignment Process

Number of aligned entries per method; alignment was first conducted by perfect matching, then without spaces and lastly with fuzzy matching

The alignment was conducted on the street level. For this, the street column (“rue”) of the Bottin dataset was used to carry out a left join, using three different versions of the street network data as the dataset “on the right side”:

(1) The column with the full (long) street name of the street data,

(2) the column with the short street name (meaning without type of the street) of the street data, where the short street names are unique, and

(3) the street data with non-unique short street names.

The reason for joining both on the full and short street names lies in the fact that the entries in the Bottin dataset often only contain short versions of the street names and thus cannot be matched on the full street names. Moreover, the division into unique and non-unique short street names was made with regards to the analysis which followed afterwards. As the non-unique data is by nature ambiguous, it is not used in the analysis. However, as the streets still appear in the Bottin data, they have to be matched. Otherwise, we would have risked letting them match to wrong streets during the fuzzy matching step.

After each alignment, the newly aligned data was appended to already aligned data and the next alignment was repeated with all entries which had not been successfully aligned yet. Several matching methods were conducted consecutively, namely perfect matching, matching without spaces and fuzzy matching. Within each method, the alignment was first executed for the long street names (1), then for the unique short street names (2) and in the end for non-unique short street names (3).

The overall achieved alignment is 89.30%, with 84% of the aligned data matched with perfect matching, 4% without spaces and 12% with fuzzy matching.

Preprocessing

Before the alignment could be conducted, the same preprocessing function used for the street networks was run, meaning that characters with accents were substituted by their counterpart without accent and the signs "-_" as well as double spaces were turned into one space. Additionally, as the dataset contained around 1.5 million abbreviations, we wrote two dictionaries with which we could substitute those abbreviations:

1. The first dictionary was built using the Opendata Paris street network, as it lists streets in three versions: their long streetname (e.g. “Allée d'Andrézieux”), the abbreviated streetname (prefix abbreviated, e.g. “All. d'Andrézieux”), short streetname (e.g. “ANDREZIEUX”). We computed the difference between long and short street names (“Allée d’”) and the difference between abbreviated and short street names (“All. d‘”) and saved the resulting pair in the dictionary. This way, around 500’000 abbreviations could be substituted.
2. The second dictionary was built manually, by printing the most common abbreviations left in data and saving them with their corresponding correct form in the dictionary. With only 152 dictionary entries, the number of abbreviations could be further lowered from around 1 million to less than 50’000.

Perfect Matching

The first method for the alignment consisted in perfect matching. Therefore, the preprocessed street (“rue_processed”) from Bottin Dataset was joined (left join) with the preprocessed street columns of the street dataset (on (1), (2) and (3) respectively, see Alignment Process).

With this method, already 75% of the whole dataset could be aligned.

More specifically, 28.08% were perfectly aligned on the long street names (1), 35.86% on the unique short street names (2) and 10.93% on the non-unique short street names (3).

Perfect Matching without Spaces

As the OCR often did not recognize spaces correctly, we decided to align the streets after having deleted all whitespaces. Additionally, we deleted the characters | . : \ and saved the resulting strings in new columns respectively. On those columns, perfect matching like above was carried out. Although the relative number seems small – only 1.44% of the Bottin data was aligned on (1), 1.71% on (2) and 0.31% on (3) with this method – this still amounts to around 145’000 datapoints in the Bottin dataset.

Fuzzy Matching

Until this point of the process, the alignment was based on perfect matching, including some manual changes of the data. In order to be able to account for the OCR mistakes which are present in the Bottin data, fuzzy matching was included, allowing for very similar strings to align.

The process of finding the most similar street out of a list of possible streets was implemented with the help of the library fuzzywuzzy, which computes a similarity ratio based on the Levenshtein distance and the length of the strings. The basic idea of the Levenshtein distance is to count how many substitutions, deletions or insertions are needed to transform one string into another. The fuzzywuzzy library allows for a score cutoff value: If the similarity between two words falls below the given cutoff value, the algorithm returns None. This speeds up the computation significantly.

For the alignment, the thresholds 80% and 85% for the similarity ratio were chosen, based on manual assessment of the alignment quality. Afterwards, for each threshold, a dictionary was computed, pairing a street from the Bottin data which was not yet aligned with its most similar street from the street dataset (provided it met the threshold). For computation efficiency reasons, only the 10’000 most common streets of the Bottin dataset were considered, amounting to 386’678 of all the data. For the alignment, a new column in the Bottin dataset was created, mapping each street entry to its counterpart in the dictionary (if it existed). Afterwards, this column was aligned with the long street data from the street dataset. Here, the short street names were neglected, as previous short analysis showed that the quality of the alignment was too low.

In the end, 7.43% of the Bottin data could be aligned with fuzzy matching and threshold 85. Afterwards, the threshold 80 was applied, which means that only streets with threshold between 80 and 85 were matched, as the rest was already aligned before. This yielded an additional 3.71% of alignment.

Quality Assessment of Alignment

Number of aligned entries per method combined with the probability of correct alignment (alignment was first conducted by perfect matching, then without spaces and lastly with fuzzy matching)

In order to assess the quality of the alignment, for each method (perfect, without spaces, fuzzy 85 and fuzzy 80), we randomly sampled 100 aligned entries and checked if the match was correct.

For the perfect matching as well as for the matching without spaces, we found that 100% of the matches were correct. This is not surprising, as those methods basically only allow for an alignment if the strings are the same. Hence, we can state with almost certainty that 88% of the aligned data is correct.

The numbers were also good for fuzzy matching using the threshold of 85: Still 82% of the sampled data was matched correctly and 5% had matching short street names, but the street type was not the same (e.g. “avenue Chateaubriand” matched to “Rue Chateaubriand”). The reason was that the street type from the Bottin data did not exist in the street network dataset. For the 13% of wrongly aligned datapoints, the whole street name often could not be found in street network dataset and thus the fuzzywuzzy algorithm matched it another street entirely (e.g. “boul. de la Gare” matched to “Boulevard de la Guyane”).

Those tendencies intensified for the fuzzy matching with threshold of 80, which led to a correct alignment of only 49%. However, one has to take into account that the alignment was only done on the data which had not been matched before with threshold 85. Moreover, 21% of the sampled datapoints were not correctly matched as a whole, but matched on the short street name, while the street type in the Bottin data did not appear in street network dataset. As we could see during the street processing, more than 60% of the streets which had the same name but not the same street type lay very close to each other (buffer of 100m), so there is a high chance that at least half of the streets matched on the name and not the type has a geolocation close to the ground truth. This would raise the correct alignment to around 60%. Finally, 30% of the sample were wrongly classified. This was mainly due to missing street names in the street network dataset and still existing abbreviations in the Bottin data, which complicated matching (e.g. “Fbg-Poissonnière” matched to “Rue Poissonnière”).

All in all, we can calculate the following error rate:

Probability(error) = 0.88 * 0 + 0.08 * 0.18 + 0.04 * 0.51 = 0.0348

This means that with a conservative estimation of the error, we can state that the data is correctly aligned with a probability of more than 96%.

Analysis

Development of Professions

Development of the ratio of the 10 most frequent jobs in the Bottin data

Professions being born and dying out

Development of Streets

Development of the ratio of the 20 most frequent streets in the Bottin data

Case Study: Comparing two Streets

Map of Paris showing the streets of Saint Honoré and Boulevard Voltaire

Development of most frequent jobs at Rue Saint Honoré

Development of most frequent jobs at Boulevard Voltaire

Limitations of the Project

faced some challenges when working on project

OCR Mistakes

• OCR mistakes together with abbreviations leading to wrong fuzzy matching -> idea: use customized distance function which punishes character substitutions/insertions at the end of the string more than at the beginning of the string (EXAMPLE)

Missing streets

• streets not in either one of the two datasets of 1836 or 2022: rue d'Allemagne -> get other street data from 19th century
• given data in Bottin dataset not necessarily a street, e.g. "cloître"

Ambiguous streetnames

• in 1836 street network dataset: datapoints with same streetname, but not located near each other
• many bottin datapoints aligned on "short streetname", while this streetname might refer to many different streets (at point in time when address book published clear?) -> try to incorporate when which street was built

Matching on street level

• worked with centroids of streets in order to represent each data point, which especially for long streets is imprecise -> additional step to clean and align address numbers

Small part of potential analysis

• time constraint: semester project, already resources for alignment
• knowledge constraint: get more (historical) knowledge about Paris to be able to put analysis into context

Outlook

Perfection of Alignment include address numbers, include more street network datasets, two-step alignment of short streetname and then type of street, other distance function with heavier penalty for substitution at the end of string, align on whole datapoint (name, profession, street) to account for businesses existing more than one year

• get more street data, maybe other dataset from 19th century
• if short street name of street dataset contained in a given bottin entry, then do perfect match and try to find corresponding street type afterwards
• or: write custom ratio which punishes substitutions, deletions or insertions in last part of string more, as probability of mistake/non-alignment in first part of string higher (see experience of fuzzy matching, often abbreviations at beginning cause problems)

Possible Directions of Research

• Work on Professions: Cluster in thematic fields, maybe classify by social reputation (-> gentrification of a quartier)
• Analysis: gentrification, development of arrondissements, influence of political decisions of social/economic landscape,...
• Visualization: interactive map

-> good alignment needed to derive facts/knowledge from data