Data Ingestion of Guide Commericiale

Project Timeline & Milestones

Timeframe	Task	Completion
Week 4	Explore and define possible approaches to the problem Explore the existing Guida Commerciale documents	✓
Week 5	Establish a pipeline for the project Explore pricing plans for OpenAI GPT models and Geocoding platforms Set up the GitHub repository	✓
Week 6	Start to experiment with different OCR models and potential alternatives, such as PDF text extraction libraries Attempt to cluster pages based on different page types using an unsupervised learning approach	✓
Week 7	Autumn Break	✓
Week 8	Create a new approach/pipeline that relies on basic domain knowledge to obtain better results Set up the semantic annotation platform, INCEpTION, for the new approach	✓
Week 9	Complete midterm presentation Run OCR with Pytesseract Extract text using PDFPlumber Compare the two text extraction approaches, choose the most suitable approach, and pre-process the text output	✓
Week 10	Set up existing model of annotations completed on the 1864 Guida Commericiale Use INCEpTION to complete semantic annotation and named-entity recognition of 1853 Guida Commericiale Convert this into a CSV file for further processing	✓
Week 11	Complete processing and cleaning of the CSV file Occupation Broadcasting Logical Segmentation Parish/Church Reconciliation Complete mapping of parishes to districts (sestiere)	✓
Week 12	Plot the district and number on a map for all the entries in the 1853 Guida Commericiale Obtain accuracy metrics including accuracy of PDFPlumber and the entire pipeline Finalize potential approaches to analysis
Week 13	Complete analysis based on the map entries plotted Derive conclusions based on the analysis and report findings
Week 14	Complete the written deliverables and the GitHub repository Prepare the final presentation discussing the results

Methodology

During our work, we approached our problem in two ways: in a more generalizable way that could process any guide commercial, and a more streamlined way that would require more manual annotation to improve results. We started with our first approach, [and worked on this for the first couple months], but came to realize its margin for error was for too high, making us pivot toward our second approach. The points below highlight each step of the way for both these approaches.

Approach 1: General

Pipeline Overview

Pipeline Overview This process aims to extract, annotate, standardize, and map data from historical documents. By leveraging OCR, named entity recognition (NER), and geocoding, the pipeline converts unstructured text into geographically and semantically meaningful data.

Pipeline

1. Process pages by:
   1. Convert batch of pages into images
   2. Pre-process images for OCR using CV2 and Pillow
   3. Use Pytesseract for OCR to convert image to string
2. Perform Named Entity Recognition by:
   1. Prompt GPT-4o to identify names, professions, and addresses, and to turn this into entries in a table format.
   2. Standardize addresses that include abbreviations, shorthand, etc.
   3. Append “Venice, Italy” to the end of addresses to ensure a feasible location
3. Geocode addresses by:
   1. Use a geocoding API like LocationHQ to convert the address to coordinates
   2. Take top result of search, and append to the map

Divide Up Pages

Since we wanted this solution to work for any guide commercial, not just the ones provided to us, we intended to develop a way to parse through a document, identifying pages of interest that had usable data. While we got a working solution that was able to weed out pages with absolutely no information, we were never able to find a reliable solution for finding pages with the specific type of data we wanted. A possible idea we had was to train a model for identifying patterns within each page and grouping them off of that, but due to timeline restrictions we decided against it. Because of this, we never ended up finalizing this step for approach 1, and proceeded with the rest of our pipeline.

Perform OCR

To start, we used PDF Plumber for extracting pages from our document in order to turn them into images. With the goal of extracting text from these pages, we then used libraries such as CV2 and Pillow in order to pre-process each image with filtering and thresholding to improve clarity, and lastly performed OCR using Pytesseract, tweaking certain settings to accommodate the old scripture better.

Named Entity Recognition

After having extracted all text from a page, we can then prompt GPT-4o to identify names, professions, and addresses, and turn our text into a table format with an entry for each person. Furthermore, since these documents use abbreviations and shorthand for simplifying their writing process, we need to undo these by replacing them with their full meaning. Finally, to make geocoding in the future possible, we need to specify that each entry belongs to Venice by adding "Venice, Italy" to the end of each address.

Geocoding

For geocoding, we found LocationHQ to be a feasible solution for our cause, successfully mapping addresses to close by locations within Venice. In order to implement this into our project, we attempted to query each address in our table. While some queries narrowed down the request to a single address, some would result in multiple possible options. Our heuristic for these scenarios was to simply pick the top option, as it seemed to be the closest match each time. Finally, after gathering the coordinates for our addresses, we would map them onto a visual map that we ended up not proceeding with in the end.

Approach 2: Streamlined

Pipeline Overview

This approach outlines a systematic process for extracting, annotating, and analyzing data from historical Venetian guide commercials. The aim is to convert unstructured document data into a clean, structured dataset that supports geographic mapping and data analysis.

Pipeline

1. Manually inspect entire document for page ranges with digestible data
2. Perform text extraction using PDF Plumber
3. Semantic annotation with INCEpTION, separating our pages into entries with first names, last names, occupations, addresses, etc.
4. Clean and format data?
   1. What were the steps?
5. Map parish to provinces using dictionary
6. Plot on a map using parish and number from entry
7. Perform data analysis.

Find page ranges of interest

The main difference with this approach compared to approach 1 was that instead of relying on a program to group our pages, we manually went through our document and found a range of pages that were of interest to us. This allowed us to go into the next steps of our pipeline with confidence, knowing we don't need to deal with the uncertainty of whether our page had irrelevant data or not.

Perform Text Extraction

Furthermore, after testing both OCR and text extraction, we found that using PDF Plumber's function for extracting text from the PDF directly worked much more reliably. Because of this, we ended up using this for all of our pages, resulting in a similar output to the OCR with a much higher accuracy. This also freed us up from requiring to perform pre-processing for images, making the pipeline much simpler.

Semantic Annotation

For semantic analysis, we steered toward using INCEpTION in place of GPT-4o, largely due to previous projects already having trained a model for extracting names, addresses, and professions from similar text documents. Another large benefit from this was that we no longer needed to rely on an external API for this, allowing us to bypass the limitations with financing our requests. [ADD MORE DETAIL HERE AND REWRITE MAYBE???]

Data Clean Up

[SAHIL WRITE THIS PART PLS]

Plot on a Map

WE HAVNET DONE THIS YET

Data Analysis

WE HAVEN'T DONE THIS YET

Limitations & Future Work

Github Repository

Data Ingestion of Guide Commericiale