Data Ingestion of Guide Commericiale

Project Timeline & Milestones

Timeframe	Task	Completion
Week 4	Explore and define possible approaches to the problem Explore the existing Guida Commerciale documents	✓
Week 5	Establish a pipeline for the project Explore pricing plans for OpenAI GPT models and Geocoding platforms Set up the GitHub repository	✓
Week 6	Start to experiment with different OCR models and potential alternatives, such as PDF text extraction libraries Attempt to cluster pages based on different page types using an unsupervised learning approach	✓
Week 7	Autumn Break	✓
Week 8	Create a new approach/pipeline that relies on basic domain knowledge to obtain better results Set up the semantic annotation platform, INCEpTION, for the new approach	✓
Week 9	Complete midterm presentation Run OCR with Pytesseract Extract text using PDFPlumber Compare the two text extraction approaches, choose the most suitable approach, and pre-process the text output	✓
Week 10	Set up existing model of annotations completed on the 1864 Guida Commericiale Use INCEpTION to complete semantic annotation and named-entity recognition of 1853 Guida Commericiale Convert this into a CSV file for further processing	✓
Week 11	Complete processing and cleaning of the CSV file Occupation Broadcasting Logical Segmentation Parish/Church Reconciliation Complete mapping of parishes to districts (sestiere)	✓
Week 12	Plot the district and number on a map for all the entries in the 1853 Guida Commericiale Obtain accuracy metrics including accuracy of PDFPlumber and the entire pipeline Finalize potential approaches to analysis
Week 13	Complete analysis based on the map entries plotted Derive conclusions based on the analysis and report findings
Week 14	Complete the written deliverables and the GitHub repository Prepare the final presentation discussing the results

Methodology

During our work, we approached our problem in two ways: in a more generalizable way that could process any guide commercial, and a more streamlined way that would require more manual annotation to improve results. We started with our first approach, [and worked on this for the first couple months], but came to realize its margin for error was for too high, making us pivot toward our second approach. The points below highlight each step of the way for both these approaches.

Approach 1: General

Pipeline

1. Divide pages into groups with usable data and ones without (never got done)
2. Process pages with clean data by:
   1. Convert batch of pages into images
   2. Pre-process images for OCR using CV2 and Pillow
   3. Use Pytesseract for OCR to convert image to string
3. Perform Named Entity Recognition by:
   1. Prompt GPT-4o to identify names, professions, and addresses, and to turn this into entries in a table format.
   2. Standardize addresses that include abbreviations, shorthand, etc.
   3. Append “Venice, Italy” to the end of addresses to ensure a feasible location
4. Geocode addresses by:
   1. Use a geocoding API like LocationHQ to convert the address to coordinates
   2. Take top result of search, and append to the map

Explanation

Testing how does this looke mwdasjkldasljdasl jlkdajskl dajkdj akjd ajkd jaskdj sakjd lka

Approach 2: Streamlined

1. Manually inspect entire document for page ranges with digestible data
2. Perform text extraction using PDF Plumber
3. Semantic annotation with INCEpTION, separating our pages into entries with first names, last names, occupations, addresses, etc.
4. Clean and format data?
   1. What were the steps?
5. Map parish to provinces using dictionary
6. Plot on a map using parish and number from entry
7. Perform data analysis.

Limitations & Future Work

Github Repository

Data Ingestion of Guide Commericiale