Gallica wrapper: Difference between revisions

From FDHwiki
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
= Gallica wrapper =


== Basics of Gallica API ==
== Basics of Gallica API ==
Line 28: Line 27:


It is directly used in the main Gallica website, thus by observing the URL when doing a search, it is possible to infer the corresponding API call. The results are paginated up to 50 records per page (15 by default), it is thus necessary to handle the pagination.
It is directly used in the main Gallica website, thus by observing the URL when doing a search, it is possible to infer the corresponding API call. The results are paginated up to 50 records per page (15 by default), it is thus necessary to handle the pagination.
== Using the wrapper ==
=== Installation ===
The wrapper requires python 3.
* Clone the repository with <code>git clone https://github.com/raphaelBarman/fdh-gallica.git</code>
* First its requirements should be installed (using <code>pip install -r requirements.txt</code> or with conda <code>conda install --file requirements.txt</code>).
* Then the wrapper package can be installed using <code>python setup.py install</code>.
=== Command line tools ===
The wrapper contains two command line tools: <code>gallica_exporter.py</code> and <code>gallica_download.py</code>.
==== gallica_exporter.py ====
This tool allows to generate a [https://en.wikipedia.org/wiki/Comma-separated_values CSV] of urls and paths to the metadata, IIIF images and OCR of a given document/periodical/search.
Type <code>python gallica_exporter.py -h</code> to have more information about the different options.
The main idea is that it either takes as input either the ark of a document or a periodical either a search query and generates a list of all the relevant urls. The paths will form a hierarchy with the year at the top level for the periodical and then for each document a folder with its ark name and inside an xml with its OAI metadata, an "images" folder with its jpg images and an "alto" folder with its xml ALTO OCR (if it exists).
This can generate a lot of results, so be sure to check the file before launching the downloader script.
==== gallica_download.py ====
This tool will take the list built by the previous tool and download all the files. This is done in parallel, hopefully BNF's server are strong enough to support you download, but if you see a lot of errors, maybe reduce the number of processes ("-p" flag).
Once again, type <code>python gallica_download.py -h</code> to have more information about the different options.
=== API wrapper ===

Revision as of 14:30, 8 October 2019

Basics of Gallica API

ARK identifier

ARK (Archival Resource Key) is a unique identifier for any resource. It has for goal of creating a strong link between its id and the object. It works on the following principle:

  • Responsibility of the organization for permanent access to its resources.
  • Possibility to add a suffix to interrogate on the metadata of the resource.
  • Opacity of its ID so that it never needs to be changed even if new properties of the resource arise.
  • Complete control on all published IDs to ensure no duplicates.

An ARK has the following form: ark:/12148/bpt6k205233j, where

  • ark: is the scheme
  • 12148 is the Name Assigning Authority Number (NAAN), a unique number given to each institution by the California Digital Library (CDL). This particular number is the one of the BNF.
  • bpt6k205233j the ARK name which is the identifier to the resource of the institution of the NAAN.

ARK is the ID scheme used at the BNF and is thus used in the wrapper to identify periodical and documents. For more information on the usage of ARK a the BNF c.f. this page

Document API

The Document API allows finding information on periodicals and on documents.

Periodical

The API gives all the years and dates of their publication as well as the ARKs of the documents.

Documents

The API gives metadata of the document under the form of an OAI XML that gives dublin core properties. These properties contain many valuable information, such as authors, dates, subjects, etc.

The API can also be used to find the pagination of a document, which in turn can give access to its OCR under the XML ALTO format and its IIIF images.

Search API

Gallica offers a research API that uses the SRU norm. The details will not be explicit here but can be found on this Gallica page (in french).

It is directly used in the main Gallica website, thus by observing the URL when doing a search, it is possible to infer the corresponding API call. The results are paginated up to 50 records per page (15 by default), it is thus necessary to handle the pagination.

Using the wrapper

Installation

The wrapper requires python 3.

  • Clone the repository with git clone https://github.com/raphaelBarman/fdh-gallica.git
  • First its requirements should be installed (using pip install -r requirements.txt or with conda conda install --file requirements.txt).
  • Then the wrapper package can be installed using python setup.py install.

Command line tools

The wrapper contains two command line tools: gallica_exporter.py and gallica_download.py.

gallica_exporter.py

This tool allows to generate a CSV of urls and paths to the metadata, IIIF images and OCR of a given document/periodical/search.

Type python gallica_exporter.py -h to have more information about the different options.

The main idea is that it either takes as input either the ark of a document or a periodical either a search query and generates a list of all the relevant urls. The paths will form a hierarchy with the year at the top level for the periodical and then for each document a folder with its ark name and inside an xml with its OAI metadata, an "images" folder with its jpg images and an "alto" folder with its xml ALTO OCR (if it exists).

This can generate a lot of results, so be sure to check the file before launching the downloader script.

gallica_download.py

This tool will take the list built by the previous tool and download all the files. This is done in parallel, hopefully BNF's server are strong enough to support you download, but if you see a lot of errors, maybe reduce the number of processes ("-p" flag).

Once again, type python gallica_download.py -h to have more information about the different options.

API wrapper