oarepo-oai-pmh-harvester 2.0.0a14

oarepo OAI-PMH converter.

oarepo-oai-pmh-harvester

OAI-PMH Client for Invenio under OArepo brand.

Installation

Library is stored in PyPi repository, so it is commonly installed through pip.

pip install oarepo-oai-pmh-harvester

Configuration

Data harvesting must be set in the configuration (invenio.cfg or via app.config). All settings are made via the OAREPO_OAI_PROVIDERS key. Config is a dictionary where the key is the provider code and each provider can have several individual settings / jobs called synchronizer.

OAREPO_OAI_PROVIDERS={
            "provider-name": {
                "description": "Short provider description",
                "synchronizers": [
                    {
                        "name": "xoai",
                        "oai_endpoint": "https://example.com/oai/",
                        "set": "example_set",
                        "metadata_prefix": "oai_dc",
                        "bulk": True,
                        "from": "latest",
                        "unhandled_paths": ["/dc/unhandled"],
                        "default_endpoint": "recid",
                        "endpoint_mapping": {
                            "field_name": "doc_type",
                            "mapping": {
                                "record": "recid"
                            }
                        },
                    }
                ]
            },
        }

Parameters:

• description: Test description of provider
• synchronizers: Dictionary with individual settings

  • name: name of the setting/synchronizer

  • oai_endpoint: URL adress

  • set: name of OAI set

  • metadata_prefix: name of OAI metadata prefix

  • bulk: if bulk is True, the records are requested by ListRecords method (OAI-PMH protocol), if bulk is False every record is requested by GetRecord method(OAI-PMH protocol). Default is True.

  • from: the date from which the synchronization starts, it is optional parameter with three options:

    1. latest: checks for changes since the last sync
    2. date [YYYY-MM-DD]: check for change since entered date
    3. none: parameter is omitted, then start full synchronization

  • unhandled_paths: List of paths in json that are not handled by any rule.It must be specified, otherwise the client will report an error that the path was not processed by any rule.

  • default_endpoint: The name of the end_point defined in RECORDS_REST_ENDPOINTS from the invenio-records-rest library, which will be used as the base unless otherwise specified.

  • endpoint_mapping: If multiple invenio-records-rest endpoints are used, it is necessary to set rules for which endpoint will be assigned to a particular record. In most cases, an endpoint can be assigned based on a metadata field (field_name) that is assigned a dictionary mapping, where key is the value of the metadata field and the dictionary value is assigned to the endpoint.

Endpoint handler

If endpoint_mapping cannot be expressed using a dictionary, it is possible to use a handler. Handler is a function that receives already transformed data and returns the name rest endpoint. It is registered as entry_point (oarepo_oai_pmh_harvester.mapping) and using the endpoint_handler decorator. More in the example.

entry_points={
       'oarepo_oai_pmh_harvester.mapping': [
           '<name> = example.endpoint_handler',
       ],
   }

@endpoint_handler(<provider>, <metadataprefix>)
def mapping_handler(data):
    resource = data.get("resource"):
    if resource == "book":
        return "book_model"
    elif resource == "journal":
        return "journal"
    else:
        return "recid"

Usage

The package is used to integrate the OAI-PMH client into Invenio. It is a wrapper that is built on the Sickle library. Provides integration with invenio-records. The purpose of the package is to ensure synchronization with a remote OAI-PMH source.

Successful data collection requires several steps, which consist of:

1. Configuration (see configuration chapter)
2. Parser: function that converts XML into JSON
3. Rules: functions that convert raw JSON (from parser) into final JSON

Parsers

A function that transforms XML into JSON (implemented as a python dictionary). The module where the function is located must be specified in entry_points and the function itself marked with a decorator. The function takes lxml.etree._Element as an argument and returns a dictionary.

• entry_points:

The module is registered in entry_points under the keyword oarepo_oai_pmh_harvester.parsers, for example as follows:

entry_points={
       'oarepo_oai_pmh_harvester.parsers': [
           'xoai = example.parser',
       ],
   }

• decorator: The decorator has one parameter, the name of the metadata_format and that must be same as in config metadata_prefix. The function must accept one positional argument (etree._Element) and return a dictionary.

from oarepo_oai_pmh_harvester.decorators import parser

@parser("xoai")
def xml_to_json_parser(etree):
    ...some magic
    return dict_

Rules

The raw parsed JSON is converted to the final JSON in the transformation. The built-in transformer recursively traverses the raw JSON and remaps the raw JSON to the final JSON. The transformer searches all paths to see if a rule exists for that path or if the path is in an unhandled path in the configuration. If it does not meet any of the conditions, it raises an error to warn the user that he has forgotten about a metadata field.

A rule is a function that accepts the el (element) and kwargs (name parameters) arguments and returns the reworked element as a python dictionary. The module that contains the rule must be specified in entry_points and the function itself must be registered using a decorator.

• entry_points:

The module is registered in entry_points under the keyword oarepo_oai_pmh_harvester.rules, for example as follows:

entry_points={
       'oarepo_oai_pmh_harvester.rules': [
           'xoai = example.rule',
       ],
   }

• decorator:

The decorator has four positional arguments and one named argument:

1. provider_name: must be same as in config
2. metadata_prefix: must be same as in config
3. json_path: level is separated with "/"
4. phase:
   • pre: the rule is applied during the creation of the final JSON.
   • post: the rule is applied after the all pre rules

The rule function itself must accept the el (element) and ** kwargs arguments in the signature. El is the JSON value at the given JSON address. It must return dictionary (eg: {"title": "Example title"})

Kwargs contain several useful variables:

• paths: a set containing an absolute JSON path and all subsequent relative levels path eg (/dc/title/en, dc /title/en, title/en, en)
• results: a list of individual results, which will make up the final JSON.
• phase: pre or post phase
• record: raw json as defaultdict

Example of a rule:

from oarepo_oai_pmh_harvester.decorators import rule


@rule("provider_name", "metadata_prefix", "/dc/title/en", phase="pre")
def rule(el, **kwargs):
    value_ = el[0]["value"][0]
    return {"title": value_}

CLI

If all components (config, parser, rules) are set, the program can be run via the CLI:

Usage: invenio oai run [OPTIONS]

  Starts harvesting the resources set in invenio.cfg through the
  OAREPO_OAI_PROVIDERS environment variable.

Options:
  -p, --provider TEXT      Code name of provider, defined in invenio.cfg
  -s, --synchronizer TEXT  Code name of OAI-PMH setup, defined in invenio.cfg
  --break / --no-break     Break on error, if true program is terminated when
                           record cause error

  -o, --start_oai TEXT     OAI identifier from where synchronization begin
  -i, --start_id INTEGER   The serial number from which the synchronization
                           starts. This is useful if for some reason the
                           previous synchronization was interrupted at some
                           point.

  --help                   Show this message and exit.