amo2kinto 1.1.0

Generate a blocklists.xml file from the Kinto collections.

amo2kinto

Installation

To install the release:

pip install amo2kinto

To install the development environment:

make install

System dependencies

In order to build amo2kinto dependencies you may need the following libraries:

• Building lxml will require: libxml2-dev libxslt-dev python-dev

• Building cryptography will require: build-essential libffi-dev libssl-dev python-dev

Otherwise you will run into errors like:

• libxml/xmlversion.h: File not found

• ffi.h: File not found

• ssl.h: File not found

Importing blocklists from the addons server into Kinto

The addons-server is able to export its blocklists database in JSON.

The script will create the bucket/collection with a given schema.

The last version of the schema file can be find here: https://github.com/mozilla-services/amo-blocklist-ui/blob/master/amo-blocklist.json

You can use the make update-schemas command to fetch its last version.

You can use the json2kinto script to load this database into a Kinto server:

json2kinto --server http://localhost:8888/v1 \
           --addons-server https://addons.mozilla.org/  \
           --schema-file schemas.json

The script is able to synchronize (add new blocked items and remove old ones).

However if a blocked item already exists it won’t be altered.

json2kinto gives you the ability to configure what you want to load, with which user credentials and from which bucket and collection.

usage: json2kinto [-h] [--addons-server ADDONS_SERVER] [-s SERVER] [-a AUTH]
                  [-v] [-q] [-D] [-S SCHEMA_FILE] [--no-schema]
                  [--certificates-bucket CERTIFICATES_BUCKET]
                  [--certificates-collection CERTIFICATES_COLLECTION]
                  [--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
                  [--addons-bucket ADDONS_BUCKET]
                  [--addons-collection ADDONS_COLLECTION]
                  [--plugins-bucket PLUGINS_BUCKET]
                  [--plugins-collection PLUGINS_COLLECTION]
                  [-C] [-G] [-A] [-P]

Import the blocklists from the addons server into Kinto.

optional arguments:
  -h, --help            show this help message and exit
  --addons-server ADDONS_SERVER
                        The addons server to import from
  -s SERVER, --server SERVER
                        The location of the remote server (with prefix)
  -a AUTH, --auth AUTH  BasicAuth token:my-secret
  -v, --verbose         Show all messages.
  -q, --quiet           Show only critical errors.
  -D, --debug           Show all messages, including debug messages.
  -S SCHEMA_FILE, --schema-file SCHEMA_FILE
                        JSON Schemas file
  --no-schema           Should we handle schemas
  --certificates-bucket CERTIFICATES_BUCKET
                        Bucket name for certificates
  --certificates-collection CERTIFICATES_COLLECTION
                        Collection name for certificates
  --gfx-bucket GFX_BUCKET
                        Bucket name for gfx
  --gfx-collection GFX_COLLECTION
                        Collection name for gfx
  --addons-bucket ADDONS_BUCKET
                        Bucket name for addons
  --addons-collection ADDONS_COLLECTION
                        Collection name for addon
  --plugins-bucket PLUGINS_BUCKET
                        Bucket name for plugins
  --plugins-collection PLUGINS_COLLECTION
                        Collection name for plugin
  -C, --certificates    Only import certificates
  -G, --gfx             Only import GFX drivers
  -A, --addons          Only import addons
  -P, --plugins         Only import plugins

Generate a blocklist.xml file from Kinto collections

If you want to export blocklists stored in Kinto in the addons server XML export format, you can use the kinto2xml script:

kinto2xml -s http://localhost:8888/v1

kinto2xml gives you the ability to configure what you want to export and in which bucket and collection are the data stored.

usage: kinto2xml [-h] [-s SERVER] [-a AUTH] [-v] [-q] [-D]
                 [--certificates-bucket CERTIFICATES_BUCKET]
                 [--certificates-collection CERTIFICATES_COLLECTION]
                 [--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
                 [--addons-bucket ADDONS_BUCKET]
                 [--addons-collection ADDONS_COLLECTION]
                 [--plugins-bucket PLUGINS_BUCKET]
                 [--plugins-collection PLUGINS_COLLECTION] [--app APP]
                 [-o OUT]

Build a blocklists.xml file from Kinto blocklists.

optional arguments:
  -h, --help            show this help message and exit
  -s SERVER, --server SERVER
                        The location of the remote server (with prefix)
  -a AUTH, --auth AUTH  BasicAuth token:my-secret
  -v, --verbose         Show all messages.
  -q, --quiet           Show only critical errors.
  -D, --debug           Show all messages, including debug messages.
  --certificates-bucket CERTIFICATES_BUCKET
                        Bucket name for certificates
  --certificates-collection CERTIFICATES_COLLECTION
                        Collection name for certificates
  --gfx-bucket GFX_BUCKET
                        Bucket name for gfx
  --gfx-collection GFX_COLLECTION
                        Collection name for gfx
  --addons-bucket ADDONS_BUCKET
                        Bucket name for addons
  --addons-collection ADDONS_COLLECTION
                        Collection name for addon
  --plugins-bucket PLUGINS_BUCKET
                        Bucket name for plugins
  --plugins-collection PLUGINS_COLLECTION
                        Collection name for plugin
  --app APP             Targeted blocklists.xml APP id
  -o OUT, --out OUT     Output XML file.

Show differences between two XML files

Once you’ve imported the data and exported them, you may want to validate that both Kinto and addons server export the exact same data in XML.

You can use the xml-verifier command to validate that both files are rendered the same way.

The xml-verifier command accept both files path and files URLs.

APPID="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}"
kinto2xml --app $APPID -o generated-blocklists.xml
xml-verifier https://blocklist.addons.mozilla.org/blocklist/3/$APPID/46.0/ generated-blocklists.xml

CHANGELOG

This document describes changes between each past release.

1.1.0 (2016-05-18)

Bug fix

• In case there is a diff using xml-verifier, the command now returns an error code (#28)

Internal changes

• json2kinto does not set destination collections as publicly readable (#27)

1.0.0 (2016-05-12)

Breaking changes

• kinto2xml was renamed amo2kinto (#21)

• New JSON Schema file format with a “collection” prefix (#22)

New features

• Use the schema to validate AMO records in the importer script. (#5)

• Warn if the server does not have the schema capability (#24)

Internal changes

• Document amo2kinto commands: json2kinto, kinto2xml and verifier. (#23)

0.1.0 (2016-04-27)

Initial version

• Create collection with the definition of the JSON schema.

• Fetch AMO blocklists information from the /blocked/blocklists.json AMO endpoint.

• Handle import configuration on the CLI. - Bucket / Collection names - Remote AMO endpoint configuration - Schema file path configuration - Schema or not schema - Verbosity level - Server selection - Auth credentials - Importation type selection

• Support for kinto-signer triggering

• Full SSL support for Python 2.7

• Full Python 2.7 and Python 3.4/3.5 support

• Handle the enabled flag to activate records

• Makefile rule to update the schema definition

• Export kinto blocklists in XML blocklist file version 2

• Export kinto blocklists in XML blocklist file version 3

• XML verifier that create a diff of two XML files