Domains blocklist aggregator
Project description
Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
See the blocklist-domains repository for an implementation.
Table of contents
Installation
If you want to generate your own unified blocklist, install this module with the pip command.
pip install blocklist_aggregator
Configuration
See the default configuration file
The configuration contains:
- the ads/tracking/malware URL lists with the pattern (regex) to use
- the domains list to exclude (whitelist)
- additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(ext_cfg=cfg_yaml)
Basic fetch example
This basic example enable to get a unified list of domains. You can save-it in a file or do what you want.
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
Fetch and save-it to files
This module can be used to export the list in several format:
- text
- hosts
- CDB (key/value database)
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
For developpers
Run test units
python3 -m unittest discover tests/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for blocklist_aggregator-0.7.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25965142470837fb5b4ef32b8c96d364885969906ac0cdcbcf857e8270caf57c |
|
MD5 | 7e1c68d58cfa335b3fc475737a41bcc9 |
|
BLAKE2b-256 | b9e08096ec1ed377cda3c8b40dba40626cf6c015488f02be71d6cf24723580ee |
Close
Hashes for blocklist_aggregator-0.7.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c7281a82294f54b2851f2462c009764c23d83077f0e231317b5840fe11b59cd |
|
MD5 | 6a49cad05941b9fcec4b966ad6f84404 |
|
BLAKE2b-256 | 9fd670781d778c6ab8580bcd2366985440861a1df74d1b43867abf9f0b1bd2a3 |