Domains blocklist aggregator
Project description
Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
See the blocklist-domains repository for an implementation.
Default sources:
- winhelp2002.mvps.org
- adaway.org
- StevenBlack
- urlhaus.abuse.ch
- pgl.yoyo.org
- someonewhocares.org
- notracking
- davidonzo/Threat-Intel
- mitchellkrogza/Badd-Boyz-Hosts
- PolishFiltersTeam/KADhosts
- lists.disconnect.me
- notracking/hosts-blocklists
- easylist.to
- paulgb/BarbBlock
Table of contents
Installation
If you want to generate your own unified blocklist, install this module with the pip command.
pip install blocklist_aggregator
Get started
This basic example enable to get a unified list of domains. You can save-it in a file or do what you want.
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
Custom configuration
See the default configuration file
The configuration contains:
- the ads/tracking/malware URL lists with the pattern (regex) to use
- the domains list to exclude (whitelist)
- additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)
or loaded from external config file
unified = blocklist_aggregator.fetch(cfg_filename="/home/custom-blocklist.conf")
Fetch and save-it to files
This module can be used to export the list in several format:
- text
- hosts
- CDB (key/value database)
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
For developpers
Run test units
python3 -m unittest discover tests/ -v
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for blocklist_aggregator-0.8.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03cf1194a5ad05d37c580873ca56c7c9d483a8b353345bdf223429a44f8a54e7 |
|
MD5 | ac5a296235a55c0f6b208f6bbe4dcee1 |
|
BLAKE2b-256 | f5b338f95056823ba22ac1e9f7279bbb55fa0e4c3dc71bbab08d4d26ac9f74ab |
Hashes for blocklist_aggregator-0.8.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c20e274abb4939d108d46d73ba89689d77cd6cbeb71a69bd131b5c1da20f6772 |
|
MD5 | e4b652c65fd6a49ec32238950b36cc9a |
|
BLAKE2b-256 | c2b4f6124cfb78e3cd4cbe7e00c048229059bca8fae480bf05ed320daac0dfea |