Domains blocklist aggregator
Project description
Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
See the blocklist-domains repository for an implementation.
Default sources:
- winhelp2002.mvps.org
- adaway.org
- StevenBlack
- urlhaus.abuse.ch
- pgl.yoyo.org
- someonewhocares.org
- notracking
- davidonzo/Threat-Intel
- mitchellkrogza/Badd-Boyz-Hosts
- PolishFiltersTeam/KADhosts
- lists.disconnect.me
- notracking/hosts-blocklists
- easylist.to
- paulgb/BarbBlock
Table of contents
Installation
If you want to generate your own unified blocklist, install this module with the pip command.
pip install blocklist_aggregator
Get started
This basic example enable to get a unified list of domains. You can save-it in a file or do what you want.
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
Custom configuration
See the default configuration file
The configuration contains:
- the ads/tracking/malware URL lists with the pattern (regex) to use
- the domains list to exclude (whitelist)
- additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)
or loaded from external config file
unified = blocklist_aggregator.fetch(cfg_filename="/home/custom-blocklist.conf")
Fetch and save-it to files
This module can be used to export the list in several format:
- text
- hosts
- CDB (key/value database)
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
For developpers
Run test units
python3 -m unittest discover tests/ -v
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for blocklist_aggregator-0.9.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f5d07ba90fc99d97278b650e639ca29890904af9dc504978366e1dca086415f |
|
MD5 | 7893d832e2e08bff31836f653b914331 |
|
BLAKE2b-256 | 10d8822a09d31a9ddf0e134077c2087a254c02842afd357a3bb784655a9ec31d |
Hashes for blocklist_aggregator-0.9.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cfa016818e991d85c04575b3bb0e6278bc3e0340c52ae0ee6c29c5024d7bbc9 |
|
MD5 | cdc89a0447ece18619cd2c330943d5a1 |
|
BLAKE2b-256 | b70b78cdc57342c84bbf5ac025037491104c9c5ff9685a4afbec77bf68c71943 |