Domains blocklist aggregator
Project description
Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
See the blocklist-domains repository for an implementation.
Default sources:
- winhelp2002.mvps.org
- adaway.org
- StevenBlack
- urlhaus.abuse.ch
- pgl.yoyo.org
- someonewhocares.org
- notracking
- davidonzo/Threat-Intel
- mitchellkrogza/Badd-Boyz-Hosts
- PolishFiltersTeam/KADhosts
- lists.disconnect.me
- notracking/hosts-blocklists
- easylist.to
- paulgb/BarbBlock
Table of contents
Installation
If you want to generate your own unified blocklist, install this module with the pip command.
pip install blocklist_aggregator
Get started
This basic example enable to get a unified list of domains. You can save-it in a file or do what you want.
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
Custom configuration
See the default configuration file
The configuration contains:
- the ads/tracking/malware URL lists with the pattern (regex) to use
- the domains list to exclude (whitelist)
- additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)
or loaded from external config file
unified = blocklist_aggregator.fetch(cfg_filename="/home/custom-blocklist.conf")
Fetch and save-it to files
This module can be used to export the list in several format:
- text
- hosts
- CDB (key/value database)
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
For developpers
Run test units
python3 -m unittest discover tests/ -v
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for blocklist_aggregator-0.9.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d562295f775ba1897baeb66b8ef7aaad9964173af9f77d26e753441bf93c6802 |
|
MD5 | 9ddcfe5d3178b09647f0e9237a618a1b |
|
BLAKE2b-256 | a4ab834dd835f4e3b58cb5ee3625e5c302fc22bdbab74bc700fe22190341ea01 |
Hashes for blocklist_aggregator-0.9.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 041debaffc75a9b3612b067c5575f2a98b396f6fccd6bbe66c42edf40c9525c9 |
|
MD5 | 7f9e2ebf124f50068f3e2ffe4554b57c |
|
BLAKE2b-256 | 674d41fc6d46b8324a8637b3c7c21d3394db1fc08ab13169e2fd4aa89b11911e |