Domains blocklist aggregator
Project description
Blocklist aggregator
This python module does the aggregation of several ads/tracking/malware lists, and merges them into a unified list with duplicates removed.
See the blocklist-domains repository for an implementation.
Default sources:
- winhelp2002.mvps.org
- adaway.org
- StevenBlack
- urlhaus.abuse.ch
- pgl.yoyo.org
- someonewhocares.org
- notracking
- davidonzo/Threat-Intel
- mitchellkrogza/Badd-Boyz-Hosts
- PolishFiltersTeam/KADhosts
- lists.disconnect.me
- notracking/hosts-blocklists
- easylist.to
- paulgb/BarbBlock
Table of contents
Installation
If you want to generate your own unified blocklist, install this module with the pip command.
pip install blocklist_aggregator
Get started
This basic example enable to get a unified list of domains. You can save-it in a file or do what you want.
import blocklist_aggregator
unified = blocklist_aggregator.fetch()
print(unified)
[ "doubleclick.net", ..., "telemetry.dropbox.com" ]
print(len(unified))
152978
Custom configuration
See the default configuration file
The configuration contains:
- the ads/tracking/malware URL lists with the pattern (regex) to use
- the domains list to exclude (whitelist)
- additionnal domains list to block (blacklist)
The configuration can be overwritten at runtime.
cfg_yaml = "verbose: true"
unified = blocklist_aggregator.fetch(cfg_update=cfg_yaml)
or loaded from external config file
unified = blocklist_aggregator.fetch(cfg_filename="/home/custom-blocklist.conf")
Fetch and save-it to files
This module can be used to export the list in several format:
- text
- hosts
- CDB (key/value database)
import blocklist_aggregator
# fetch domains
unified = blocklist_aggregator.fetch()
# save to a text file
blocklist_aggregator.save_raw(filename="/tmp/unified_list.txt")
# save to hosts file
blocklist_aggregator.save_hosts(filename="/tmp/unified_hosts.txt", ip="0.0.0.0")
# save to CDB
blocklist_aggregator.save_cdb(filename="/tmp/unified_domains.cdb")
For developpers
Run test units
python3 -m unittest discover tests/ -v
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for blocklist_aggregator-1.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e362e5a0eba3ceb6195b1c64b3ffa15b2ebae2d17c9e1ab56defb26fec21912 |
|
MD5 | f7924213ded2b2678157449a663517cd |
|
BLAKE2b-256 | eb96142f4f6d1c4e4b128e0fe1108f80e43281066374a59c76919f96ebbc66bb |
Hashes for blocklist_aggregator-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8b5da399a2428d5dc0c970f8dbbbcd2736cecceb8805d9f0483b2aabdc22883 |
|
MD5 | 525fd3ac2e5dd9810cbff057585b1514 |
|
BLAKE2b-256 | ee093b7348742b82ee344ab358a68c9f7f9badbe58b5cea137f937247f2c5036 |