Simple scraper for Ebay Kleinanzeigen searches with notifications for new ads.
Project description
EK scraper
Simple scraper for Ebay Kleinanzeigen searches with notifications for new ads.
Installation
Clone this repository
git clone git@github.com:jonasehrlich/ek-scraper.git
Change into the repository
cd ek-scraper
Install the repository using
pip3 install .
Usage
For the full usage check the
ek-scraper --help
command
Create a configuration file using
ek-scraper create-config <path/to/config.json>
The example configuration file will look like this:
{
"filter": {
"exclude_topads": true,
"exclude_patterns": []
},
"notifications": {
"pushover": {
"token": "<your-app-api-token>",
"user": "<your-user-api-token>",
"device": []
}
},
"searches": [
{
"name": "Wohnungen in Hamburg Altona",
"url": "https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497",
"recursive": true
}
]
}
Modify the configuration to include your API tokens and optionally the list of devices to send the notifications to. See Notifications for details on configuring multiple notification services.
Modify the search configurations to your liking. It is recommended to only add the first page of your search results
as the url
, because the recursive
attribute will automatically resolve any pagination happening in the search
results.
Run the following command to initialize the data store without sending any notifications
ek-scraper run --no-notifications path/to/config.json
Filter
Filters can be configured in the filter
section of the configuration file to exclude specific ads from your scrape
results on the client side. The following settings can be configured.
Name | Description |
---|---|
exclude_topads |
Whether to exclude top ads from the results (optional, defaults to true) |
exclude_patterns |
Case-insensitive regular expression patterns used to exclude ads (optional) |
Searches
Searches can be configured in the searches
array of the configuration file. Each of the searches can be configured
with the following parameters.
Name | Description |
---|---|
name |
Name of the search, use a descriptive one (required) |
url |
URL of the first page of your search (required) |
recursive |
Whether to follow all pages of the search result (optional, defaults to true) |
Notifications
Notifications can be configured in the notifications
section of the configuration file.
Push notifications using Pushover
ek-scraper
supports push notifications to your devices using Pushover.
For further information on the service check their terms and conditions.
To configure Pushover for notifications from the scraper, first register at the service and create an application
(e.g. ek-scraper
). To use the service in ek-scraper
, add the pushover
object to the notifications
object in your
configuration file and fill the API tokens. Selection of the devices which will receive the notifications, is supported using the device
array.
Name | Description |
---|---|
token |
API token of the Pushover app (required) |
user |
API token of the Pushover user (required) |
device |
List of device names to send the notifications to (optional, defaults to all devices) |
Running ek-scraper
regularly
In order to run ek-scraper
regularly on a Unix-like system, configure it as a cronjob.
To configure a cronjob, run
crontab -e
Edit the crontab table to run the command you want to run. A handy tool to check schedule configurations for cronjobs is crontab.guru.
For more information on configuring cronjobs use your favorite search engine.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ek_scraper-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb6016898bfb5a8c97285fe1aa4a958e8dd2bb5aff7a1ee3e778f621f2d798c7 |
|
MD5 | 73c597932450a95908f27645089130fa |
|
BLAKE2b-256 | 0735c17fadeb833cf5fdadec4ba80c2375fc0af92b2797587551119c9f374825 |