An Easy-to-use and Fast Python Spider Framework
Project description
Distributed - Asynchronous - Easy-to-use
An Easy-to-use and Fast Python Spider Framework
Overview
- AirSpider is a high-performance asynchronous crawler framework for developers
- Based on Redis: task distribution, task deduplication, and distributed ☁️
Requirements
- Python 3.6
- Works on Linux, Windows, macOS
Features
- **Quick to Start **
- **Low Coupling **
- **High Cohesion **
- **Easy Expansion **
- **Orderly Workflow **
Installation
# For Linux && MacOS
pip3 install airspider
# For Windows
pip3 install airspider
Documents
- Introduction:Introduction to AirSpider
- Tutorials: Quick to program a Spider By AirSpider
- Plugins:Program extensions for AirSpider
Topics
- Item:Define Target in HTML
- Selector:Select Target from HTML
- Request:Request Part
- Response:Response Part
- Middleware:Make Airspider Support to Extensions
- Spider:Entrance for AirSpider
TODO
- Complete Plugins of Redis
- Complete Distributed Architecture
Contributing
AirSpider🕷️ is still under Developing
Feel free to open issues💬 and pull requests
- Report or Fix bugs
- Build Powerful plugins
- Make documentation Better
- Add Examples of Spiders
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
AirSpider-2.0.3.tar.gz
(16.0 kB
view hashes)
Built Distribution
Close
Hashes for AirSpider-2.0.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc3998cd5a57fcd87297b9458b0e1afaf28b55c2817488663e03026c4fe887cd |
|
MD5 | adde52f1b9e5b18447d9a123449fe347 |
|
BLAKE2b-256 | f0c5104981c4829de65a73cb9fc963be4c9e43989c5fbc80f42dce7569574fab |