A light and fast python Web crawler framework based on asyncio.
Project description
Distributed🌍 - Asynchronous🏃 - Light☁️ - Fast⚡️ - Easy😄
AirSpider🕷️, a Light and Fast Python Web Crawler Framework Based on Redis🕷️
Overview👀
- AirSpider is a high-performance asynchronous crawler framework for developers 🚀
- Based on Redis: task distribution, task deduplication, and distributed ☁️
Requirements☁️
- Python 3.6➕
- Works on Linux, Windows, macOS🍎
Features🌲
- Quick to Start ☑️
- Low Coupling ☑️
- High Cohesion ☑️
- Easy Expansion ☑️
- Orderly Workflow ☑️
Installation🔨
---------------------------
# For Linux && MacOS🔥
pip3 install airspider
---------------------------
# For Windows🔥
pip3 install airspider
---------------------------
Documents🔥
- Quick Start🕷️
- API Guide👨🎓
- Plugins🔨
- Examples☁️
- Architecture🏠
TODO✈️
- Complete Plugins of Redis🔥
- Complete Distributed Architecture☁️
Contributing👬
AirSpider🕷️ is still under Developing🔨
Feel free to open issues💬 and pull requests💗
- Report or Fix bugs🌈
- Build Powerful plugins🔥
- Make documentation Better📖
- Add Examples of Crawling 🕷️
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
AirSpider-2.0.0.tar.gz
(16.3 kB
view hashes)
Built Distribution
Close
Hashes for AirSpider-2.0.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d10617d3641b2c7fddeece89e86d06226cb5ef1697b506ec3f0155da1dc28b2 |
|
MD5 | 4b3767751242a2ab63c1feaf3fa5e790 |
|
BLAKE2b-256 | e592803e794aca1f7637a7de4b114ec4738093b15ec78ebe35bc2e8ec8dcf454 |