Skip to main content

An Easy-to-use and Fast Python Spider Framework

Project description

Distributed - Asynchronous - Easy-to-use

An Easy-to-use and Fast Python Spider Framework


Overview

  • AirSpider is a high-performance asynchronous crawler framework for developers
  • Based on Redis: task distribution, task deduplication, and distributed ☁️

Requirements

  • Python 3.6
  • Works on Linux, Windows, macOS

Features

  • Quick to Start
  • Low Coupling
  • High Cohesion
  • Easy Expansion
  • Orderly Workflow

Installation

# For Linux && MacOS
pip3 install airspider


# For Windows
pip3 install airspider

Documents

Topics

TODO

  • Complete Plugins of Redis
  • Complete Distributed Architecture

Contributing

AirSpider🕷️ is still under Developing

Feel free to open issues💬 and pull requests

  • Report or Fix bugs
  • Build Powerful plugins
  • Make documentation Better
  • Add Examples of Spiders

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

AirSpider-2.0.4.tar.gz (16.0 kB view hashes)

Uploaded Source

Built Distribution

AirSpider-2.0.4-py2.py3-none-any.whl (18.8 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page