scrapydweb 0.9.9

Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization

ScrapydWeb: Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization

Features

• Scrapyd Cluster Management

  • Group, filter and select any numbers of nodes
  • Execute command on multinodes with one click

• Scrapy Log Analysis

  • Stats collection
  • Progress visualization
  • Logs categorization

• All Scrapyd API Supported

  • Deploy project, Run Spider, Stop job
  • List projects/versions/spiders/running_jobs
  • Delete version/project

• Others

  • Auto eggifying
  • Basic auth for web UI
  • Accessing Scrapyd servers protected by basic auth

Devolopers

• my8100
• simplety (Front-End)

Installation

To install ScrapydWeb, simply use pip:

pip install scrapydweb

Start Up

1. Run scrapydweb -h to get help, and a config file named scrapydweb_settings_vN.py (N for a number) would be copied to current working directory, then you can custom settings in it.
2. Run scrapydweb
3. Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)

Settings

default_settings.py

Screenshots

• Overview

• Dashboard

• Log Analysis

  • Stats collection

  • Progress visualization

  • Logs categorization

• Deploy a Project

• Run a Spider

• Manage Projects