Terry toolkit sdk for tkit_scrapy_mongo ,
Project description
tkit_scrapy_mongo
use mongodb instead of redis and rewrite scrapy_redis py3 module
# 这里覆盖默认的settings配置,custom_settings settings.py
custom_settings = dict(
#MONGODB_URI="mongodb://192.168.1.18:27017/",
#REDIS_URL = 'redis://192.168.1.18:6379', # 指定redis的地址
#POOL_PROXY = "http://192.168.1.18:5010"
CONCURRENT_REQUESTS_PER_IP = 4,
CONCURRENT_REQUESTS_PER_DOMAIN = 4,
DUPEFILTER_CLASS = 'pet.tkit_scrapy_mongo.dupefilter.RFPDupeFilter',
# 去重复
SPIDER_MIDDLEWARES={
# "scrapy_splash.SplashDeduplicateArgsMiddleware": 100,
"pet.tkit_scrapy_mongo.dupefilter.RFPDupeFilter":101
}
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for tkit_scrapy_mongo-0.0.0.116654862.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4ba3e9b797c5259af1d363ac863bc7d9f9d8f5497f3cba0ace1a708ac1c268f |
|
MD5 | 85e4e096c94557e2d59acaaffe72ed83 |
|
BLAKE2b-256 | fd838813087686c6ac0bcf5993fe502bae8430987b0b436159218ed210775d87 |
Close
Hashes for tkit_scrapy_mongo-0.0.0.116654862-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 398ab1cedba514b549b204836e561c9a4c2307bb3822b89b8e177867c27e6a30 |
|
MD5 | e8598831dba35c1b838317c200b94ed8 |
|
BLAKE2b-256 | aa58ea51d68ced8f1dec47d689ab7016c315a002382d90ac30081342770ba4a7 |