Original link: https://www.kingname.info/2023/08/20/scrapy-redis-no-crawler/
Many students use Scrapy + scrapy_redis to implement distributed crawlers when writing crawlers. However, scrapy_redis has been updated less and less in recent years, and it feels quite old. With many updates of Scrapy, scrapy_redis can no longer keep up.
When you install Scrapy, if you do not specify a specific version, the latest version will be installed by default.
If some students have installed the latest version of Scrapy and scrapy_redis in the past two days, the following error will appear after running:
1 |
TypeError: crawl() got an unexpected keyword argument 'spider' |
As shown below:
In this case, the solution is very simple, just don’t install the latest version of Scrapy. When installing with pip , bind the Scrapy version:
1 |
python3 -m pip install scrapy==2.9.0 |
This article is transferred from: https://www.kingname.info/2023/08/20/scrapy-redis-no-crawler/
This site is only for collection, and the copyright belongs to the original author.