Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
298 views
in Technique[技术] by (71.8m points)

在Scrapy的Downloader Middleware中使用Selenium下载页面时,如何使用异步?

在下载中间件中,Request请求似乎是以同步的方式传入process_request方法的,而我在这里使用selenium下载页面并返回Response对象,就不会使用Scrapy的下载器。最后,全程只有一个浏览器实例在运行,每次只能处理一个请求,请问应该如何处理以提高爬取效率?非常感谢。


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)
等待大神答复

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...