开发者

Python cookielib and urllib2 max connections?

开发者 https://www.devze.com 2023-03-18 18:34 出处:网络
I have made a URL scanner that relies on cookielib and urllib2 to scan webpages. I have noticed that every time I reach 100 connections that the program just stops with no error. I am assuming the err

I have made a URL scanner that relies on cookielib and urllib2 to scan webpages. I have noticed that every time I reach 100 connections that the program just stops with no error. I am assuming the error is because I've hit 100 connections. I have tried various times on different domains and eventually the program will stop investigating links and stop once it hits 100 outgoing connections. How do you get around this error?

My setup code is as follows:

domain = "http://dotwhat.net"
cj = cookielib.CookieJar()
开发者_开发百科opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
html = opener.open(domain).read()
soup = BeautifulSoup(html)

I open up a new connection on line 4 of the code in a loop.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号