开发者

C# htmlagility pack, capturing redirct

开发者 https://www.devze.com 2023-03-10 05:11 出处:网络
HI all, this one is really simple (I hope).I\'m using htmlagility pack to do my webcrawling.So what happens if I input url whatever, that then directs开发者_StackOverflow中文版 me to a new url, how do

HI all, this one is really simple (I hope). I'm using htmlagility pack to do my webcrawling. So what happens if I input url whatever, that then directs开发者_StackOverflow中文版 me to a new url, how do I capture that new redirected URL?

If htmlagility pack doesnt have a way, can someone suggest another method?


Using the HtmlWeb class that comes with the Html Agility Pack, you can tweak the request before it's actually executed, like this:

    HtmlWeb web = new HtmlWeb();
    web.PreRequest = OnPreRequest;
    HtmlDocument doc = web.Load("http://wwwblablahh.com");


private static bool OnPreRequest(HttpWebRequest request)
{
    request.AllowAutoRedirect = true;
    return true;
}


When you create your HttpWebRequest you can set AllowAutoRedirect property to true and it will automatically follow any redirects you have.

HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://www.contoso.com");  
myHttpWebRequest.MaximumAutomaticRedirections=1;
myHttpWebRequest.AllowAutoRedirect=true;
HttpWebResponse myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse(); 

you can find more info at msdn

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号