I'm making a website scraper for a project I'm doing. I've gotten everything to work great, however, loading the actual page takes F-O-R-E-V-E-R. You can see the page it's loading here: MCServerList.Net
Here is the code I am using:
private CONST REQUESTURL as string = "http://www.MCServerList.net/?page="
private chunkId as int32 = 1
Dim req As HttpWebRequest = WebRequest.Create(REQUESTURL & chunkId)
Dim res As HttpWebResponse = req.GetResponse()
Dim Stream As Stream = res.GetResponseStream()
I开发者_Go百科 then use "Stream" and load it through the HTMLAgilityPack found free online. It loads the page quickly, however, the initial request usually takes ~20-30 seconds.
Any help would be appreciated!
I just ran the following code and ignoring the first initial compile I average about 3.3 seconds for GetResponse() and 0.2 more seconds for Load(). Are you on a fast connection? Are you sure this is where the bottleneck is?
Option Explicit On
Option Strict On
Imports System.Net
Public Class Form1
    Private Const REQUESTURL As String = "http://www.MCServerList.net/?page="
    Private chunkId As Int32 = 1
    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        Dim ST As New System.Diagnostics.Stopwatch()
        ST.Start()
        Dim req = WebRequest.Create(REQUESTURL & chunkId)
        Dim res = req.GetResponse()
        Trace.WriteLine(String.Format("GetResponse() : {0}", ST.Elapsed))
        Using Stream As System.IO.Stream = res.GetResponseStream()
            Dim X As New HtmlAgilityPack.HtmlDocument()
            X.Load(Stream)
        End Using
        Trace.WriteLine(String.Format("Load() : {0}", ST.Elapsed))
        ST.Stop()
    End Sub
End Class
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
 加载中,请稍侯......
      
精彩评论