vb.net - Wait for internet explorer to load everything?

Question

I am scraping a webpage, and waiting for internet explorer to get done loading but for some reason it is not. I'm trying to get a value on the page but the wait part is not waiting therefore the value comes back blank when there should be a value. The IE page has done loading but the value for the elements on the page has not been loaded yet. Is there a way to wait for all elements to get done loading before proceeding to next line of code? Here's my code:

Dim IE As Object 
Dim myvalue as string

IE = CreateObject("internetexplorer.application")
IE.navigate("mypage")

While Not IE.ReadyState = WebBrowserReadyState.Complete
    Application.DoEvents()
End While

myValue = IE.document.getElementById("theValue").getAttribute("value")
Debug.Print(myValue)

Answer 1

You SHOULD NOT use Application.DoEvents() in order to keep your UI responsive! I really can't stress this enough! More than often using it is a bad hack which only creates more problems than it solves.

For more information please refer to: Keeping your UI Responsive and the Dangers of Application.DoEvents.

The correct way is to use the InternetExplorer.DocumentComplete event, which is raised when the page (or a sub-part of it, such an an iframe) is completely loaded. Here's a brief example of how you can use it:

1. Right-click your project in the Solution Explorer and press Add Reference...

2. Go to the COM tab, find the reference called Microsoft Internet Controls and press OK.

   

3. Import the SHDocVw namespace to the file where you are going to use this, and create a class-level WithEvents variable of type InternetExplorer so that you can subscribe to the event with the Handles clause.

And voila!

Imports SHDocVw

Public Class Form1

    Dim WithEvents IE As New InternetExplorer

    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
        IE.Navigate("http://www.google.com/")
    End Sub

    Private Sub IE_DocumentComplete(pDisp As Object, ByRef URL As Object) Handles IE.DocumentComplete
        MessageBox.Show("Successfully navigated to: " & URL.ToString())
    End Sub
End Class

Alternatively you can also subscribe to the event in-line using a lambda expression:

Imports SHDocVw

Public Class Form1

    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
        Dim IE As New InternetExplorer
        IE.Navigate("http://www.google.com/")

        AddHandler IE.DocumentComplete, Sub(pDisp As Object, ByRef URL As Object)
                                            MessageBox.Show("Successfully navigated to: " & URL.ToString())
                                        End Sub
    End Sub
End Class