visual studio - Coded UI Test is slow waiting for UI thread

Question

I've added Coded UI Tests to my ASP.NET MVC solution in Visual Studio 2013. I was dismayed to see how slowly the tests run; each page just sits there for up to a minute or more before the test machinery wakes up and starts filling in the form fields.

After some experimentation (including turning off SmartMatch), I've discovered that simply calling

Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.Disabled;

solves the problem. But, as expected, the test frequently fails because the UI thread isn't ready for the test machinery to interact with the controls on the form.

Calling

Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.UIThreadOnly;

makes the test run reliably, if slowly.

Any thoughts or suggestions? Any hope that someone might have some insight into the magic baked into the WaitForReady machinery? Are there any other settings related to WaitForReady that I can fiddle with besides WaitForReadyLevel?

Answer 1

After a bit of experimentation, I've worked out what appears to be a combination of settings that allows my Coded UI Tests to reliably run at full speed -- faster than I could interact with the website by hand.

Note: The relevant "documentation" (if you call a blog "documentation") can be found here:

• Playback configuration settings
• Retrying failed playback actions.

The trick requires several modifications to the default playback settings:

• Setting WaitForReadyLevel = WaitForReadyLevel.Disabled allows the test to run at full speed. But it also disables the (slow!) magic that waits until it's safe to interact with controls on the page.

• Setting a MaximumRetryCount and attaching an error handler deals with most of the errors that result from disabling the "wait for ready" magic. Because I've baked a 1 second Sleep into the retry logic, this value is effectively the number of seconds I'm willing to wait for the page to load and become responsive.

• Apparently, failure to find the control under test is not one of the errors handled by the error handler/retry mechanism. If the new page takes more than a few seconds to load, and the test is looking for a control that doesn't exist until the new page loads, the test fails to find the control and the test fails. Setting ShouldSearchFailFast = false solves that problem by giving you the full timeout time for your page to load.

• Setting DelayBetweenActions = 500 appears to work around a problem that I see occasionally where the UI misses a button click that occurs immediately after a page has loaded. The test machinery seems to think that the button was clicked, but the web page doesn't respond to it.

• The "documentation" says that the default search timeout is 3 minutes, but it's actually something greater than 10 minutes, so I explicitly set SearchTimeout to 1 second (1000 ms).

To keep all of the code in one place, I've created a class that contains code used by all of the tests. MyCodedUITests.StartTest() is called by the [TestInitialize] method in each of my test classes.

This code really should be executed only once for all of the tests (rather than once per test), but I couldn't figure out a way to get the Playback.PlaybackSettings calls to work in the [AssemblyInitialization] or [ClassInitialization] routines.

/// <summary> A class containing Coded UI Tests. </summary>
[CodedUITest]
public class UI_Tests
{
    /// <summary> Common initialization for all of the tests in this class. </summary>
    [TestInitialize]
    public void TestInit()
    {
        // Call a common routine to set up the test
        MyCodedUITests.StartTest();
    }

    /// <summary> Some test. </summary>
    [TestMethod]
    public void SomeTest()
    {
        this.UIMap.Assert_HomePageElements();
        this.UIMap.Recorded_DoSomething();
        this.UIMap.Assert_FinalPageElements();
    }
}


/// <summary> Coded UI Test support routines. </summary>
class MyCodedUITests
{
    /// <summary> Test startup. </summary>
    public static void StartTest()
    {
        // Configure the playback engine
        Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.Disabled;
        Playback.PlaybackSettings.MaximumRetryCount = 10;
        Playback.PlaybackSettings.ShouldSearchFailFast = false;
        Playback.PlaybackSettings.DelayBetweenActions = 500;
        Playback.PlaybackSettings.SearchTimeout = 1000;

        // Add the error handler
        Playback.PlaybackError -= Playback_PlaybackError; // Remove the handler if it's already added
        Playback.PlaybackError += Playback_PlaybackError; // Ta dah...
    }

    /// <summary> PlaybackError event handler. </summary>
    private static void Playback_PlaybackError(object sender, PlaybackErrorEventArgs e)
    {
        // Wait a second
        System.Threading.Thread.Sleep(1000);

        // Retry the failed test operation
        e.Result = PlaybackErrorOptions.Retry;
    }
}