selenium implicitly wait doesn't work

Question

This is the first time I use selenium and headless browser as I want to crawl some web page using ajax tech.

The effect is great, but for some case it takes too much time to load the whole page(especially when some resource is unavailable),so I have to set a time out for the selenium.

First of all I tried set_page_load_timeout() and set_script_timeout(),but when I set these timeouts, I won't get any page source if the page doesn't load completely, as the codes below:

driver = webdriver.Chrome(chrome_options=options)
driver.set_page_load_timeout(5)
driver.set_script_timeout(5)
try:
    driver.get(url)
except Exception:
    driver.execute_script('window.stop()')

print driver.page_source.encode('utf-8')  # raise TimeoutException this line.

so I try to using Implicitly Wait and Conditional Wait, like this:

driver = webdriver.Firefox(firefox_options=options, executable_path=path)
print("Firefox Headless Browser Invoked")
wait = WebDriverWait(driver, timeout=10)
driver.implicitly_wait(2)
start = time.time()
driver.get(url)
end = time.time()
print 'time used: %s s' % str(end - start)
try:
    WebDriverWait(driver, 2, 0.5).until(expected.presence_of_element_located((By.TAG_NAME, 'body')))
    print driver.find_element_by_tag_name('body').text
except Exception:
    driver.execute_script('window.stop()')

This time I got the content that I want.However,it takes a very long time(40+ seconds),that means the timeout I set for 2 seconds doesn't work at all.

In my view, it seems like the driver.get() call ends until the browser stop loading the page, only after that the codes below can work, and you can not kill the get() call or you'll get nothing. But this is very different from the selenium docs, I REALLY wonder where is the mistake.

environment: OSX 10.12, selenium 3.0.9 with FireFox & GoogleChrome Headless(both latest version.)

--- update ----

Thanks for help.I change the code as below, using WebDriverWait() alone, but there still exist cases that the call last for a very long time, far more than the timeout that I set. Wonder if I can stop the page load immediately as the time is out?

driver = webdriver.Firefox(firefox_options=options, executable_path=path)
print("Firefox Headless Browser Invoked")
start = time.time()
driver.get('url')
end = time.time()
print 'time used: %s s' % str(end - start)
try:
    WebDriverWait(driver, 2, 0.5).until(expected.presence_of_element_located((By.TAG_NAME, 'body')))
    print driver.find_element_by_tag_name('body').text
except Exception:
    driver.execute_script('window.stop()')
driver.quit()

Here is a terminal output in test:

Firefox Headless Browser Invoked
time used: 44.6049938202 s

according to the code this means the driver.get() call takes 44 seconds to finish call, which is unexpected,I wonder if I misunderstood the behavior of the headless browsers?

Answer 1

As you mentioned in your question it takes too much time to load the whole page(especially when some resource is unavailable) is pretty much possible if the Application Under Test (AUT) uses JavaScript or AJAX Calls.

• In your first scenario you have induced both set_page_load_timeout(5) and set_script_timeout(5)
  • set_page_load_timeout(time_to_wait) : Sets the amount of time to wait for a page load to complete before throwing an exception.
  • set_script_timeout(time_to_wait) : Sets the amount of time that the script should wait during an execute_async_script call before throwing an exception.

Hence the Application Under Test being dependent on JavaScript or AJAX Calls in presence of both the conditions raises TimeoutException.

• In your second scenario you have induced both implicitly_wait(2) and WebDriverWait(driver, 2, 0.5).

  • implicitly_wait(time_to_wait) : Sets the timeout to implicitly wait for an element to be found or a command to complete.
  • WebDriverWait(driver, timeout, poll_frequency=0.5, ignored_exceptions=None) : Sets the timeout in-conjunction with different expected_conditions
  • But you are experiancing a very long timeout(40+ seconds) as it is clearly mentioned in the docs Do not mix implicit and explicit waits which can cause unpredictable wait times

WARNING : Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times. For example setting an implicit wait of 10 seconds and an explicit wait of 15 seconds, could cause a timeout to occur after 20 seconds.

Solution :

The best solution would be to remove all the instance of implicitly_wait(time_to_wait) and replace with WebDriverWait() for a stable behavior of the Application Under Test (AUT).

---

Update

As per your counter question, the current code block looks perfect. The measurement of time which you are seeing as time used: 44.6049938202 s is the time required for the Web Page to load completely and functionally that is the time required for the Client (i.e. the Web Browser) to return back the control to the WebDriver instance once 'document.readyState' equals to "complete" is achieved. Selenium or as an user you have no control on this rendering process. However for a better performance you may follow the best practices as follows :

• Keep your JDK version updated currently Java SE Development Kit 8u162
• Keep your Selenium Client version updated currently selenium 3.9.0
• Keep your WebDriver version updated.
• Keep your Web Browser version updated.
• Clean you Project Workspace within your IDE regularly to build your project with required dependencies only.
• Use CCleaner tool to wipe away the OS chores before and after your Test Suite execution.
• If your Web Browser base version is too old uninstall the Web Browser through Revo Uninstaller and install a recent GA released version of the Web Browser.
• Execute your Test.