Selenium
So I finally came to a point where Selenium was of some use for me. I needed to automate a journey around the site I was working on for something else and Selenium ended up being the solution that worked.
I first came across Selenium about 15 years ago when we hired a contractor to build a site to update articles in the db. The site would be for internal use to allow editors to make simple changes (corrections etc) to articles that we had live on our sites. Seemed a simple enough task and the dev wanted to use test driven development which sounded great. Three months later and all we had were some Selenium tests and a half functioning site. Of course, we’d all been bored senseless about how amazing Selenium was…and I’ve avoided it like the plague till now.
I still don’t believe Selenium is the answer to everything, but for specific use cases, it can be of use.
Getting started
I decided to use Python to create my script. Once that was done the next decision to make was what to browser to use to run the Selenium script…and I picked Chrome as it’s what the project used.
So then you reach the first potential issue, how to keep your driver up to date. You see, the chromium driver will only work for a certain version of chrome on your machine. When Chrome updates, you’ll have to update your chromium driver as well…independently. Updating the browser doesn’t update the driver so its very easy to get the out of sync and hit errors. A similar situation will probably apply to other browsers, but I didn’t really look into them.
To start with install the selenium module
pip install selenium
Download the chromium driver to your project directory, you’ll need to reference it soon.
Create your Driver
So now, you’ve picked a language to use and a browser to work with. Now you can start coding. All actions are sent to the driver - browsing to a page, entering text into a form, clicking buttons etc
from selenium import webdriver
from selenium.webdriver.common.keys import keys
import time
driver = webdriver.Chrome('.chromedriver')
This creates the driver and the keys import allows you to emulate keyboard presses.
Browse your site
Once you have your driver created, you can start to browse a site. Lets say you want to do a google search for elephants
driver.get("https://www.bing.com/")
search = driver.find_by_name("q")
search.send_keys("elephant")
search.send_keys(Keys.RETURN)
You can find elements in the page either by name, element id or xpath query. The above code find the element by the name of “q” which is the search box on the Bing page. The .send_keys method allows you to enter text into text boxes.
# identifying the radio button with xpath then click
driver.find_element_by_xpath("//input[@value='Female']").click()
# identifying the checkbox with xpath then click
browser.find_element_by_xpath(".//*[@id='Marketing-opt-in-box']").click()
# identifying the dropdown with the id
ddelement= Select(driver.find_element_by_id('id_of_element'))
# select the value
ddelement.select_by_value('1')
Selenium also includes methods to select radio buttons, checkboxes and dropdowns.
Things to make life easier
Now that’s pretty much the basics. You may notice that selenium can try to select/click things before they appear on the screen. This will throw errors and there are ways around this.
The most basic way is to add in a “wait” between calls, but this can be tricky to time right. There are two Selenium methods that can help with this: WebDriverWait and Expected Conditions.
WebDriverWait will tell Selenium to wait for a max of x seconds to find the matching element before throwing a TimeoutException. ExpectedConditions are a list of predefined conditions that you can just plug in and use.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
...
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.NAME, "username"))).send_keys("myusername")
This will wait for 10 seconds max for the element with then name “username” to be clickable and add the value “myusername” into it. Again, there are methods for the elements that you will find on the page.
Too many browsers open
Once your script runs, you will want to close the browser with a driver.quit(). But if you end up with something throwing an error the code never gets to this statement and you’ll find that you have lots of browsers open in the background. Simply enclose the code in a try catch finally block.