Selenium Usage Notes

Since I have been experimenting with Selenium recently, I thought I would make a note of how to use Selenium so that if I forget how to use it, I can remember again.

install

Selenium

pip install selenium

The browser and the WebDriver that drives the browser must be installed separately.

WebDriver

• WebDriver must be installed that matches the browser version.
• Since it is time-consuming to do it manually, use webdriver_manager for automatic installation.

pip install webdriver-manager

How to use webdriver_manager

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install())

sample

Example: Search for "test" on Google and view the source on the second page of results.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.implicitly_wait(30)

driver.get('https://www.google.com/')
driver.find_element(By.CSS_SELECTOR, 'input[name="q"]').send_keys("test\n")
driver.find_element(By.CSS_SELECTOR, '#pnnext').click()
print(driver.page_source)

driver.quit()

Element (selection)

The flow of the browser operation is to acquire the target dom and repeat the operation on the dom.

Auto standby

• When dom is obtained, dom may not have been generated yet.
• Set to automatically wait until the target dom is generated.
• If dom is not found until the set time, an error occurs.

# selector auto-wait 30sec
driver.implicitly_wait(30)

CSS Selector

from selenium.webdriver.common.by import By
driver.find_element(By.CSS_SELECTOR, 'button').click()

Selector Extension

Calling another selector from the selector searches for the child dom.

# same as 'div input'
driver.find_element(By.CSS_SELECTOR, 'div')\
    .find_element(By.CSS_SELECTOR, 'input').click()

plural element

A selector may match multiple elements.

• driver.find_element() returns the first element found
• driver.find_elements() returns list for all elements found

elems = driver.find_elements(By.CSS_SELECTOR, 'a')
for elem in elems:
    print(elem.get_attribute('href'))

Element (operation/attribute)

Operation

• webelement.click()

  • click

• webelement.send_keys(<string>)

  • Text Entry Power
  • In the case of input, value is set.
  • If type of input is file, the file path is set.

• webelement.clear()

  • input string clear

Selecting options for radio buttons, checkboxes, and selects is done with webelement.click().

scroll

If you get the bottom element in the window and refer to webelement.location_once_scrolled_into_view, it will scroll until you see that element.

• Since webelement.location_once_scrolled_into_view is a property, you only need to refer to it.
• Internally, scrollIntoView() is called on the element

driver.find_element(By.CSS_SELECTOR, '#bottom-elem').location_once_scrolled_into_view

Properties

• webelement.text

• webelement.get_attribute(<attribute_name>)

  • Property Value Acquisition

  • Attribute Example

    • name
    • id
    • value

• webelement.is_selected()

  • Get the selection status of radio button, check box, and select options

Example (optional)

# inverse option
options = driver.find_elements(By.CSS_SELECTOR, 'select option')
for option in options:
    if option.is_selected():
        option.click()

navigation

transition

• driver.get(<url>)
• driver.refresh()

• driver.quit()

  • Call it whenever you exit the browser.

Page load completion is detected by using driver.find_element() to see if a specific element of the page has been found.

Properties

• driver.title
• driver.current_url
• driver.page_source

frame

• driver is tied to a single frame
• To manipulate elements in iframe within html, use driver.switch_to.frame(<frame_elem>) to switch the frame that driver points to
• Use driver.switch_to.default_content() to restore the original frame

frame = driver.find_element(By.CSS_SELECTOR, "iframe")
driver.switch_to.frame(frame)
driver.find_element(By.CSS_SELECTOR, "button").click()
driver.switch_to.default_content()

Tab (window)

How to handle the case where a new tab (window) is opened with <a target="_blank">.

• When a tab is created, a window is generated. In other words, tab and window are the same.
• driver is operated by one window.
• Tab list is in driver.window_handles
• The current window_handle is acquired by driver.current_window_handle
• Use driver.switch_to.windoe(<window_handle>) to switch the target window of driver

Example

Manipulate an element in a newly opened tab, then return to the original tab and manipulate the element again.

• Wait until tabs are generated and the number of window_handle is increased.
• Extract the window_handle of the new tab from the difference of the window_handle list before and after the tab is created.

from selenium.webdriver.support import expected_conditions as EC

handle_count = len(driver.window_handles)
before_handles = driver.window_handles
original_handle = driver.current_window_handle

# open new tab
driver.find_element(by=By.CSS_SELECTOR, value='#new_tab').click()

# wait for new window created
WebDriverWait(driver, 30).until(
    EC.number_of_windows_to_be(handle_count+1))

# find new window handle
after_handles = driver.window_handles
diff_handles = list(set(after_handles) ^ set(before_handles))

# switch new tab
driver.switch_to.window(diff_handles[0])
driver.find_element(By.CSS_SELECTOR, 'button').click()

# switch original tab
driver.switch_to.window(original_handle)
driver.find_element(By.CSS_SELECTOR, 'button').click()

accept-language

Not possible with Selenium functionality. It is set in the browser startup options.

options = webdriver.ChromeOptions()
options.add_experimental_option('prefs', {'intl.accept_languages': 'ja'})
driver = webdriver.Chrome(
    ChromeDriverManager().install(),
    options=options,
)

Browser state saving

Usually, the browser starts up in a new state each time.

Specifying a user data storage location in the browser startup options allows the browser state to be saved and reused.

This is used when opening a multi-factor authentication page, for example.

options = webdriver.ChromeOptions()
options.add_argument('--user-data-dir=<user_data_dir>')
driver = webdriver.Chrome(
    ChromeDriverManager().install(),
    options=options,
)

File Download

File downloading cannot be done with Selenium functionality alone and requires a browser-specific implementation for each browser.

What Selenium cannot do

• Retrieving the path of a downloaded file
• Acquisition of download completion status

procedure

Google Chrome

• Specify the download destination folder in the browser startup options

• Watch for updates to the file being downloaded and retrieve the download completion and path to the downloaded file

  • Use watchdog for file update detection
  • Temporary files are created when downloading, so they are excluded from detection

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

import os
import re
import time


class ChromeDownloadHandler(FileSystemEventHandler):
    def __init__(self, observer):
        self.result = None
        self.observer = observer
        self.tmpfiles = \
            [re.compile('^\.com\.google\.Chrome\..+$'),
             re.compile('^.+\.crdownload$')]

    def on_closed(self, event):
        if event.is_directory == True:
            return

        basename = os.path.basename(event.src_path)
        for tmpfile in self.tmpfiles:
            if re.match(tmpfile, basename) != None:
                return

        self.result = event.src_path
        self.observer.stop()


# setup browser
download_dir = os.path.abspath('./download')
options = webdriver.ChromeOptions()
options.add_experimental_option('prefs', {
    'download.default_directory': download_dir,
})

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=options,
)
driver.implicitly_wait(30)

# setup watchdog
observer = Observer()
handler = ChromeDownloadHandler(observer)
observer.schedule(
    handler,
    download_dir,
    recursive=False
)
observer.start()

# file download
driver.get('https://the-internet.herokuapp.com/download')
driver.find_element(By.CSS_SELECTOR, '#content a').click()

# wait for file download finish
try:
    while observer.is_alive():
        time.sleep(1)
except KeyboardInterrupt as e:
    observer.stop()
    observer.join()
    raise (e)

# file download finished
observer.join()

# print download file path
print(handler.result)

driver.quit()

file upload

form

• Set webelement.send_key() to the absolute path of the file to be uploaded to <input type="file" />

driver.find_element(By.CSS_SELECTOR, 'input[type="file"]').send_key(<full_file_path>)
driver.find_element(By.CSS_SELECTOR, 'input[type="submit"]').click()

What Selenium cannot do

The following types of file uploads cannot be reproduced by Selenium functionality alone.

• Click to open a file selection dialog, select a file and upload it.
• Drop and upload files

In these cases, there may be hidden forms on the page, so try the same method as for the aforementioned forms.

JavaScript

Basic

• JavaScript can be executed in the browser with driver.execute_script().
• Return value can be returned by return
• Return value is a scalar, or JSON, or a reference to a JavaScript object in the browser
• If the return value is JSON, it is dict

res = driver.execute_script("""
    return 'abc';
    """)

# abc
print(res)

res = driver.execute_script("""
    return { val1: 123, val2: 'abc' };
    """)

# {'val1': 123, 'val2': 'abc'}
print(res)

argument

• Putting arguments after a JavaScript statement allows you to pass arguments to JavaScript
• The value is stored in the arguments array on the JavaScript side

res = driver.execute_script("""
    return { val1: arguments[0], val2: arguments[1] };
    """, 123, 'abc')

# {'val1': 123, 'val2': 'abc'}
print(res)

References to JavaScript objects

• When a JavaScript object is passed as the return value, a reference to the JavaScript object is returned.
• Passing a reference to a JavaScript object to driver.execute_script() allows you to reference it in JavaScript

Example

html

<input name="test" value="before" />

code

# print->"before"
print(driver.find_element(By.CSS_SELECTOR, 'input[name="test"]').get_attribute('value'))

obj = driver.execute_script("""
    return document.querySelector('input[name="test"]');
    """)

# print->"<selenium.webdriver.remote.webelement.WebElement (session="...", element="...")>"
print(obj)

driver.execute_script("""
    arguments[0].value = 'after';
    """, obj)

# print->after
print(driver.find_element(By.CSS_SELECTOR, 'input[name="test"]').get_attribute('value'))

driver.find_element()

Since the driver.find_element() returned is also a reference to a JavaScript object, it can be referenced in JavaScript by passing it as an argument to driver.execute_script() as in the previous example.

# print->"before"
print(driver.find_element(By.CSS_SELECTOR, 'input[name="test"]').get_attribute('value'))

obj = driver.find_element(By.CSS_SELECTOR, 'input[name="test"]')

# print->"<selenium.webdriver.remote.webelement.WebElement (session="...", element="...")>"
print(obj)

driver.execute_script("""
    arguments[0].value = 'after';
    """, obj)

# print->after
print(driver.find_element(By.CSS_SELECTOR, 'input[name="test"]').get_attribute('value'))

document

• https://www.selenium.dev/

  • Official page
  • A quick read of WebDriver in the documentation will give you a basic idea of how to use Selenium.

• https://www.selenium.dev/selenium/docs/api/py/index.html

  • reference
  • Use this page as a starting point to examine classes, functions, variables, and arguments

• Particularly frequently referenced pages in the references

  • https://www.selenium.dev/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webdriver.html

    • Reference for driver
    • The driver function and its arguments are described.

  • https://www.selenium.dev/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webelement.html

    • Reference for webelement
    • The webelement function and its arguments are described in this page.

Caution when viewing references

Base class not listed

Click on [source] to see and examine the source.

For example, the base class for selenium.webdriver.chrome.webdriver is found to be ChromiumDriver because the class definition is class WebDriver(ChromiumDriver):.

The location of ChromiumDriver can be found in import, which is from selenium.webdriver.chromium.webdriver import ChromiumDriver, and selenium.webdriver.chromium.webdriver, which is import.

Functions/variables of the base class are not known from the derived class reference

I steadily found the base class in [source] and traced its reference.

Impressions, etc.

Some sites behave differently when run in Chrome's headless mode, so you may want to avoid using headless mode.

Related Category Articles

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com