Automating web applications and websites

Web automation tools allow you to optimize and automate a lot of processes that almost every office employee must perform from time to time, among them:

• Filling in online forms
• Scraping data from web pages
• Generating and downloading online reports
• Checking information online

Automating these web tasks can allow you to spend time on more valuable work and outsource the tedious parts to bots. Let’s see how you can use RPA tools for web automation and what practices to apply to ensure success.

Web Automation approaches

When it comes to web automation, there are several common approaches, depending on the task, the resource you need to automate and your coding skills (or lack thereof). In this post, we will focus on the codeless automation capabilities, which don’t require any programming skills and are available for business users who want to automate their web tasks.

Mimicking user actions 

One of the ways to automate tasks performed by a person on a web page is to mimic the actions they do, e.g. hover over menu elements, click on buttons, etc. In this case, a bot would repeat a person’s behavior exactly, action by action.

Interacting with elements in the background

Using the special pre-built actions, you can program your bot to perform some tasks that you do online, but faster and more efficiently, by having the interactions with web elements occur through special locators: XPaths. In this case, the actions the bot takes can differ from the actions a person would do in this situation.

Let’s see how these two approaches can be used in a web automation script, based on Example 2 from the post “How a Bot Thinks,” published recently.

Task: Fill in and submit an online form

Person’s actions:

1. Open the browser
2. Type in the URL and press ENTER
3. Click on the field that needs to be filled in
4. Type the required information
5. Repeat actions 3 and 4 for each field, scrolling if required
6. Click the Submit button

You can program your bot to perform the same actions as a person would do in this situation. However, let’s see what the best way to automate a web form would be.

Bot’s actions:

1. Launch the required URL in the required browser
2. Paste the required information in the field using XPaths
3. Repeat action 2 for each field
4. Click the Submit button

You can see there are similarities and differences in the person’s and the bot’s actions. There is no need for the web automation bot to launch the browser first and then type in the URL of the page it needs to navigate to. It can be done in a single action. Also, there is no need to click on the field and then type the data as the bot can insert the data in the required field (or another web element) by using special Web actions and XPath locators. There is also no need to scroll the page as the bot can find elements outside the visible part of the webpage. At the same time, the bot clicks on the Submit button just as a human does (mimics the human behavior), although it finds it on the page by XPath, too.

From the workflow above, it is obvious that understanding the differences between human and bot behavior will allow you to not only automate actions on the web but also optimize them to make more efficient. So, before automating your process, break it down in actions, and analyze which of them needs the bot to mimic the human behavior, and which can be performed more efficiently using special web automation capabilities of the tool. It will help you automate better.
 

Web Automation best practices

Here are some other practices you should follow when automating websites and web applications, especially when using Intelligent Automation Cloud.

Store credentials in Secrets Vault

If the bot needs to log into the automated web application in the process, it is a good security practice to store the necessary username and password in a special component known as Secrets Vault. It is also advisable to create special credentials for the bot rather than use an existing employee’s username and password.

Store action parameters in variables

If possible, store such parameters as URLs, XPaths, and others, in String variables, and don’t hard-code them in the actions. It will make it easier to audit and change the script when required.

Use relative XPaths over absolute ones

The structure of the web page that you automate can change with time, thus changing the absolute XPaths of its elements. As a result, your web automation script might break. On the other hand, elements’ attributes, such as “ID” or “name”, rarely change, so it is a good idea to use relative XPaths in your scripts. If a relative XPath is not generated in Dev Tools, you can create it manually or use special add-ons in the browser.

Pay attention to iframes

When people are working with websites, we don’t care if the button we press or the form we fill in is in an iframe. However, it is important to a bot. When building your web automation scripts, always check for iframes on the page and provide all parent iframes’ XPaths in the web actions.

Try to work with just one page at a time

Often, you need to automate several websites or applications — for example, to copy information from a website and use it to fill in a web form elsewhere. In this case, try to work with only one page at a time, rather than switch from one page to another and back. Gather all data from the first page, and then open the web form and start filling it in. It will make the script more transparent and easier to maintain, and the process faster.

Provide fluent waits in the script

In case the Internet connection can be slow, or if the web application can sometimes take a longer time to load than usual, it is a good idea to provide fluent waits (timeouts) in the action’s properties.

This is better than using hard waits, as that way, the bot will not need to wait before moving on if, for example, the application has loaded faster than expected. 

Examples of automating web applications and web pages

To help business users get started faster with web automation, we have prepared several workflows that you can download and run to see how web automation works. (The following links lead to more detailed pages in our Knowledge Base.)

Translation: Translates a list of English phrases into Spanish (or another language of your choice) using Google Translate service. This shows how to use loops when automating web applications.

Craigslist: Conducts a search on the Craigslist website with key phrases, returns 10 top results and saves them in Excel. This shows how searching for information online can be automated.

Web scraping: Collects press releases from the U.S. Securities and Exchange Commission’s site for the latest seven days, saves the information in an Excel file and sends the file via Outlook. Demonstrates web scraping and email capabilities.

Salesforce: Uploads a .CSV file with contact information to a Salesforce account with two-factor authentication.

To learn more about automating web applications with Intelligent Automation Cloud, we suggest you take this free course: Automating the Business Task.