Robot Framework 101: Comprehensive Overview, Pros & Cons

Anushree Chatterjee

What is Robot Framework?

Robot Framework is an open-source tool used for automating software testing. It’s designed to be simple and easy to use. It is widely used for acceptance testing and acceptance test-driven development (ATDD) and for automation tasks involving testing APIs, GUIs, databases, and more.

The key part of Robot Framework is that it uses something called keywords. These keywords are like simple instructions, such as “Open browser,” “Click on a button,” or “Verify text.” You can create these instructions in a text file, and Robot Framework will run the tests automatically, just like a robot following a list of tasks.

It also supports different technologies and tools, like Selenium (for web testing) or Appium (for mobile testing), making it very versatile. Plus, it provides clear reports after running the tests, so you can easily see if anything is wrong.

Key Features of Robot Framework

Here are some of Robot Framework’s key features:

• Open Source and Free: Robot Framework is open-source software released under the Apache License 2.0, which means it is free to use and modify. This open nature encourages contributions from the community and leads to a wide array of libraries, tools, and extensions.
• Keyword-Driven Testing: In Robot Framework, you write tests using “keywords” that represent actions or checks. These keywords are like simple commands in plain language. These keywords are either built-in or custom-defined.
• Extensibility: Robot Framework can be extended by libraries implemented in Python, Java, or any other language that can run on the Java Virtual Machine (JVM). This extensibility allows for integration with a wide range of tools and technologies like Selenium, Appium, REST APIs and database connectors.
• Support for Parallel Execution: To improve efficiency, Robot Framework supports parallel test execution through tools like Pabot. This allows multiple test cases to run simultaneously.
• Rich Ecosystem: The Robot Framework community has developed a rich ecosystem of libraries and tools that extend its capabilities. Some popular libraries include SeleniumLibrary for web testing, DatabaseLibrary for database operations, and SSHLibrary for SSH connections.
• Data-Driven Testing: The framework supports data-driven testing which lets the execution of the same test cases with multiple sets of data. This is particularly useful for testing applications with various input values and conditions.

Components of a Robot Framework Test

***Settings***
Documentation   This suite tests user login functionality.

***Test Cases***
Valid Login
  Open Browser  ${LOGIN_URL}  ${BROWSER}
  Input Text   ${USERNAME_FIELD}  valid_user
  Input Text   ${PASSWORD_FIELD}  secure_password
  Click Element  ${LOGIN_BUTTON}
  Wait Until Page Contains Element  xpath=//div[@class='dashboard']

Invalid Login - Incorrect Password
  Open Browser  ${LOGIN_URL}  ${BROWSER}
  Input Text   ${USERNAME_FIELD}  valid_use
  Input Text   ${PASSWORD_FIELD}  wrong_password
  Click Element  ${LOGIN_BUTTON}
  Page Should Contain Text  Invalid credentials

Invalid Login - Non-existent User
  Open Browser  ${LOGIN_URL}  ${BROWSER}
  Input Text   ${USERNAME_FIELD}  non_existent_user
  Input Text   ${PASSWORD_FIELD}  some_password
  Click Element  ${LOGIN_BUTTON}
  Page Should Contain Text  User not found

Let’s look at the main components you’ll encounter while working with the Robot Framework:

• Test Case: A test case is a set of steps that describe the actions to be performed and the expected results. It’s the core of testing in Robot Framework. Test cases and their related data are written in a specific format, typically in plain text files with the .robot extension. These files define the tests, their steps, and the data they use. It consists of a name and a series of test steps (keywords). Example (from above): Valid Login, Invalid Login - Incorrect Password, Invalid Login - Non-existent User are individual test cases.
• Test Suites: A test suite is a collection of related test cases. You can organize your tests into multiple test suite files for better management. In the above code example, the entire .robot file represents a test suite focused on login functionality.
• Keywords: Keywords are the building blocks of Robot Framework. They are simple commands that perform specific actions. Robot Framework provides built-in keywords (like Open Browser, Input Text, Click Element, Wait Until Page Contains Element, Page Should Contain Text) and allows you to create custom keywords. These keywords make the tests human-readable and reusable. Example (from above): Open Browser ${LOGIN_URL} ${BROWSER} is a test step that uses the built-in keyword Open Browser.
• Variables: Variables store values that can be reused throughout your test cases or test suites. You can think of them as placeholders for values like URLs, credentials, or test data. Variables help make tests more flexible and maintainable. If you need to change a value, you only need to update it in one place rather than in every test case. Variables can be defined in the ***Variables*** section or directly within test cases. Example:

  ***Variables***
  ${BROWSER}    Chrome
  ${LOGIN_URL}   https://example.com/login
  ${USERNAME_FIELD}  id:username
  ${PASSWORD_FIELD}  id:password
  ${LOGIN_BUTTON}   xpath://button[@type='submit']

  Here, ${BROWSER}, ${LOGIN_URL}, etc., are variables holding specific values.
• Settings: The ***Settings*** section is used to configure the test suite. It includes settings for importing libraries, setting documentation, defining test tags, etc. Example (from above): Documentation sets the description for the test suite, and Library SeleniumLibrary (implicitly imported in the ***Settings*** of the first example) imports the SeleniumLibrary, which provides keywords for web automation.
• Libraries: Libraries are collections of keywords. These can be built-in (like the SeleniumLibrary for web automation) or custom (created by users to perform specific tasks). Libraries are great for extending Robot Framework’s functionality to make it capable of testing different types of applications (web, desktop, APIs, etc.).
• Execution Engine: The execution engine is the core part of the Robot Framework that runs the tests. It takes the test cases and executes them step-by-step using the appropriate keywords. You interact with the execution engine primarily through the command-line interface. You provide the path to your test data files, and Robot Framework takes care of the rest.
• Reports and Logs: After running tests, Robot Framework generates reports and logs that detail the execution results. For example, the HTML report will give you a summary of test results, while the log provides more detailed information (like screenshots, error messages, and step-by-step execution details).
• Output Files: Robot Framework generates three output files by default after running tests:
  • Output.xml: The detailed execution log in XML format.
  • Log.html: A human-readable log showing what happened during the test run.
  • Report.html: A summary report showing which tests passed or failed.

Testing with Robot Framework

Installing Robot Framework

Before you begin, ensure you have the following:

• Python Installed: Robot Framework is written in Python, so you need Python installed on your system. You can check if Python is installed by opening your terminal or command prompt and typing:

  python --version

• pip (Python Package Installer): pip is the package installer for Python and is usually installed along with Python. You can check if pip is installed by typing:

  pip --version

pip install robotframework

Installing Additional Libraries (Robot Framework with Selenium)

pip install robotframework-seleniumlibrary

If you are using SeleniumLibrary, you also need to have a web browser driver installed for the browser you want to automate. You can also install other libraries for different testing needs, like AppiumLibrary for mobile testing, DatabaseLibrary for database testing, etc.

Create a Test Case in Robot Framework

Now that you have Robot Framework installed, let’s write a simple test case.

• Create a New File: Open any text editor (like Notepad, Visual Studio Code, or Sublime Text) and create a new file with the extension .robot. For example, my_first_test.robot.
• Write a Test Case: In the .robot file, write a simple test. Here’s an example that checks if we can open a web browser and go to a website.

  *** Settings ***
  Library SeleniumLibrary
  
  *** Variables ***
  ${URL} http://example.com
  
  *** Test Cases ***
  Open Browser and Visit Example Website
  Open Browser ${URL} Chrome
  Title Should Be Example Domain
  Close Browser

Run the Test

Now that you’ve written your test, you can run it using the following command in the terminal.

• Navigate to the folder where you saved your .robot file. Use the cd command to go to that folder.
• Once you’re in the correct directory, use this command to run the test:

  robot my_first_test.robot

Check the Results

After running the test, Robot Framework generates a log and a report that you can view.

• Log: A detailed log of what happened during the test run (errors, successes, etc.). It’s saved as log.html.
• Report: A summary report that shows the overall results (passed, failed, etc.). It’s saved as report.html.

Both files will be generated in the same folder where you ran the test. To open them, just double-click the files in your file explorer, and they’ll open in your default web browser.

Advantages of Robot Framework

• Free and Open-source: Robot Framework is free and open-source, which makes it an attractive option for businesses, especially startups and smaller companies that cannot afford expensive test automation tools.
• Ease of Use for Automation Engineers: Robot Framework’s keyword-driven approach and readable syntax make it accessible to testers with even basic automation or coding knowledge.
• Extensibility: The framework can be extended with custom libraries written in Python, Java, or other languages. This allows users to tailor it to their specific needs and integrate it with a wide array of tools and services.
• Versatile Testing Capabilities: Robot Framework supports various types of testing, including web, mobile, API and desktop applications, thanks to its extensive library ecosystem, such as SeleniumLibrary for web and AppiumLibrary for mobile testing.
• Cross-Platform Compatibility: It runs on multiple operating systems like Windows, macOS, and Linux and provides flexibility and convenience for teams working in different environments.
• Data-Driven Testing: The framework supports data-driven testing by allowing the separation of test data from test scripts. This enables the reuse of test logic with different data sets and enhances test coverage.
• Comprehensive Reporting: Robot Framework automatically generates detailed logs and reports in HTML format, which helps teams quickly identify test failures and analyze results for debugging and improvements.
• Integration with CI/CD Pipelines: It integrates smoothly with popular CI/CD tools like Jenkins, GitLab CI, and Travis CI. This supports automated testing workflows and promotes continuous testing within development cycles.
• Active Community and Support: A strong, active, open-source community continuously contributes to Robot Framework. This helps keep the framework current and adaptable to new testing needs.

Disadvantages of Robot Framework

• Not Ideal for Complex Test Logic: Robot Framework uses simple keywords to define tests, which works well for basic tasks. However, it can be challenging to write more complex logic or handle sophisticated scenarios. For example, if your test requires complex calculations, custom error handling, or advanced programming logic, Robot Framework might not be the best choice.
• Performance Overhead: The keyword-driven approach, while user-friendly, can introduce performance overhead, especially when dealing with large test suites or highly complex test cases, compared to more code-centric frameworks.
• Limited Control Over Execution: The high-level abstraction of keywords can sometimes limit the tester’s ability to control specific details of the execution flow, which may necessitate creating custom keywords or scripts.
• Dependency on External Libraries: For many advanced testing scenarios, Robot Framework relies heavily on third-party libraries (e.g., Selenium for web testing, Appium for mobile testing), which can add complexity in terms of setup, maintenance, and version compatibility.
• Steeper Learning Curve for Advanced Features: While basic tests are easy to write, leveraging the full potential of Robot Framework, especially for custom extensions or complex integrations, may require in-depth knowledge of its architecture and the underlying programming languages.
• Verbose Test Scripts: Tests in Robot Framework can become verbose, as even simple operations require multiple keywords and arguments, which can make test scripts longer and potentially harder to manage.
• Debugging Challenges: Debugging in Robot Framework can be less intuitive compared to traditional programming-based test frameworks, as the abstraction layer can obscure the details of what is happening at the code level.

Is Robot Framework in Demand?

Robot Framework’s strength lies in its modularity and keyword-driven approach to testing. This makes it a popular pick for many test automation engineers.

However, with better tools in the market like testRigor that utilize AI, you’ll see that Robot Framework falls short in many aspects.

• Automating ATDD
  • Robot Framework isn’t inherently the best choice for ATDD. To implement ATDD effectively in Robot Framework, you need to manually map your business requirements (expressed in Gherkin or similar formats) to the Robot Framework keywords and test steps. This requires a conscious effort to structure your tests in a BDD-like manner. It doesn’t have built-in NLP capabilities to directly understand and translate natural language acceptance criteria into executable tests.
  • On the other hand, testRigor is built ground-up with the user in mind, and hence, can handle ATDD and BDD easily. The tool uses NLP and smart element locators to allow its users to automate test cases in simple English language. testRigor’s syntax is centered around describing user behavior and expected outcomes, directly reflecting acceptance criteria. For example, you might write: “Click ‘Login’ button. Verify that the text ‘Welcome, User’ is visible.”
• Keyword Maintenance
  • The keyword-driven approach works very well until you have to manage a whole lot of them. Imagine the nightmare when test cases start to fail as the keyword values change. Especially given that modern web pages tend to have a lot of dynamic elements that need to be tested. In Robot Framework, you need to manually track and update your keywords to make sure that the element locators being used are the right ones.
  • While testRigor lets you mention “keywords”, it does not rely on you to identify them on the screen. It uses AI for better element locator strategies that can mitigate changes with the implementation details of the UI elements.
• Dependency on Libraries
  • In Robot Framework, to ensure that you are able to test a host of applications (web, mobile, API, desktop, etc.) you need to integrate with different libraries. This can increase the complexity of you test automation process even more.
  • In testRigor, you get all of these capabilities and much more under one umbrella. The tool has built-in libraries to support all of these different types of scenarios in simple English language.
• Challenges with Testing Complex Scenarios
  • Modern applications involve many advanced components like working with OTPs, email verification, CAPTCHA resolution, and more. Trying to do this with Robot Framework, even if you import Selenium, will be tedious.
  • You can automate a vast number of test cases with ease using testRigor. The tool is equipped to take on these challenging test cases.
• Knowledge of Programming
  • You need to know some amount of coding to be able to test using Robot Framework. For example, if you want to add new keywords to the list, someone has to code it. Even using the framework on the whole is not as straightforward as a few clicks. This inevitably leaves out those who have the most understanding of user behavior – manual testers and business analysts.
  • Since testRigor offers plain English-based test creation through its use of AI, everyone can participate in the testing process. The tool even allows plain English as a way for you to write custom logic, unlike most other tools.
• Test Maintenance
  • Using Robot Framework can incur test maintenance overhead, especially if you have many tests to run. This is mainly due to the multiple dependencies involved in making a test case work.
  • On the contrary, testRigor uses AI to reduce test maintenance to a bare minimum. The tool uses AI to cut out the reliance on implementation details of UI elements like XPaths or CSS selectors, auto-heal test steps and reusable rules, and mitigate flakey test runs.

Testing with testRigor

By now, you have already seen why testRigor is the next step ahead for your automation testing needs.

Let’s look at some test case examples to see how testRigor compares to Robot Framework.

Example 1: Searching text in the browser search bar

This is a simple test case to check that we are able to enter and search a string using the search bar provided by the browser.

Robot Framework Test Case

***Settings***
Library      SeleniumLibrary

***Variables***
${BROWSER}    Chrome
${URL}      https://www.example-ecommerce.com # Replace with the actual website URL
${SEARCH_TERM}  Laptop
${SEARCH_INPUT_LOCATOR}  id:search-input # Replace with the actual locator for the search input field
${SEARCH_BUTTON_LOCATOR}  xpath://button[@type='submit'] | //input[@type='submit' and @value='Search'] # Replace with the actual locator for the search button
${RESULTS_CONTAINER_LOCATOR}  css:.search-results # Replace with the actual locator for the container holding search results
${EXPECTED_RESULT_TEXT}  Laptop # Expected text to appear in the results (can be more specific)

***Test Cases***
Verify Successful Search
  Open Browser To Homepage
  Input Search Term
  Submit Search
  Verify Search Results Present
  Verify Search Results Contain Term

***Keywords***
Open Browser To Homepage
  Open Browser  ${URL}  ${BROWSER}
  Maximize Browser Window
  Wait Until Page Contains Element  xpath=//h1[contains(text(), 'Welcome')] | //div[@class='homepage-content'] # Example: Wait for a welcome element

Input Search Term
  Input Text  ${SEARCH_INPUT_LOCATOR}  ${SEARCH_TERM}

Submit Search
  Click Element  ${SEARCH_BUTTON_LOCATOR}
  Wait Until Page Contains Element  ${RESULTS_CONTAINER_LOCATOR}  timeout=10s

Verify Search Results Present
  Element Should Be Visible  ${RESULTS_CONTAINER_LOCATOR}

Verify Search Results Contain Term
  Page Should Contain  ${SEARCH_TERM}

testRigor Test Case

enter stored value “searchTerm” into “Search”
enter enter
check that page contains stored value “searchTerm”

Example 2: Checking table values

This test case is meant to check that the table contains the name “Stella” at row 102.

Robot Framework Test Case

*** Settings ***
Library      SeleniumLibrary

*** Variables ***
${BROWSER}    Chrome
${URL}      https://your-website-url.com # Replace with the actual website URL
${TABLE_NAME}   Actions
${ROW_NUMBER}   102
${COLUMN_HEADER} Name
${EXPECTED_VALUE} Stella

*** Test Cases ***
Verify Table Cell Content
  Open Browser To Website
  Verify Table Cell Value

*** Keywords ***
Open Browser To Website
  Open Browser  ${URL}  ${BROWSER}
  Maximize Browser Window
  Wait Until Page Contains Element  xpath=//table[@id='${TABLE_NAME}']

Verify Table Cell Value
  ${table_locator} =  Set Variable  xpath=//table[@id='${TABLE_NAME}'] # Assuming table has an ID
  ${row_xpath} =  Evaluate  ${table_locator} + '/tbody/tr[' + str(${ROW_NUMBER}) + ']'
  ${header_xpath} =  Evaluate  ${table_locator} + '//thead/tr/th[text()="${COLUMN_HEADER}"]'
  ${header_index} =  Get Element Index  ${header_xpath}
  ${cell_xpath} =  Evaluate  ${row_xpath} + '/td[' + str(${header_index} + 1) + ']'
  ${cell_text} =  Get Text  ${cell_xpath}
  Should Be Equal  ${cell_text}  ${EXPECTED_VALUE}

testRigor Test Case

check that table "Actions" at row "102" and column "Name" contains "Spock"

As you can so clearly see to do a simple UI-based test, Robot Framework requires a lot of configuration and setup. While it offers the luxury to write English statements (keywords), the caveat is that you have to define every single keyword with pre-defined commands. If you want to further customize the keyword, you must know how to code. Moreover, you need to depend on traditional UI element locators like XPaths to tell the engine where to look. This again means that only those who are familiar working with such code-level implementations can build tests in the real sense.

On the other hand, a simple test case remains simple in testRigor. Any configuration that you want to do, like setting up the default navigation URL, login credentials, variable values, etc., can easily be done from the UI through some simple settings. Most importantly, the keywords that you use in testRigor are interpreted by the tool’s powerful AI engine. So if the element moves around or is renamed or looks different than it used to, it will be handled by testRigor.

There’s a lot more that testRigor offers to its users to make all forms of software testing possible.

• Reusable Rules (Subroutines): You can easily create functions for the test steps that you use repeatedly. You can use the Reusable Rules to create such functions and call them in test cases by simply writing their names. See the example of Reusable Rules.
• Global Variables and Data Sets: You can import data from external files or create your own global variables and data sets in testRigor to use them in data-driven testing.
• 2FA, QR Code, and Captcha Resolution: testRigor easily manages the 2FA, QR Code, and Captcha resolution through its simple English commands.
• Email, Phone Call, and SMS Testing: Use simple English commands to test the email, phone calls, and SMS. These commands are useful for validating 2FA scenarios, with OTPs and authentication codes being sent to email, phone calls, or via phone text.
• File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
• Database Testing: Execute database queries and validate the results fetched.
• Data-driven Testing: testRigor supports data-driven testing, which can help test different real-world applications.
• File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
• Parallel Testing and CI/CD: For faster delivery cycles, you can completely rely on testRigor for parallel testing, AI, and automation testing to keep the CI/CD pipelines running smoothly.
• Visual Testing: testRigor’s Vision AI lets you compare the current screenshot with a baselined one, as a human would “see” it. This let you perform visual testing just in plain English commands, saving your team’s enormous time and effort.
• Accessibility and Exploratory Testing: You can efficiently perform accessibility testing through testRigor. Read here how to build an ADA-compliant app. This intelligent tool helps you run exploratory testing with its AI-powered features. Read how to perform exploratory testing with testRigor.
• AI Features Testing: Using testRigor, you can test chatbots (LLMs), and AI features such as negative/positive intentions, true/false statements, sensitive information, etc.

testRigor enables you to test web, mobile (hybrid, native), API, and desktop apps with minimum effort and maintenance. Here are all the things you can do with testRigor – testRigor’s Top Features.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT