Robot Framework 101: Comprehensive Overview, Pros & Cons

Anushree Chatterjee

What is Robot Framework?

Robot Framework is an open-source tool used for automating software testing. It’s designed to be simple and easy to use. It is widely used for acceptance testing and acceptance test-driven development (ATDD) and for automation tasks involving testing APIs, GUIs, databases, and more.

The key part of Robot Framework is that it uses something called keywords. These keywords are like simple instructions, such as “Open browser,” “Click on a button,” or “Verify text.” You can create these instructions in a text file, and Robot Framework will run the tests automatically, just like a robot following a list of tasks.

It also supports different technologies and tools, like Selenium (for web testing) or Appium (for mobile testing), making it very versatile. Plus, it provides clear reports after running the tests, so you can easily see if anything is wrong.

Key Features of Robot Framework

Here are some of Robot Framework’s key features:

• Open Source and Free: Robot Framework is open-source software released under the Apache License 2.0, which means it is free to use and modify. This open nature encourages contributions from the community and leads to a wide array of libraries, tools, and extensions.
• Keyword-Driven Testing: In Robot Framework, you write tests using “keywords” that represent actions or checks. These keywords are like simple commands in plain language. These keywords are either built-in or custom-defined.
• Extensibility: Robot Framework can be extended by libraries implemented in Python, Java, or any other language that can run on the Java Virtual Machine (JVM). This extensibility allows for integration with a wide range of tools and technologies like Selenium, Appium, REST APIs and database connectors.
• Support for Parallel Execution: To improve efficiency, Robot Framework supports parallel test execution through tools like Pabot. This allows multiple test cases to run simultaneously.
• Rich Ecosystem: The Robot Framework community has developed a rich ecosystem of libraries and tools that extend its capabilities. Some popular libraries include SeleniumLibrary for web testing, DatabaseLibrary for database operations, and SSHLibrary for SSH connections.
• Data-Driven Testing: The framework supports data-driven testing which lets the execution of the same test cases with multiple sets of data. This is particularly useful for testing applications with various input values and conditions.

Components of a Robot Framework Test

***Settings***
Documentation   This suite tests user login functionality.

***Test Cases***
Valid Login
  Open Browser  ${LOGIN_URL}  ${BROWSER}
  Input Text   ${USERNAME_FIELD}  valid_user
  Input Text   ${PASSWORD_FIELD}  secure_password
  Click Element  ${LOGIN_BUTTON}
  Wait Until Page Contains Element  xpath=//div[@class='dashboard']

Invalid Login - Incorrect Password
  Open Browser  ${LOGIN_URL}  ${BROWSER}
  Input Text   ${USERNAME_FIELD}  valid_use
  Input Text   ${PASSWORD_FIELD}  wrong_password
  Click Element  ${LOGIN_BUTTON}
  Page Should Contain Text  Invalid credentials

Invalid Login - Non-existent User
  Open Browser  ${LOGIN_URL}  ${BROWSER}
  Input Text   ${USERNAME_FIELD}  non_existent_user
  Input Text   ${PASSWORD_FIELD}  some_password
  Click Element  ${LOGIN_BUTTON}
  Page Should Contain Text  User not found

Let’s look at the main components you’ll encounter while working with the Robot Framework:

• Test Case: A test case is a set of steps that describe the actions to be performed and the expected results. It’s the core of testing in Robot Framework. Test cases and their related data are written in a specific format, typically in plain text files with the .robot extension. These files define the tests, their steps, and the data they use. It consists of a name and a series of test steps (keywords). Example (from above): Valid Login, Invalid Login - Incorrect Password, Invalid Login - Non-existent User are individual test cases.
• Test Suites: A test suite is a collection of related test cases. You can organize your tests into multiple test suite files for better management. In the above code example, the entire .robot file represents a test suite focused on login functionality.
• Keywords: Keywords are the building blocks of Robot Framework. They are simple commands that perform specific actions. Robot Framework provides built-in keywords (like Open Browser, Input Text, Click Element, Wait Until Page Contains Element, Page Should Contain Text) and allows you to create custom keywords. These keywords make the tests human-readable and reusable. Example (from above): Open Browser ${LOGIN_URL} ${BROWSER} is a test step that uses the built-in keyword Open Browser.
• Variables: Variables store values that can be reused throughout your test cases or test suites. You can think of them as placeholders for values like URLs, credentials, or test data. Variables help make tests more flexible and maintainable. If you need to change a value, you only need to update it in one place rather than in every test case. Variables can be defined in the ***Variables*** section or directly within test cases. Example:

  ***Variables***
  ${BROWSER}    Chrome
  ${LOGIN_URL}   https://example.com/login
  ${USERNAME_FIELD}  id:username
  ${PASSWORD_FIELD}  id:password
  ${LOGIN_BUTTON}   xpath://button[@type='submit']

  Here, ${BROWSER}, ${LOGIN_URL}, etc., are variables holding specific values.
• Settings: The ***Settings*** section is used to configure the test suite. It includes settings for importing libraries, setting documentation, defining test tags, etc. Example (from above): Documentation sets the description for the test suite, and Library SeleniumLibrary (implicitly imported in the ***Settings*** of the first example) imports the SeleniumLibrary, which provides keywords for web automation.
• Libraries: Libraries are collections of keywords. These can be built-in (like the SeleniumLibrary for web automation) or custom (created by users to perform specific tasks). Libraries are great for extending Robot Framework’s functionality to make it capable of testing different types of applications (web, desktop, APIs, etc.).
• Execution Engine: The execution engine is the core part of the Robot Framework that runs the tests. It takes the test cases and executes them step-by-step using the appropriate keywords. You interact with the execution engine primarily through the command-line interface. You provide the path to your test data files, and Robot Framework takes care of the rest.
• Reports and Logs: After running tests, Robot Framework generates reports and logs that detail the execution results. For example, the HTML report will give you a summary of test results, while the log provides more detailed information (like screenshots, error messages, and step-by-step execution details).
• Output Files: Robot Framework generates three output files by default after running tests:
  • Output.xml: The detailed execution log in XML format.
  • Log.html: A human-readable log showing what happened during the test run.
  • Report.html: A summary report showing which tests passed or failed.

Testing with Robot Framework

Installing Robot Framework

Before you begin, ensure you have the following:

• Python Installed: Robot Framework is written in Python, so you need Python installed on your system. You can check if Python is installed by opening your terminal or command prompt and typing:

  python --version

• pip (Python Package Installer): pip is the package installer for Python and is usually installed along with Python. You can check if pip is installed by typing:

  pip --version

pip install robotframework

Installing Additional Libraries (Robot Framework with Selenium)

pip install robotframework-seleniumlibrary

If you are using SeleniumLibrary, you also need to have a web browser driver installed for the browser you want to automate. You can also install other libraries for different testing needs, like AppiumLibrary for mobile testing, DatabaseLibrary for database testing, etc.

Create a Test Case in Robot Framework

Now that you have Robot Framework installed, let’s write a simple test case.

• Create a New File: Open any text editor (like Notepad, Visual Studio Code, or Sublime Text) and create a new file with the extension .robot. For example, my_first_test.robot.
• Write a Test Case: In the .robot file, write a simple test. Here’s an example that checks if we can open a web browser and go to a website.

  *** Settings ***
  Library SeleniumLibrary
  
  *** Variables ***
  ${URL} http://example.com
  
  *** Test Cases ***
  Open Browser and Visit Example Website
  Open Browser ${URL} Chrome
  Title Should Be Example Domain
  Close Browser

Run the Test

Now that you’ve written your test, you can run it using the following command in the terminal.

• Navigate to the folder where you saved your .robot file. Use the cd command to go to that folder.
• Once you’re in the correct directory, use this command to run the test:

  robot my_first_test.robot

Check the Results

After running the test, Robot Framework generates a log and a report that you can view.

• Log: A detailed log of what happened during the test run (errors, successes, etc.). It’s saved as log.html.
• Report: A summary report that shows the overall results (passed, failed, etc.). It’s saved as report.html.

Both files will be generated in the same folder where you ran the test. To open them, just double-click the files in your file explorer, and they’ll open in your default web browser.

Pros and Cons of Robot Framework

Robot Framework has become one of the most widely used open-source automation frameworks due to its simplicity, flexibility, and extensive ecosystem. However, like any testing framework, it comes with both strengths and limitations that teams should carefully evaluate before adopting it for large-scale automation projects.

Is Robot Framework in Demand?

Robot Framework’s strength lies in its modularity and keyword-driven approach to testing. This makes it a popular pick for many test automation engineers.

However, with better tools in the market like testRigor that utilize AI, you’ll see that Robot Framework falls short in many aspects.

• Automating ATDD
  • Robot Framework isn’t inherently the best choice for ATDD. To implement ATDD effectively in Robot Framework, you need to manually map your business requirements (expressed in Gherkin or similar formats) to the Robot Framework keywords and test steps. This requires a conscious effort to structure your tests in a BDD-like manner. It doesn’t have built-in NLP capabilities to directly understand and translate natural language acceptance criteria into executable tests.
  • On the other hand, testRigor is built ground-up with the user in mind, and hence, can handle ATDD and BDD easily. The tool uses NLP and smart element locators to allow its users to automate test cases in simple English language. testRigor’s syntax is centered around describing user behavior and expected outcomes, directly reflecting acceptance criteria. For example, you might write: “Click ‘Login’ button. Verify that the text ‘Welcome, User’ is visible.”
• Keyword Maintenance
  • The keyword-driven approach works very well until you have to manage a whole lot of them. Imagine the nightmare when test cases start to fail as the keyword values change. Especially given that modern web pages tend to have a lot of dynamic elements that need to be tested. In Robot Framework, you need to manually track and update your keywords to make sure that the element locators being used are the right ones.
  • While testRigor lets you mention “keywords”, it does not rely on you to identify them on the screen. It uses AI for better element locator strategies that can mitigate changes with the implementation details of the UI elements.
• Dependency on Libraries
  • In Robot Framework, to ensure that you are able to test a host of applications (web, mobile, API, desktop, etc.) you need to integrate with different libraries. This can increase the complexity of you test automation process even more.
  • In testRigor, you get all of these capabilities and much more under one umbrella. The tool has built-in libraries to support all of these different types of scenarios in simple English language.
• Challenges with Testing Complex Scenarios
  • Modern applications involve many advanced components like working with OTPs, email verification, CAPTCHA resolution, and more. Trying to do this with Robot Framework, even if you import Selenium, will be tedious.
  • You can automate a vast number of test cases with ease using testRigor. The tool is equipped to take on these challenging test cases.
• Knowledge of Programming
  • You need to know some amount of coding to be able to test using Robot Framework. For example, if you want to add new keywords to the list, someone has to code it. Even using the framework on the whole is not as straightforward as a few clicks. This inevitably leaves out those who have the most understanding of user behavior – manual testers and business analysts.
  • Since testRigor offers plain English-based test creation through its use of AI, everyone can participate in the testing process. The tool even allows plain English as a way for you to write custom logic, unlike most other tools.
• Test Maintenance
  • Using Robot Framework can incur test maintenance overhead, especially if you have many tests to run. This is mainly due to the multiple dependencies involved in making a test case work.
  • On the contrary, testRigor uses AI to reduce test maintenance to a bare minimum. The tool uses AI to cut out the reliance on implementation details of UI elements like XPaths or CSS selectors, auto-heal test steps and reusable rules, and mitigate flakey test runs.

Robot Framework and AI Testing

Artificial Intelligence is rapidly transforming the software testing industry. Modern applications now include AI-powered features such as chatbots, recommendation systems, voice assistants, image recognition, predictive analytics, and intelligent workflows. As a result, traditional automation frameworks like Robot Framework are also being explored for AI testing purposes.

Robot Framework can support AI testing to some extent through its extensible architecture and integration capabilities. Since the framework allows integration with Python libraries, testers can connect AI and machine learning modules with Robot Framework test cases. For example, teams can integrate NLP libraries, image-processing tools, OCR engines, or AI APIs into Robot Framework using custom keywords. Using external libraries, Robot Framework can help automate:

• Chatbot validation
• API testing for AI services
• OCR-based validations
• Data-driven AI model testing
• Machine learning workflow verification
• AI prediction result validation

For example, a tester could create custom Python keywords that:

• Send prompts to an AI chatbot
• Validate response accuracy
• Compare prediction confidence levels
• Verify generated outputs
• Check AI model response times

Robot Framework can also integrate with tools such as TensorFlow, OpenAI APIs, Hugging Face models, OCR engines like Tesseract, and computer vision libraries like OpenCV.

However, despite these possibilities, Robot Framework was not originally designed as an AI-native testing platform. Most AI-related capabilities require heavy customization, third-party libraries, scripting knowledge, and manual framework integration. This increases implementation complexity and maintenance overhead.

Read: Prompt Engineering in QA and Software Testing.

Testing with testRigor

By now, you have already seen why testRigor is the next step ahead for your automation testing needs.

Let’s look at some test case examples to see how testRigor compares to Robot Framework.

Example 1: Searching text in the browser search bar

This is a simple test case to check that we are able to enter and search a string using the search bar provided by the browser.

Robot Framework Test Case

***Settings***
Library      SeleniumLibrary

***Variables***
${BROWSER}    Chrome
${URL}      https://www.example-ecommerce.com # Replace with the actual website URL
${SEARCH_TERM}  Laptop
${SEARCH_INPUT_LOCATOR}  id:search-input # Replace with the actual locator for the search input field
${SEARCH_BUTTON_LOCATOR}  xpath://button[@type='submit'] | //input[@type='submit' and @value='Search'] # Replace with the actual locator for the search button
${RESULTS_CONTAINER_LOCATOR}  css:.search-results # Replace with the actual locator for the container holding search results
${EXPECTED_RESULT_TEXT}  Laptop # Expected text to appear in the results (can be more specific)

***Test Cases***
Verify Successful Search
  Open Browser To Homepage
  Input Search Term
  Submit Search
  Verify Search Results Present
  Verify Search Results Contain Term

***Keywords***
Open Browser To Homepage
  Open Browser  ${URL}  ${BROWSER}
  Maximize Browser Window
  Wait Until Page Contains Element  xpath=//h1[contains(text(), 'Welcome')] | //div[@class='homepage-content'] # Example: Wait for a welcome element

Input Search Term
  Input Text  ${SEARCH_INPUT_LOCATOR}  ${SEARCH_TERM}

Submit Search
  Click Element  ${SEARCH_BUTTON_LOCATOR}
  Wait Until Page Contains Element  ${RESULTS_CONTAINER_LOCATOR}  timeout=10s

Verify Search Results Present
  Element Should Be Visible  ${RESULTS_CONTAINER_LOCATOR}

Verify Search Results Contain Term
  Page Should Contain  ${SEARCH_TERM}

testRigor Test Case

enter stored value “searchTerm” into “Search”
enter enter
check that page contains stored value “searchTerm”

Example 2: Checking table values

This test case is meant to check that the table contains the name “Stella” at row 102.

Robot Framework Test Case

*** Settings ***
Library      SeleniumLibrary

*** Variables ***
${BROWSER}    Chrome
${URL}      https://your-website-url.com # Replace with the actual website URL
${TABLE_NAME}   Actions
${ROW_NUMBER}   102
${COLUMN_HEADER} Name
${EXPECTED_VALUE} Stella

*** Test Cases ***
Verify Table Cell Content
  Open Browser To Website
  Verify Table Cell Value

*** Keywords ***
Open Browser To Website
  Open Browser  ${URL}  ${BROWSER}
  Maximize Browser Window
  Wait Until Page Contains Element  xpath=//table[@id='${TABLE_NAME}']

Verify Table Cell Value
  ${table_locator} =  Set Variable  xpath=//table[@id='${TABLE_NAME}'] # Assuming table has an ID
  ${row_xpath} =  Evaluate  ${table_locator} + '/tbody/tr[' + str(${ROW_NUMBER}) + ']'
  ${header_xpath} =  Evaluate  ${table_locator} + '//thead/tr/th[text()="${COLUMN_HEADER}"]'
  ${header_index} =  Get Element Index  ${header_xpath}
  ${cell_xpath} =  Evaluate  ${row_xpath} + '/td[' + str(${header_index} + 1) + ']'
  ${cell_text} =  Get Text  ${cell_xpath}
  Should Be Equal  ${cell_text}  ${EXPECTED_VALUE}

testRigor Test Case

check that table "Actions" at row "102" and column "Name" contains "Spock"

As you can so clearly see to do a simple UI-based test, Robot Framework requires a lot of configuration and setup. While it offers the luxury to write English statements (keywords), the caveat is that you have to define every single keyword with pre-defined commands. If you want to further customize the keyword, you must know how to code. Moreover, you need to depend on traditional UI element locators like XPaths to tell the engine where to look. This again means that only those who are familiar working with such code-level implementations can build tests in the real sense.

On the other hand, a simple test case remains simple in testRigor. Any configuration that you want to do, like setting up the default navigation URL, login credentials, variable values, etc., can easily be done from the UI through some simple settings. Most importantly, the keywords that you use in testRigor are interpreted by the tool’s powerful AI engine. So if the element moves around or is renamed or looks different than it used to, it will be handled by testRigor.

There’s a lot more that testRigor offers to its users to make all forms of software testing possible.

• Reusable Rules (Subroutines): You can easily create functions for the test steps that you use repeatedly. You can use the Reusable Rules to create such functions and call them in test cases by simply writing their names. See the example of Reusable Rules.
• Global Variables and Data Sets: You can import data from external files or create your own global variables and data sets in testRigor to use them in data-driven testing.
• 2FA, QR Code, and Captcha Resolution: testRigor easily manages the 2FA, QR Code, and Captcha resolution through its simple English commands.
• Email, Phone Call, and SMS Testing: Use simple English commands to test the email, phone calls, and SMS. These commands are useful for validating 2FA scenarios, with OTPs and authentication codes being sent to email, phone calls, or via phone text.
• File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
• Database Testing: Execute database queries and validate the results fetched.
• Data-driven Testing: testRigor supports data-driven testing, which can help test different real-world applications.
• File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
• Parallel Testing and CI/CD: For faster delivery cycles, you can completely rely on testRigor for parallel testing, AI, and automation testing to keep the CI/CD pipelines running smoothly.
• Visual Testing: testRigor’s Vision AI lets you compare the current screenshot with a baselined one, as a human would “see” it. This let you perform visual testing just in plain English commands, saving your team’s enormous time and effort.
• Accessibility and Exploratory Testing: You can efficiently perform accessibility testing through testRigor. Read here how to build an ADA-compliant app. This intelligent tool helps you run exploratory testing with its AI-powered features. Read how to perform exploratory testing with testRigor.
• AI Features Testing: Using testRigor, you can test chatbots (LLMs), and AI features such as negative/positive intentions, true/false statements, sensitive information, etc.

testRigor enables you to test web, mobile (hybrid, native), API, and desktop apps with minimum effort and maintenance. Here are all the things you can do with testRigor – testRigor’s Top Features.

FAQs

What types of applications can Robot Framework test?
Robot Framework can test web applications, mobile apps, APIs, desktop software, databases, and even command-line applications using different libraries and integrations.

Is Robot Framework suitable for beginners in test automation?
Yes, Robot Framework is considered beginner-friendly because of its readable keyword-driven syntax, though advanced customization may still require programming knowledge.

Can Robot Framework be used for CI/CD automation?
Yes, Robot Framework integrates with CI/CD tools such as Jenkins, GitHub Actions, GitLab CI, and Azure DevOps for automated test execution pipelines.

Does Robot Framework support cloud testing platforms?
Yes, it can integrate with cloud-based testing platforms like BrowserStack, Sauce Labs, and LambdaTest for cross-browser and cross-device testing.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT