Cypress vs. Selenium: which is best for testing chatbots?

Speed and agility of Cypress, or the advanced capabilities of Selenium WebDriver? Our tester, Mateusz, shares what counts more in testing conversational marketing chatbots

This post discusses:

  • how to prepare test automation for a chatbot application
  • what are the challenges of testing chatbot conversations
  • why we changed the test automation framework during the project
  • what is the difference between Selenium and Cypress
  • what are the pros and cons of Selenium and Cypress
  • why did CCA Europe chose Selenium after all

Conversational marketing, i.e., engaging customers in personalized conversations about products or services, is an effective way to engage potential buyers in social media or corporate websites. However, creating such a solution is not easy. Chatbots use advanced algorithms that guide visitors in a conversation based on keywords, specific activities, or other details.

For the last few months, here at CCA, we have been testing an application used in a conversational marketing chatbot for a foreign end client. The task of one of our testers, Mateusz, was to create a mechanism by which each test executed on the created application would establish a unique visitor-agent connection. A properly conducted test was supposed to create a new agent persona each time, with a new set of skills activated by the given keywords. Establishing communication between the agent and initiating a chat with the correct agent.

Test automation that is ready for a flood of changes

When it comes to preparing test cases, the challenge was to determine what requirements must be met to initiate a chat with an agent within a given skill set. The second challenge was to adapt the code to the communication interface. As it turned out, the technical aspect was the greatest design challenge.

“In the early phases of the project, we used the JavaScrip-based framework, Cypress. It is a tool that runs directly in the browser and is quite convenient to use”, says Mateusz Cierpka, the Full-Stack Developer responsible for the project on behalf of CCA Europe.

Since the basic testing automation framework in CCA Europe is Selenium WebDriver, whose scripts are written in Java, the new framework had to be grasped using documentation. In the course of the project, however, it turned out that the limitations of Cypress prevent properly testing the chatbot application.

A pivot that saved the testing project

“Early on, each test script actually created a new agent, a new visitor made a connection, and the test went well – as long as you ran it one at a time. However, when several tests were run in parallel during the CI/CD process, e.g. when the developer published a new feature or a fix in the application code, a bug appeared”, explains Mateusz.

The bug was that the system created several agents with the same skill and tried to launch a chat with them. When the visitor from test A wanted to connect with the agent from test A, they were connected to the agent from test B just because it had the same skill set, or the connection attempt failed altogether. Either way, no chat appeared for any of the agents and the test execution failed. The test script could not proceed to subsequent statements and the system restarted it. This caused a number of bizarre, unpredictable situations.

To proceed with the testing project, it was necessary to stabilize the connections between the agent and the visitor. Some test cases had to be written in such a way that the tester had to launch the tabs manually on the agent’s site, and on the visitor’s. The solution was ultimately to switch from Cypress to Selenium WebDriver, a framework that gives a lot of freedom, allowing testers to independently build various functionalities, but is much more technologically advanced.

Both Selenium WebDriver and Cypress are open-source solutions, used to automate web application tests. However, it is Selenium that has been recognized as the undisputed leader in the ranking of web application test automation tools for years.

Selenium vs Cypress: versatility, or agility and speed?

Unlike Cypress, Selenium WebDriver works with every popular web browser, based on native drivers. Apart from Java, Selenium supports other programming languages: C#, JavaScript, Python, or Ruby. Developers preparing automation testing tests also appreciate the fact that there is a large community of Selenium users where answers to most popular challenges can be found quickly and easily.

The Cypress framework is a completely different tool from Selenium not only in terms of its features, but also (and maybe most of all) its architecture. This is why Cypress is so light, agile, and, above all, fast. Cypress code is written in JavaScript and runs directly in the browser, unlike the demanding Selenium WebDriver. This is why it works so well in front-end testing of web applications.

Cypress offers only one programming language, which unfortunately prevents its use in more advanced contexts. Although it is constantly being developed and improved, its circle of enthusiasts who are ready to help in the search for a solution, is not significant yet.

Cypress and Selenium testing frameworks compared

 CCA Europe opts for Selenium

In the end, Mateusz managed to solve the identified problems by changing the entire process of creating a chatbot agent and visitor, as well as their skills and campaigns. He prepared appropriate HTTP requests communicating via API, which also allowed him to launch chats and deploy other functionalities.

Ultimately, the test automation framework selected in our project was Selenium rather than Cypress. “The decisive factor was the wide range of possibilities of this test automation framework. My teammates aUltimately, the test automation framework selected in our project was Selenium rather than Cypress. “The decisive factor was the wide range of possibilities of this test automation framework. My teammates and I agreed that continuing to use Cypress simply will not allow testing of all designed test cases in a given time,” explains Mateusz.

“In automated tests of complex solutions, such as chatbots, experience and knowledge of advanced solutions are of great importance. The CCA Europe test team consists of experienced Selenium professionals, especially in the field of testing banking applications,” comments Arkadiusz Stanimir, Managing Director of CCA Europe.

Thanks to a two-way integration of a proprietary PATT tool and Selenium, CCA Europe testers perform extensive, end-to-end tests for the banking industry, including entire banking processes. “Integration of PATT with Selenium allows not only to accelerate the work of testers, but also to reduce business costs” Arkadiusz Stanimir points out.

Looking for support in the analytical or execution phase of your test automation? Need an experienced business partner to run a functional test project for banking software? Join our CEO, Jacek Nowak for a 30-minute chat, no strings attached.