Are you annoyed by unrecognized UI changes? Are you still wasting time making sure your changes are not crashing something? Use BackstopJS as an option to improve your regression testing process locally and in your CI/CD process. We tried it out and developed this application that now makes our lives easier.
Usage of BackstopJS
BackstopJS runs through a predefined list of pages on a website, takes screenshots, and compares these screenshots to highlight any changes. It can be run locally and can be easily integrated into a pipeline.
Open a command prompt and type the following command, to install BackstopJS globally:
After installing, you have to close your command prompt and reopen it, so you can use the
First, you have to initialize Backstop. This command will create all the files you need for the test run later.
The pages are configured in the generated
backstop.json file. You can add your pages as
scenarios in that JSON file. It is also advisable to adjust the
delay to give the tool some time before the screenshot is taken so that the page with images has loaded completely.
Most websites have a cookie-consent overlay that blocks large parts of the page. To prevent this from appearing on the screenshots, there is the
cookies.json file, which was generated under the path
.\backstop_data\engine_scripts by the Init command. Update the file with the cookie values of your test page.
Adding a resolution
To test different resolutions and thus devices, you can create as many entries as you like in the
backstop.json file under the
You need to make an initial run so that you have screenshots for later comparison.
This creates a folder with the screenshots under
After the screenshots have been taken, the status must be approved and used as a reference for the subsequent screenshots.
With a new
test command, the previously created screenshots are compared with new ones and an analysis is made.
Interactive test report
After the test has been run, the
.\backstop_data\html_report is opened and the results are visible.
- 4 tests passed
- 2 tests have failed
- First defined entry “Merkle DACH Homepage en” with “phone” resolution
- Green bar for passed tests
- Red bar for failed tests
- Diff with errors marked (click on it)
The magenta colored areas show the difference between the screenshots. In this case, the dots were slightly shifted and can be ignored. You can slide the vertical line to see the difference between the reference and test image.
Integration to Azure DevOps Pipeline
Include configs in your repository
Before you change the pipeline, you must check in the config files created and customized by BackstopJS in the repository.
Set tokens to replace later
If you have several environments that are to be tested, it makes sense to use a configuration file in which the URL is defined as a token that can be replaced for different stages. Here we defined
$(Target.WebsiteUrl) that will be replaced by
https://www.merkle.com/, but it could also be something like
Create a storage
In order to store the test results somewhere, a storage must first be created, e.g. an Azure Blob Storage can be used for this.
Steps on Azure DevOps
Set variable CurrentDateTime
Unfortunately, there is no variable like a timestamp that we could use to add to the file name when we archive the backstop data. Therefore, we added a variable called
CurrentDateTime, which we set in the
Set CurrentDateTime step. The script looks as follows:
Define BackstopConfig path
Add a variable with the name
BackstopConfig and set it to the path of the checked-in file
Use the task Replace token to replace the website URL. In this case,
Target.WebsiteUrl is a variable defined in the pipeline. The step could look like this:
Use the Node.js tool installer and install version 14.x
Install BackstopJS using the Command line task
Use again the Command line task and run the following command
Deploy your app
Now you have to deploy your web app and warm up your cd instance.
Run the test and make sure that a failed test does not cause the pipeline to stop. Since BackstopJS will most likely return some false positives, this could lead to an unwanted stop of the pipeline. Set
Archive Backstop Data
You can archive the entire
backstop_data folder so that you can upload it to a storage such as Azure Blob Storage.
Upload it to Azure
You can use Azure File Copy to upload the archive to storage on Azure
Check your logs and test results
After each deployment, the developers can look specifically at the BackstopJS logs from the pipeline and see if there are any failed tests. If this is the case, the test results can be downloaded as a zip file, unpacked and analyzed.
In our experience, there are often false positives, so the tool should not be used as a 100% reliable test tool. Nevertheless, it saves a lot of testing effort if several dozen pages are checked automatically and only those from the error messages need to be checked manually.