Integrate your TestNG tests with TestRail
Have you ever thought that would be great if all the results of test automation run will be reported and stored in TestRail automatically?
Here you are, that moment came just now! A Library, which helps to implement auto-reporting of your TestNG-based tests to your TestRail management system. You need to follow just a few steps and it will be doing all stuff by its own. Review the article and try it out!
This guide explains how to integrate TestNG-based tests with TestRail in an easily manner.
Library link:
I wrote this library in order to integrate your TestNG-based tests with TestRail just in an easily manner. Let’s see how to do that.
IMPORTANT:
For integration you need to provide TestRail link, Project Id and Credentials (username & password) into your code.
Now we can move further.
- So, firstly, you need to add this Library to your project. Since it’s not pushed to the MVN central repository yet, there are two installation options:
- Download already compiled and packaged TestRail-TestNG-1.0.jar file HERE and proceed to the 2nd step
- Download source code HERE to your local machine, make a jar by running
mvn package
in the repo directory and proceed to the 2nd step
2. Put a TestRail-TestNG-1.0.jar file into your project directory (for instance — create root dir ~/libs/ and paste there)
3. After that, add this dependency to your pom.xml (systemPath value is for instance)
Now when we’re done with adding a Library to the project, we need to annotate our tests with @TestRailCase annotation and pass the TestRail id for that test respectively.
Succeeded ones:
Failed/skipped ones:
Once you’re done with annotating your tests with corresponding TestRail Ids, you need to create a TestRailListener class that implements ITestListener, ISuiteListener interfaces and override their methods:
package com.github.rsheremeta.listener;
import org.testng.*;
public class TestRailListener implements ITestListener, ISuiteListener {
@Override
public void onStart(ISuite iSuite) { }
@Override
public void onFinish(ISuite iSuite) { }
@Override
public void onTestStart(ITestResult iTestResult) { }
@Override
public void onTestSuccess(ITestResult iTestResult) { }
@Override
public void onTestFailure(ITestResult iTestResult) { }
@Override
public void onTestSkipped(ITestResult iTestResult) { }
@Override
public void onTestFailedButWithinSuccessPercentage(ITestResult iTestResult) { }
@Override
public void onStart(ITestContext iTestContext) { }
@Override
public void onFinish(ITestContext iTestContext) { }
}
Now we’ll be implementing the key logic for all these methods one-by-one. Let’s start with onStart() method, which is needed for creation a new test run and setting up all needed data like User creds, Project id etc. Let’s define your TestRail Project Id and Name for automatically generated runs.
Please note, Test run name will be appended by a timestamp in format MM-dd-yyyy HH:mm:ss (For instance — 02–07–2021 13:21:11).
We’ll use a TestRailCreds builder class for creating TestRail data object for further passing as a parameter for the static method createNewTestRailRun() of TestRailHelper class, like this:
Now we’ll be implementing listener methods for the corresponding test results. Setting up statuses will be done by calling static method setTestRailStatus() of the well-known TestRailHelper class.
This setTestRailStatus() method accept two parameters: instance of ITestResult and TestResult enumeration with one of the corresponding values — PASSED, FAILED, BLOCKED.
Please pay attention on the onTestSkipped() method, which should behave with BLOCKED status, as TestRail does not allow to add result as “Untested”, which is a default value for each and every test in a run, so there is no need to define and set it explicitly.
Now we’re all set with setting statuses for our tests and the last one step is to annotate our test classes with our Listener written just now.
There are basically two ways to associate our Listener with our tests:
- Just add Listener with corresponding tag into your suite.xml:
- If all your test classes are inherited from BaseTest (as in code mentioned above), you can annotate only parent class like this:
So finally, we’re done with code-related activities! Now we just need to run our test suite and navigate to the TestRail runs listing page and see what the magic happened :)
- Run your test suite, in my case it’s done by mvn clean test command in Terminal.
- Navigate to the TestRail in browser, switch to the TEST RUNS & RESULTS tab and locate just created run.
Let’s take a look on my results.
Here is a listing page with all my test runs generated automatically:
Here is a page of the particular Test Run which contains a list of tests with corresponding statuses:
And here is a failed Test with provided StackTrace of its failure for better and easier debugging:
So that’s pretty much it. Thank you very much for reading! Hope my Library will be a useful one for you.
Also I invite you read my other articles:
- Benchmarking Gob vs JSON, XML & YAML
- The testing feature you’ve been missing in pure Go / Golang
- How to save Allure Environment data automatically
- Integrate your JUnit tests with TestRail
I’ll be thankful if you’ll support me by making a clap.
More articles will be released shortly, so stay tuned!