Integrate your TestNG tests with TestRail

Have you ever thought that would be great if all the results of test automation run will be reported and stored in TestRail automatically?

Here you are, that moment came just now! A Library, which helps to implement auto-reporting of your TestNG-based tests to your TestRail management system. You need to follow just a few steps and it will be doing all stuff by its own. Review the article and try it out!

This guide explains how to integrate TestNG-based tests with TestRail in an easily manner.

I wrote this library in order to integrate your TestNG-based tests with TestRail just in an easily manner. Let’s see how to do that.

IMPORTANT:

For integration you need to provide TestRail link, Project Id and Credentials (username & password) into your code.

Now we can move further.

  1. So, firstly, you need to add this Library to your project. Since it’s not pushed to the MVN central repository yet, there are two installation options:
  • Download already compiled and packaged TestRail-TestNG-1.0.jar file HERE and proceed to the 2nd step
  • Download source code HERE to your local machine, make a jar by running mvn package in the repo directory and proceed to the 2nd step

2. Put a TestRail-TestNG-1.0.jar file into your project directory (for instance — create root dir ~/libs/ and paste there)

3. After that, add this dependency to your pom.xml (systemPath value is for instance)

<dependency>
<groupId>com.github.rsheremeta</groupId>
<artifactId>TestRail-TestNG</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${basedir}/libs/TestRail-TestNG-1.0.jar</systemPath>
</dependency>

Now when we’re done with adding a Library to the project, we need to annotate our tests with @TestRailCase annotation and pass the TestRail id for that test respectively:

package com.github.rsheremeta.test;

import com.github.rsheremeta.testrail.TestRailCase;
import org.testng.annotations.Test;

public class FirstTest extends BaseTest {

@Test
@TestRailCase(id = "1")
public void firstOne() {
System.out.println("This should be passed");
}

@Test(enabled = false)
@TestRailCase(id = "2")
public void firstTwo() {
System.out.println("This should be skipped");
}
}

Once you’re done with annotating your tests with corresponding TestRail Ids, you need to create a TestRailListener class that implements ITestListener, ISuiteListener interfaces and override their methods:

package com.github.rsheremeta.listener;

import org.testng.*;

public class TestRailListener implements ITestListener, ISuiteListener {

@Override
public void onStart(ISuite iSuite) { }

@Override
public void onFinish(ISuite iSuite) { }

@Override
public void onTestStart(ITestResult iTestResult) { }

@Override
public void onTestSuccess(ITestResult iTestResult) { }

@Override
public void onTestFailure(ITestResult iTestResult) { }

@Override
public void onTestSkipped(ITestResult iTestResult) { }

@Override
public void onTestFailedButWithinSuccessPercentage(ITestResult iTestResult) { }

@Override
public void onStart(ITestContext iTestContext) { }

@Override
public void onFinish(ITestContext iTestContext) { }
}

Now we’ll be implementing the key logic for all these methods one-by-one. Let’s start with onStart() method, which is needed for creation a new test run and setting up all needed data like User creds, Project id etc. Let’s define your TestRail Project Id and Name for automatically generated runs.

Please note, Test run name will be appended by a timestamp in format MM-dd-yyyy HH:mm:ss (For instance — 02–07–2021 13:21:11).

We’ll use a TestRailCreds builder class for creating TestRail data object for further passing as a parameter for the static method createNewTestRailRun() of TestRailHelper class, like this:

@Override
public void onStart(ISuite iSuite) {
String projectId = "1";
String testRunName = "TestRail TestNG run - ";
TestRailCreds creds = new TestRailCreds.Builder()
.withProjectUrl("https://link-to-your-testrail.testrail.io/")
.withUsername("your-mail@gmail.com")
.withPassword("your-password")
.build();
TestRailHelper.createNewTestRailRun(creds, projectId, testRunName);
}

Now we’ll be implementing listener methods for the corresponding test results. Setting up statuses will be done by calling static method setTestRailStatus() of the well-known TestRailHelper class.

This setTestRailStatus() method accept two parameters: instance of ITestResult and TestResult enumeration with one of the corresponding values — PASSED, FAILED, BLOCKED.

@Override
public void onTestSuccess(ITestResult iTestResult) {
TestRailHelper.setTestRailStatus(iTestResult, TestResult.PASSED);
}

@Override
public void onTestFailure(ITestResult iTestResult) {
TestRailHelper.setTestRailStatus(iTestResult, TestResult.FAILED);
}

@Override
public void onTestSkipped(ITestResult iTestResult) {
TestRailHelper.setTestRailStatus(iTestResult, TestResult.BLOCKED);
}

Please pay attention on the onTestSkipped() method, which should behave with BLOCKED status, as TestRail does not allow to add result as “Untested”, which is a default value for each and every test in a run, so there is no need to define and set it explicitly.

Now we’re all set with setting statuses for our tests and the last one step is to annotate our test classes with our Listener written just now.

There are basically two ways to associate our Listener with our tests:

  • Just add Listener with corresponding tag into your suite.xml:
<?xml version = "1.0" encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Smoke" parallel="tests">

<listeners>
<listener class-name= "com.github.rsheremeta.listener.TestRailListener"/>
</listeners>


<test name="Smoke Suite">

<packages>
<package name="com.github.rsheremeta.test.*"/>
</packages>

</test>

</suite>
  • If all your test classes are inherited from BaseTest (as in code mentioned above), you can annotate only parent class like this:
package com.github.rsheremeta.test;

import com.github.rsheremeta.listener.TestRailListener;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Listeners;

@Listeners(TestRailListener.class)
public class BaseTest {

@BeforeSuite
public void beforeSuite() {
System.out.println("This is being run before the entire suite");
}

@AfterSuite
public void afterSuite() {
System.out.println("This is being run after the entire suite");
}
}

So finally, we’re done with code-related activities! Now we just need to run our test suite and navigate to the TestRail runs listing page and see what the magic happened :)

  1. Run your test suite, in my case it’s done by mvn clean test command in Terminal.
  2. Navigate to the TestRail in browser, switch to the TEST RUNS & RESULTS tab and locate just created run.

Let’s take a look on my results.

Here is a listing page with all my test runs generated automatically:

Test runs list

Here is a page of the particular Test Run which contains a list of tests with corresponding statuses:

Test result list

And here is a failed Test with provided StackTrace of its failure for better and easier debugging:

Failed Test with StackTrace

So that’s pretty much it. Thank you very much for reading! Hope my Library will be a useful one for you.

Also I invite you read my other articles:

I’ll be thankful if you’ll support me by making a clap.

More articles will be released shortly, so stay tuned!

Experienced Test Automation Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store