Integrate your TestNG tests with TestRail

4 min readFeb 20, 2021

Have you ever thought that would be great if all the results of test automation run will be reported and stored in TestRail automatically?

Here you are, that moment came just now! A Library, which helps to implement auto-reporting of your TestNG-based tests to your TestRail management system. You need to follow just a few steps and it will be doing all stuff by its own. Review the article and try it out!

This guide explains how to integrate TestNG-based tests with TestRail in an easily manner.

Library link:

https://github.com/RSheremeta/TestRail-TestNG

I wrote this library in order to integrate your TestNG-based tests with TestRail just in an easily manner. Let’s see how to do that.

IMPORTANT:
For integration you need to provide TestRail link, Project Id and Credentials (username & password) into your code.

Now we can move further.

So, firstly, you need to add this Library to your project. Since it’s not pushed to the MVN central repository yet, there are two installation options:

Download already compiled and packaged TestRail-TestNG-1.0.jar file HERE and proceed to the 2nd step
Download source code HERE to your local machine, make a jar by running mvn package in the repo directory and proceed to the 2nd step

2. Put a TestRail-TestNG-1.0.jar file into your project directory (for instance — create root dir ~/libs/ and paste there)

3. After that, add this dependency to your pom.xml (systemPath value is for instance)

Mvn dependency

Now when we’re done with adding a Library to the project, we need to annotate our tests with @TestRailCase annotation and pass the TestRail id for that test respectively.

Succeeded ones:

succeded methods

Failed/skipped ones:

failed/skipped methods

Once you’re done with annotating your tests with corresponding TestRail Ids, you need to create a TestRailListener class that implements ITestListener, ISuiteListener interfaces and override their methods:

package com.github.rsheremeta.listener;

import org.testng.*;

public class TestRailListener implements ITestListener, ISuiteListener {

  @Override
  public void onStart(ISuite iSuite) { }

  @Override
  public void onFinish(ISuite iSuite) { }

  @Override
  public void onTestStart(ITestResult iTestResult) { }

  @Override
  public void onTestSuccess(ITestResult iTestResult) { }

  @Override
  public void onTestFailure(ITestResult iTestResult) { }

  @Override
  public void onTestSkipped(ITestResult iTestResult) { }

  @Override
  public void onTestFailedButWithinSuccessPercentage(ITestResult iTestResult) { }

  @Override
  public void onStart(ITestContext iTestContext) { }

  @Override
  public void onFinish(ITestContext iTestContext) { }
}

Now we’ll be implementing the key logic for all these methods one-by-one. Let’s start with onStart() method, which is needed for creation a new test run and setting up all needed data like User creds, Project id etc. Let’s define your TestRail Project Id and Name for automatically generated runs.

Please note, Test run name will be appended by a timestamp in format MM-dd-yyyy HH:mm:ss (For instance — 02–07–2021 13:21:11).

We’ll use a TestRailCreds builder class for creating TestRail data object for further passing as a parameter for the static method createNewTestRailRun() of TestRailHelper class, like this:

onStart() listener method

Now we’ll be implementing listener methods for the corresponding test results. Setting up statuses will be done by calling static method setTestRailStatus() of the well-known TestRailHelper class.

This setTestRailStatus() method accept two parameters: instance of ITestResult and TestResult enumeration with one of the corresponding values — PASSED, FAILED, BLOCKED.

another listener methods

Please pay attention on the onTestSkipped() method, which should behave with BLOCKED status, as TestRail does not allow to add result as “Untested”, which is a default value for each and every test in a run, so there is no need to define and set it explicitly.

Now we’re all set with setting statuses for our tests and the last one step is to annotate our test classes with our Listener written just now.

There are basically two ways to associate our Listener with our tests:

Just add Listener with corresponding tag into your suite.xml:

testng-suite.xml

If all your test classes are inherited from BaseTest (as in code mentioned above), you can annotate only parent class like this:

listener in BaseTest.class

So finally, we’re done with code-related activities! Now we just need to run our test suite and navigate to the TestRail runs listing page and see what the magic happened :)

Run your test suite, in my case it’s done by mvn clean test command in Terminal.
Navigate to the TestRail in browser, switch to the TEST RUNS & RESULTS tab and locate just created run.

Let’s take a look on my results.

Here is a listing page with all my test runs generated automatically: