Is it possible to create a timeout at suite level in testng for a test class rather than individual methods?

I'm looking at a problem we have within testng. I would like to know if there is a way to specify a timeout at a test class level, rather than individual methods. I'm aware of "time-out" at suite xml level, but this then applies to each test method in each test class.

I would really to be able to specify a timeout that applies to the each class in an xml, so we can then look at improving these classes.

2 Answers

This is not supported out of the box. But TestNG does provide you with capabilities using which you should be able to build this on your own, with very little effort. [ I am using TestNG 7.0.0-beta7 for this, and this is the latest released version of TestNG at the time of posting this answer ]

1. You first need to create a custom annotation which can be used to define timeouts at class level.
2. You then build an implementation of org.testng.IAnnotationTransformer which can read the value of this custom annotation from the declaring class of the test method, and if the annotation is present, then it would alter the @Test annotation's timeout attribute.

Here's a full fledged example that shows this in action.

Custom annotation looks like below:

import static java.lang.annotation.ElementType.TYPE;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({TYPE})
public @interface Timeout { long inMs() default 0;
}

The test classes that consume the above custom annotation would look like below

import java.util.concurrent.TimeUnit;
import org.testng.annotations.Test;
@Timeout(inMs = 15)
public class TestClass1 { @Test public void sleepFor10Ms() throws InterruptedException { TimeUnit.MILLISECONDS.sleep(10); } @Test public void sleepFor20Ms() throws InterruptedException { TimeUnit.MILLISECONDS.sleep(20); }
}

import java.util.concurrent.TimeUnit;
import org.testng.annotations.Test;
@Timeout(inMs = 5)
public class TestClass2 { @Test public void sleepFor10Ms() throws InterruptedException { TimeUnit.MILLISECONDS.sleep(10); } @Test public void sleepFor20Ms() throws InterruptedException { TimeUnit.MILLISECONDS.sleep(20); }
}

The annotation transformer looks like below

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
public class TimeoutSetter implements IAnnotationTransformer { @Override public void transform( ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) { if (testMethod == null) { return; } Timeout timeout = testMethod.getDeclaringClass().getAnnotation(Timeout.class); if (timeout == null) { return; } annotation.setTimeOut(timeout.inMs()); }
}

Here's how the suite file looks like

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "">
<suite name="Suite" verbose="2"> <listeners> <listener class-name="com.rationaleemotions.stackoverflow.qn57307245.TimeoutSetter"/> </listeners> <test name="Test"> <classes> <class name="com.rationaleemotions.stackoverflow.qn57307245.TestClass1"/> <class name="com.rationaleemotions.stackoverflow.qn57307245.TestClass2"/> </classes> </test>
</suite>

Execution output

... TestNG 7.0.0-beta7 by Cédric Beust ([email protected])
...
org.testng.internal.thread.ThreadTimeoutException: Method com.rationaleemotions.stackoverflow.qn57307245.TestClass1.sleepFor20Ms() didn't finish within the time-out 15 at java.util.concurrent.ThreadPoolExecutor$Worker.unlock(ThreadPoolExecutor.java:652) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
org.testng.internal.thread.ThreadTimeoutException: Method com.rationaleemotions.stackoverflow.qn57307245.TestClass2.sleepFor10Ms() didn't finish within the time-out 5 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:51) at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:41) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
org.testng.internal.thread.ThreadTimeoutException: Method com.rationaleemotions.stackoverflow.qn57307245.TestClass2.sleepFor20Ms() didn't finish within the time-out 5 at sun.misc.Unsafe.compareAndSwapInt(Native Method) at java.util.concurrent.locks.AbstractQueuedSynchronizer.compareAndSetWaitStatus(AbstractQueuedSynchronizer.java:2303) at java.util.concurrent.locks.AbstractQueuedSynchronizer.transferForSignal(AbstractQueuedSynchronizer.java:1674) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.doSignalAll(AbstractQueuedSynchronizer.java:1888) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signalAll(AbstractQueuedSynchronizer.java:1957) at java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:716) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1014) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
PASSED: sleepFor10Ms
FAILED: sleepFor20Ms
org.testng.internal.thread.ThreadTimeoutException: Method com.rationaleemotions.stackoverflow.qn57307245.TestClass1.sleepFor20Ms() didn't finish within the time-out 15 at java.util.concurrent.ThreadPoolExecutor$Worker.unlock(ThreadPoolExecutor.java:652) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
FAILED: sleepFor10Ms
org.testng.internal.thread.ThreadTimeoutException: Method com.rationaleemotions.stackoverflow.qn57307245.TestClass2.sleepFor10Ms() didn't finish within the time-out 5 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:51) at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:41) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
FAILED: sleepFor20Ms
org.testng.internal.thread.ThreadTimeoutException: Method com.rationaleemotions.stackoverflow.qn57307245.TestClass2.sleepFor20Ms() didn't finish within the time-out 5 at sun.misc.Unsafe.compareAndSwapInt(Native Method) at java.util.concurrent.locks.AbstractQueuedSynchronizer.compareAndSetWaitStatus(AbstractQueuedSynchronizer.java:2303) at java.util.concurrent.locks.AbstractQueuedSynchronizer.transferForSignal(AbstractQueuedSynchronizer.java:1674) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.doSignalAll(AbstractQueuedSynchronizer.java:1888) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signalAll(AbstractQueuedSynchronizer.java:1957) at java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:716) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1014) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
=============================================== Test Tests run: 7, Failures: 3, Skips: 0
===============================================
===============================================
Suite
Total tests run: 4, Passes: 1, Failures: 3, Skips: 0
===============================================
Process finished with exit code 0

Timeout can be configured only in two ways.

1. At Suite level.
2. At each test method.

Essentially test classes in testng contains your testng methods which are your assertions of test conditions(in your case based on timeouts), so they are just normal java classes.

You can try keeping single test method per testng class and add timeout to that method or you can create a external method which will make the pass or fail decision based on the time taken for method execution.