Simple presentation of property based testing in Java Main tools in Java

1. Property-based testing with Java
By Franck Benault
Created 06/07/2016
Last update 10/08/2016

2. Property-based testing introduction
● Tests = often (or should be) only one example
– We can say « Example-based testing »
@Test
TestFoo() { //no parameters
//my code of test
Int x =1 ;
Int res =foo(x) ;
Assert(..)
}
● Yes but if my test is ok for x=1, is it ok for x=25 ?

3. Property-based testing introduction
● Property based testing is
– Based on existing check
– Randomly generate a lot of input parameter
● Which satisfy predefined properties
@Test
TestFoo2(String s1, int x ) { //parameters !
//my code of test
int res =foo(x) ; //the parameter are used in the test
Assert(..)
}

4. Property-based testing plan

5. Property-based testing introduction
● Property based testing functionalities
– Tests with parameters
● Values of parameters (randonly or following an order) generated
– N times
● Values of parameters satisfying properties
– >0, <0
– Filter of the bad values (« assume »)
● Seed value
– Value that allow us to relaunch the tests with the same random values
● Shrinking
– If the test fails, the framework tries to find the lowest values which fail
the test

6. Property-based testing links
● My examples in Github
– https://github.com/franck-benault/property-based-testing
● Other links
– http://www.ontestautomation.com/an-introduction-to-property
● Good introduction

7. Seed and Shrinking
● Shrinking
– If the test fails, repeat the test until it finds the simpliest
test case that still fails
● Seed
– There is a random generation
– But the seed value allows us to reproduce the exact
same run
– The seed value is the only source of randomness

8. Goal of this presentation
● Goal (in java world)
– Limitation of the classical test
– Simple presentation of property based testing
– First Comparison of the main tools
– Notion of seed and shrinking

9. Remind over Assume in JUnit
● Assume = assumption
– Ignore if it fails
– In method @BeforeClass, @Before, @Test
– Possible assumption
● AssumeTrue
● AssumeEquals, AssumeSame
● AssumeNull, AssumeNotNull
● AssumeThat (matcher)

10. Annotation @Theory of JUnit
● Annotation @Theory
– since Junit 4.4
– Junit only (not testNG)
– Specific Runner (class Theories)
● But all the junit functionalities are honored
– @Test, @Before, @After, @BeforeClass, @AfterClass
● Two parts
– Data creation
– Theory (test with parameters and @Theory)

11. Annotation @Theory of JUnit
● Data creation
– Annotations
● Datapoint (on value)
● Datapoints (on tabs public static method or variable)
● FromDataPoints annotation
● TestedOn (on parameters type int)
● It is possible to write your own annotations
● Theory (test with parameters and @Theory)
– Parameter nullsAccepted

12. Annotation @Theory of JUnit
● Limitation
– Experimental package
– No advanced data generation
● No random values
– No seed, no shrinking
– Known bug with enumerations

13. Junit-Quickcheck
● Junit-Quickcheck
– Property based testing for Java
– Runner = JUnitQuickcheck
– Random data generated
– Test annotated with @Property
● Variable trials (number of tests default = 100)
– Seed and Shrinking

14. Junit-Quickcheck
● Basic type supported random generation
– Primitives (int, boolean ...)
– String ...
● Annotation ValuesOf
– Enum, boolean (no random generation, sequential generation)
● Annotation @InRange (min , max)
– integer
● Possible to write our own generator
– @From(XXX.class)
● where XXX is an extension of Generator

15. Junit-Quickcheck
● Limitations
– java8 only
– Junit only (not testNG)
– Shrinking is not working well

16. Quicktheory
● Quickcheck
– Junit or TestNG
– The loop is inside the test (default loop of 1000 times)
● qt.forAll(...)
● Random generation (with seed option)
– Assumption using assuming method
● The loop size is still 1000
– Seed and shrinking working fine

17. Quicktheory generator and source
● Integer / long generation
– All, AllPositive (for integer only), between
– More possibilities using assuming
● Float / Double
– All, from 0 to 1, positive, negative
● Boolean
– All
● You can define your own source and generator

18. Quicktheory
● Limitations
– Less integrated in the test framework
– java8 only
– Lack of documentation

19. Conclusion
● Useful for the robustness tests ?