Property-based testing

ScalaTest supports property-based testing, where a property is a high-level specification of behavior that should hold for a range of data points. For example, a property might state that the size of a list returned from a method should always be greater than or equal to the size of the list passed to that method. This property should hold no matter what list is passed.

The difference between a traditional test and a property is that tests traditionally verify behavior based on specific data points checked by the test. A test might pass three or four specific lists of different sizes to a method under test that takes a list, for example, and check the results are as expected. A property, by contrast, would describe at a high level the preconditions of the method under test and specify some aspect of the result that should hold no matter what valid list is passed.

In ScalaTest, properties are specified as functions and the data points used to check properties can be supplied by either tables or generators. Generator-driven property checks are performed via integration with ScalaCheck through ScalaTest + ScalaCheck library, by including:

libraryDependencies += "org.scalatestplus" %% "scalacheck-1-17" % "3.2.17.0" % "test"

in your SBT file, or if you use Maven:

<dependency>
    <groupId>org.scalatestplus</groupId>
    <artifactId>scalacheck-1-17_2.13</artifactId>
    <version>3.2.17.0</version>
    <scope>test</scope>
</dependency>    

To use this style of testing, mix in trait org.scalatestplus.scalacheck.ScalaCheckPropertyChecks (previously known as org.scalatest.props.PropertyChecks, deprecated in 3.0.6), or import the members of its companion object.

As an example property-based testing using both table- and generator-driven properties, imagine you want to test this Fraction class:

class Fraction(n: Int, d: Int) {

  require(d != 0)
  require(d != Integer.MIN_VALUE)
  require(n != Integer.MIN_VALUE)

  val numer = if (d < 0) -1 * n else n
  val denom = d.abs

  override def toString = numer + " / " + denom
}

If you mix in ScalaCheckPropertyChecks, you could use a generator-driven property check to test that the passed values for numerator and denominator are properly normalized, like this:

forAll { (n: Int, d: Int) =>

  whenever (d != 0 && d != Integer.MIN_VALUE
      && n != Integer.MIN_VALUE) {

    val f = new Fraction(n, d)

    if (n < 0 && d < 0 || n > 0 && d > 0)
      f.numer should be > 0
    else if (n != 0)
      f.numer should be < 0
    else
      f.numer should be === 0

    f.denom should be > 0
  }
}

The forAll method takes a function that expresses a property of Fraction that should hold for any valid values passed as n and d. (This property specifies how the numerator and denominator passed to the Fraction constructor should be normalized.) The whenever clause indicates invalid values for n and d. Any other values are valid. When run, ScalaCheck generators will be passed implicitly to forAll and supply integer values for n and d. By default the property will be checked against 10 valid pairs of n and d. Note that this is different from ScalaCheck's default of 100.

You might place the previous property check in its own test whose name describes the property at a high level, such as "Fractions should be normalized". In another test you might use a table-driven property check to test that all combinations of invalid values passed to the Fraction constructor produce the expected IllegalArgumentException, like this:

val invalidCombos =
  Table(
    ("n",               "d"),
    (Integer.MIN_VALUE, Integer.MIN_VALUE),
    (1,                 Integer.MIN_VALUE),
    (Integer.MIN_VALUE, 1),
    (Integer.MIN_VALUE, 0),
    (1,                 0)
  )

forAll (invalidCombos) { (n: Int, d: Int) =>
  evaluating {
    new Fraction(n, d)
  } should produce [IllegalArgumentException]
}

In this example, invalidCombos is a table of two columns, produced by the Table factory method, which takes a comma-separated list of tuples. The first tuple contains the headings (the names of the columns, "n" and "d"). The remaining tuples represent the rows of data. In this example, each row is one example of how invalid data that could be passed to the Fraction constructor.

After the declaration of the table, forAll is invoked. This forAll method takes the table, invalidCombos, and a property. forAll checks to make sure the property holds for each row of the table.

When you get a failed property check, ScalaTest will report the failure with initial seed so that you can reproduce the failure. You'll need to pass the initial seed value through -S argument, here's an example you pass it through build.sbt:

testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-S", "123456789")

sbt> testOnly MyTest -- -S 123456879

For more information check out the user guide pages for:

• Table-driven property checks
• Generator-driven property checks
• Writing ScalaCheck-style properties

Next, learn about Asynchronous testing.