Writing ScalaCheck-style properties

ScalaTest supports two styles of property-based testing: ScalaTest style and ScalaCheck style. Both approaches use ScalaCheck to actually check properties when tests are run. This page describes the ScalaCheck style. (For information on the tradeoffs of each style, see the Selecting a property-based testing style section below.)

To write properties in the ScalaCheck style, mix in trait Checkers or import the members of its companion object. Trait Checkers contains several check methods that perform ScalaCheck property checks. If ScalaCheck finds a test case for which a property doesn't hold, the problem will be reported as a ScalaTest test failure.

To use ScalaCheck, you specify properties and, in some cases, generators that generate test data. You need not always create generators, because ScalaCheck provides many default generators for you that can be used in many situations. ScalaCheck will use the generators to generate test data and with that data run tests that check that the property holds. Property-based tests can, therefore, give you a lot more testing for a lot less code than assertion-based tests. Here's an example of using ScalaCheck from a JUnitSuite:

import org.scalatest.junit.JUnitSuite
import org.scalatestplus.scalacheck.Checkers
import org.scalacheck.Arbitrary._
import org.scalacheck.Prop._

class MySuite extends JUnitSuite with Checkers {
  @Test
  def testConcat() {
    check((a: List[Int], b: List[Int]) => a.size + b.size == (a ::: b).size)
  }
}

The check method, defined in Checkers, makes it easy to write property-based tests inside ScalaTest, JUnit, and TestNG test suites. This example specifies a property that List's ::: method should obey. ScalaCheck properties are expressed as function values that take the required test data as parameters. ScalaCheck will generate test data using generators and repeatedly pass generated data to the function. In this case, the test data is composed of integer lists named a and b. Inside the body of the function, you see:

a.size + b.size == (a ::: b).size

The property in this case is a Boolean expression that will yield true if the size of the concatenated list is equal to the size of each individual list added together. With this small amount of code, ScalaCheck will generate possibly hundreds of value pairs for a and b and test each pair, looking for a pair of integers for which the property doesn't hold. If the property holds true for every value ScalaCheck tries, check returns normally. Otherwise, check will complete abruptly with a TestFailedException that contains information about the failure, including the values that cause the property to be false.

For more information on using ScalaCheck properties, see the documentation for ScalaCheck, which is available from http://www.scalacheck.org/.

To execute a suite that mixes in Checkers with ScalaTest's Runner, you must include ScalaCheck's jar file on the class path or runpath.

Property check configuration

The property checks performed by the check methods of this trait can be flexibly configured via the services provided by supertrait Configuration. The five configuration parameters for property checks along with their default values and meanings are described in the following table:

The check methods of trait Checkers each take a PropertyCheckConfiguration object as an implicit parameter. This object provides values for each of the five configuration parameters. Trait Configuration provides an implicit val named generatorDrivenConfig with each configuration parameter set to its default value. If you want to set one or more configuration parameters to a different value for all property checks in a suite you can override this val (or hide it, for example, if you are importing the members of the Checkers companion object rather than mixing in the trait.) For example, if you want all parameters at their defaults except for minSize and maxSize, you can override generatorDrivenConfig, like this:

implicit override val generatorDrivenConfig =
  PropertyCheckConfiguration(minSize = 10, maxSize = 20)

Or, if hide it by declaring a variable of the same name in whatever scope you want the changed values to be in effect:

implicit val generatorDrivenConfig =
  PropertyCheckConfiguration(minSize = 10, maxSize = 20)

In addition to taking a PropertyCheckConfiguration object as an implicit parameter, the check methods of trait Checkers also take a variable length argument list of PropertyCheckConfigParam objects that you can use to override the values provided by the implicit PropertyCheckConfiguration for a single check invocation. You place these configuration settings after the property or property function, For example, if you want to set minSuccessful to 500 for just one particular check invocation, you can do so like this:

check((n: Int) => n + 0 == n, minSuccessful(500))

This invocation of check will use 500 for minSuccessful and whatever values are specified by the implicitly passed PropertyCheckConfiguration object for the other configuration parameters. If you want to set multiple configuration parameters in this way, just list them separated by commas:

check((n: Int) => n + 0 == n, minSuccessful(500), maxDiscarded(300))

The previous configuration approach works the same in Checkers as it does in ScalaCheckDrivenPropertyChecks. Trait Checkers also provides one check method that takes an org.scalacheck.Test.Params object, in case you want to configure ScalaCheck that way.

import org.scalacheck.Prop
import org.scalacheck.Test.Params
import org.scalatestplus.scalacheck.Checkers._

check(Prop.forAll((n: Int) => n + 0 == n), Params(minSuccessfulTests = 5))

For more information, see the documentation for supertrait Configuration.

Selecting a property-based testing style

For the ScalaTest style, you mix in trait ScalaCheckDrivenPropertyChecks and express properties with whenever clauses and matcher expressions. For the ScalaCheck style, you mix in trait Checkers and express properties as boolean expressions, using ScalaCheck's native operators such as ==>, :|, |:, and so on, as needed. The purpose of the ScalaTest style is to make property-based testing more consistent with the other styles of testing that ScalaTest supports.

ScalaCheckDrivenPropertyChecks is an alternative to Checkers. Both are essentially front-ends to ScalaCheck, and both require that ScalaCheck's jar file be on the class path. The difference is Checkers facilitates the traditional ScalaCheck style of writing properties, whereas ScalaCheckDrivenPropertyChecks facilitates a ScalaTest style of writing properties that takes advantage of ScalaTest's assertions and matchers. To see the difference, consider an example written in each style. With Checkers you'd usually write a property function like this:

check { (n: Int) =>
  n > 1 ==> n / 2 > 0
}

This is the native ScalaCheck style: you write a function that takes some data, then use that data in a boolean expression, returning the result. If the property is only valid for a subset of the full range of values of the types being passed (for example, in this case the expression only makes sense for integers greater than one), you would use the implication operator ==>. The code, n > 1 ==> in front of n / 2 > 0 essentially means that whenever n > 1, n / 2 should be greater than 0. In the ScalaTest property style you use the word whenever instead of ==> and either an assertion or matcher expression instead of a boolean expression:

forAll { (n: Int) =>
  whenever (n > 1) { n / 2 should be > 0 }
}

The two main goals I have for providing a ScalaTest style of writing properties are consistency and clarity. By allowing people to write properties similarly to how they write regular unit test, I'm hoping it will make property-based tests easier to write and read, and thereby make property-based testing more accessible to the many programmers who have never seen it before. That's the consistency.

Clarity is a general goal I have when I make design decisions for ScalaTest. When someone looks at some ScalaTest code written by someone else, to the extent possible I want them to be able to understand the intent of the programmer without having to look anything up in the ScalaTest documentation. I want it to be plainly obvious. The ==> operator does not pass that test, but whenever does. Also I think readers may find it hard to parse properties where => is near ==> because they look so similar. Moreover, things can get harder to read with the traditional ScalaCheck style when properties get more complex. Here's a Checkers example inspired by code in the ScalaCheck documentation describing ScalaCheck labels:

def myMagicFunction(n: Int, m: Int) = n + m

check { (m: Int, n: Int) =>
  val res = myMagicFunction(n, m)
  (res >= m)    :| "result > #1" &&
  (res >= n)    :| "result > #2" &&
  (res < m + n) :| "result not sum"
}

Like ==>, the :| operator doesn't pass the test of not needing to look things up in the documentation. What :| (and its cousin, |:) does is attach a string label to a part of the boolean expression. That way if a complex property fails, you can more easily figure out which part of the expression caused the failure. When you check the above labeled property it will fail with arguments (0, 0) and indicate the label of the failing part was "result not sum".

By contrast, the equivalent property in the ScalaTest style might look like this:

forAll { (m: Int, n: Int) =>
  val res = myMagicFunction(n, m)
  res should be >= m
  res should be >= n
  res should be < m + n
}

I think this style is easier to both write and read. When it fails, it tells you the failing arguments were (0, 0), gives you a failure message that reads, "0 was not less than 0," and provides the line number of the failing expression, which is res should be < m + n. Alternatively, you could write it this way:

forAll { (m: Int, n: Int) =>
  val res = myMagicFunction(n, m)
  res should (be >= m and be >= n and be < m + n)
}

This form would also fail indicating the args were (0, 0), giving the line number of the offending matcher expression and failure message that reads, "0 was greater than or equal to 0, and 0 was greater than or equal to 0, but 0 was not less than 0." So either way you write this property in ScalaTest style, it is pretty easy to figure out what went wrong, and you figure it out the same way you would for regular (non-property-based) tests.

Next, learn about ScalaTest's philosophy and design.