ScalaTest User Guide

Getting started

Selecting testing styles

Defining base classes

Writing your first test

Using assertions

Tagging your tests

Running your tests

Sharing fixtures

Sharing tests

Using matchers

Testing with mock objects

Property-based testing

Asynchronous testing

Using Selenium

Using Scala-js

Other goodies

Philosophy and design

Migrating to 3.0

Generator-driven property checks

To use generator-driven property checks, you must mix in trait GeneratorDrivenPropertyChecks (or import the members of its companion object). If you are also using table-driven property checks, you can mix in trait PropertyChecks, which extends both GeneratorDrivenPropertyChecks and TableDrivenPropertyChecks. Generator-driven property checks uses ScalaCheck, so you must also include the ScalaCheck JAR file on your classpath when you compile and run your tests.

Trait GeneratorDrivenPropertyChecks contains forAll methods that provide various ways to check properties using generated data. It also contains a wherever method that can be used to indicate a property need only hold whenever some condition is true.

For an example of trait GeneratorDrivenPropertyChecks in action, imagine you want to test this Fraction class:

class Fraction(n: Int, d: Int) {

require(d != 0) require(d != Integer.MIN_VALUE) require(n != Integer.MIN_VALUE)
val numer = if (d < 0) -1 * n else n val denom = d.abs
override def toString = numer + " / " + denom }

To test the behavior of Fraction, you could mix in or import the members of GeneratorDrivenPropertyChecks (and ShouldMatchers) and check a property using a forAll method, like this:

forAll { (n: Int, d: Int) =>

whenever (d != 0 && d != Integer.MIN_VALUE && n != Integer.MIN_VALUE) {
val f = new Fraction(n, d)
if (n < 0 && d < 0 || n > 0 && d > 0) f.numer should be > 0 else if (n != 0) f.numer should be < 0 else f.numer should be === 0
f.denom should be > 0 } }

Trait GeneratorDrivenPropertyChecks provides overloaded forAll methods that allow you to check properties using the data provided by a ScalaCheck generator. The simplest form of forAll method takes two parameter lists, the second of which is implicit. The first parameter list is a "property" function with one to six parameters. An implicit Arbitrary generator and Shrink object needs to be supplied for The forAll method will pass each row of data to each parameter type. ScalaCheck provides many implicit Arbitrary generators for common types such as Int, String, List[Float], etc., in its org.scalacheck.Arbitrary companion object. So long as you use types for which ScalaCheck already provides implicit Arbitrary generators, you needn't worry about them. Same for Shrink objects, which are provided by ScalaCheck's org.scalacheck.Shrink companion object. Most often you can simply pass a property function to forAll, and the compiler will grab the implicit values provided by ScalaCheck.

The forAll methods use the supplied Arbitrary generators to generate example arguments and pass them to the property function, and generate a GeneratorDrivenPropertyCheckFailedException if the function completes abruptly for any exception that would normally cause a test to fail in ScalaTest other than DiscardedEvaluationException. An DiscardedEvaluationException, which is thrown by the whenever method (defined in trait Whenever, which this trait extends) to indicate a condition required by the property function is not met by a row of passed data, will simply cause forAll to discard that row of data.

Supplying argument names

You can optionally specify string names for the arguments passed to a property function, which will be used in any error message when describing the argument values that caused the failure. To supply the names, place them in a comma separated list in parentheses after forAll before the property function (a curried form of forAll). Here's an example:

forAll ("a", "b") { (a: String, b: String) =>
  a.length + b.length should equal ((a + b).length + 1) // Should fail
}

When this fails, you'll see an error message that includes this:

Occurred when passed generated values (
  a = "",
  b = ""
)

When you don't supply argument names, the error message will say arg0, arg1, etc.. For example, this property check:

forAll { (a: String, b: String) =>
  a.length + b.length should equal ((a + b).length + 1) // Should fail
}

Will fail with an error message that includes:

Occurred when passed generated values (
  arg0 = "",
  arg1 = ""
)

Supplying generators

ScalaCheck provides a nice library of compositors that makes it easy to create your own custom generators. If you want to supply custom generators to a property check, place them in parentheses after forAll, before the property check function (a curried form of forAll).

For example, to create a generator of even integers between (and including) -2000 and 2000, you could write this:

import org.scalacheck.Gen

val evenInts = for (n <- Gen.choose(-1000, 1000)) yield 2 * n

Given this generator, you could use it on a property check like this:

forAll (evenInts) { (n) => n % 2 should equal (0) }

Custom generators are necessary when you want to pass data types not supported by ScalaCheck's arbitrary generators, but are also useful when some of the values in the full range for the passed types are not valid. For such values you would use a whenever clause. In the Fraction class shown above, neither the passed numerator or denominator can be Integer.MIN_VALUE, and the passed denominator cannot be zero. This shows up in the whenever clause like this:

whenever (d != 0 && d != Integer.MIN_VALUE
    && n != Integer.MIN_VALUE) { ...

You could in addition define generators for the numerator and denominator that only produce valid values, like this:

val validNumers =
  for (n <- Gen.choose(Integer.MIN_VALUE + 1, Integer.MAX_VALUE)) yield n
val validDenoms =
  for (d <- validNumers if d != 0) yield d

You could then use them in the property check like this:

forAll (validNumers, validDenoms) { (n: Int, d: Int) =>

whenever (d != 0 && d != Integer.MIN_VALUE && n != Integer.MIN_VALUE) {
val f = new Fraction(n, d)
if (n < 0 && d < 0 || n > 0 && d > 0) f.numer should be > 0 else if (n != 0) f.numer should be < 0 else f.numer should be === 0
f.denom should be > 0 } }

Note that even if you are use generators that don't produce the invalid values, you still need the whenever clause. The reason is that once a property fails, ScalaCheck will try and shrink the values to the smallest values that still cause the property to fail. During this shrinking process ScalaCheck may pass invalid values. The whenever clause is still needed to guard against those values. (The whenever clause also clarifies to readers of the code exactly what the property is in a succinct way, without requiring that they find and understand the generator definitions.)

Supplying both generators and argument names

If you want to supply both generators and named arguments, you can do so by providing a list of (<generator>, <name>) pairs in parentheses after forAll, before the property function. Here's an example:

forAll ((validNumers, "n"), (validDenoms, "d")) { (n: Int, d: Int) =>

whenever (d != 0 && d != Integer.MIN_VALUE && n != Integer.MIN_VALUE) {
val f = new Fraction(n, d)
if (n < 0 && d < 0 || n > 0 && d > 0) f.numer should be > 0 else if (n != 0) f.numer should be < 0 else f.numer should be === 0
f.denom should be > 0 } }

Were this property check to fail, it would mention the names n and d in the error message, like this:

Occurred when passed generated values (
  n = 17,
  d = 21
)

Property check configuration

The property checks performed by the forAll methods of this trait can be flexibly configured via the services provided by supertrait Configuration. The five configuration parameters for property checks along with their default values and meanings are described in the following table:

Configuration Parameter Default Value Meaning
minSuccessful 100 the minimum number of successful property evaluations required for the property to pass
maxDiscarded 500 the maximum number of discarded property evaluations allowed during a property check
minSize 0 the minimum size parameter to provide to ScalaCheck, which it will use when generating objects for which size matters (such as strings or lists)
maxSize 100 the maximum size parameter to provide to ScalaCheck, which it will use when generating objects for which size matters (such as strings or lists)
workers 1 specifies the number of worker threads to use during property evaluation

The forAll methods of trait GeneratorDrivenPropertyChecks each take a PropertyCheckConfig object as an implicit parameter. This object provides values for each of the five configuration parameters. Trait Configuration provides an implicit val named generatorDrivenConfig with each configuration parameter set to its default value. If you want to set one or more configuration parameters to a different value for all property checks in a suite you can override this val (or hide it, for example, if you are importing the members of the GeneratorDrivenPropertyChecks companion object rather than mixing in the trait.) For example, if you want all parameters at their defaults except for minSize and maxSize, you can override generatorDrivenConfig, like this:

implicit override val generatorDrivenConfig =
  PropertyCheckConfig(minSize = 10, maxSize = 20)

Or, if hide it by declaring a variable of the same name in whatever scope you want the changed values to be in effect:

implicit val generatorDrivenConfig =
  PropertyCheckConfig(minSize = 10, maxSize = 20)

In addition to taking a PropertyCheckConfig object as an implicit parameter, the forAll methods of trait GeneratorDrivenPropertyChecks also take a variable length argument list of PropertyCheckConfigParam objects that you can use to override the values provided by the implicit PropertyCheckConfig for a single forAll invocation. For example, if you want to set minSuccessful to 500 for just one particular forAll invocation, you can do so like this:

forAll (minSuccessful(500)) { (n: Int, d: Int) => ...

This invocation of forAll will use 500 for minSuccessful and whatever values are specified by the implicitly passed PropertyCheckConfig object for the other configuration parameters. If you want to set multiple configuration parameters in this way, just list them separated by commas:

forAll (minSuccessful(500), maxDiscarded(300)) { (n: Int, d: Int) => ...

If you are using an overloaded form of forAll that already takes an initial parameter list, just add the configuration parameters after the list of generators, names, or generator/name pairs, as in:

// If providing argument names
forAll ("n", "d", minSuccessful(500), maxDiscarded(300)) {
  (n: Int, d: Int) => ...

// If providing generators forAll (validNumers, validDenoms, minSuccessful(500), maxDiscarded(300)) { (n: Int, d: Int) => ...
// If providing (<generators>, <name>) pairs forAll ((validNumers, "n"), (validDenoms, "d"), minSuccessful(500), maxDiscarded(300)) { (n: Int, d: Int) => ...

For more information, see the documentation for trait Configuration, a supertrait of GeneratorDrivenPropertyChecks.

Next, learn about ScalaTest's philosophy and design.

ScalaTest is brought to you by Bill Venners and Artima.
ScalaTest is free, open-source software released under the Apache 2.0 license.

If your company loves ScalaTest, please consider sponsoring the project.

Copyright © 2009-2016 Artima, Inc. All Rights Reserved.

artima