Selecting a style

ScalaTest supports multiple style traits, approaches to sharing fixtures, and ways to express assertions. Each is focused on solving a particular set of problems. Although having multiple options can help you more easily address different kinds of problems as they arise, it can be daunting to make choices when you are unfamiliar with the strengths and weaknesses of each option. To help you decide which choice best fits your needs, this page will describe the design goals and tradeoffs of each of ScalaTest's style traits, shared fixture approaches, and assertion options.

If you don't enjoy shopping

If you would rather be told which approach to take rather than pick one yourself, the recommended style to use is:

• FunSpec as your style trait,
• ShouldMatchers for your assertions,
• BeforeAndAfter to share fixtures between tests.

Here's an example:

import org.scalatest.FunSpec
import org.scalatest.BeforeAndAfter
import org.scalatest.matchers.ShouldMatchers

class ExampleSpec extends FunSpec with ShouldMatchers with BeforeAndAfter {

  val builder = new StringBuilder

  before {
    builder.append("ScalaTest is ")
  }

  after {
    builder.clear()
  }

  describe("Testing") {

    it("should be easy") {
      builder.append("easy!")
      builder.toString should be ("ScalaTest is easy!")
    }

    it("should be fun") {
      builder.append("fun!")
      builder.toString should be ("ScalaTest is fun!")
    }
  }
}

Using FunSpec will give you the benefits of behavior-driven development (BDD), unlimited nesting of describe clauses, the ability to share and programatically register tests, and a recommended structure for your specification text, guided by the describe and it clauses and demonstrated by the Scaladoc examples. Using ShouldMatchers will give you the greater expressiveness and clarity of matcher statements and their failure messages compared to Assertions. Although not appropriate for every situation, BeforeAndAfter is a good default because it requires zero boilerplate, the code is easy to remember how to write, and the result should be obvious and understandable to most any reader.

Selecting a style trait

If you would rather make your own choices, the following sections will explain the advantages and disadvantages of different options. But first, a table that gives you a quick overview:

Style Trait	Intended Use Case
Suite	This base trait of Suite family can be useful as a style trait in large projects where class file generation is a concern, because Suite represents tests as methods not functions, saving one class file per test.
FunSuite	For teams coming from xUnit, FunSuite feels comfortable and familiar while still giving some of the benefits of BDD: FunSuite makes it easy to write descriptive test names, natural to write focused tests, and generates specification-like output that can facilitate communication among stakeholders.
FlatSpec	A good first step for teams wishing to move from xUnit to BDD, FlatSpec's structure is flat like xUnit, so it is simple and familiar, but the test names must be written in a specification style: "X should Y," "A must B," etc.
FunSpec	For teams coming from Ruby's RSpec tool, FunSpec will feel very familiar; More generally, for any team that prefers BDD, FunSpec's nesting and gentle guide to structuring text (with describe and it) provides an excellent general-purpose choice for writing specification-style tests.
WordSpec	For teams coming from specs or specs2, WordSpec will feel familiar, and is often the most natural way to port specsN tests to ScalaTest. WordSpec is very prescriptive in how text must be written, so a good fit for teams who want a high degree of discipline enforced upon their specification text.
FreeSpec	Because it gives absolute freedom (and no guidance) on how specification text should be written, FreeSpec is a good choice for teams experienced with BDD and able to agree on how to structure the specification text.
PropSpec	Perfect for teams that want to write tests exclusively in terms of property checks; also a good choice for writing the occasional test matrix when a different style trait is chosen as the main unit testing style.
fixture._	The style trait variants in package org.scalatest.fixture are useful if you want or need to get rid of vars in your test classes, such as if you are writing multi-threaded tests. (These are intended to be used in special situations, with a sister trait in org.scalatest used for general needs.)
path._	The style traits in package org.scalatest.path (new in ScalaTest 1.8), path.FunSpec and path.FreeSpec provide a great way to test mutable objects. (These are intended to be used in special situations, with a sister trait in org.scalatest used for general needs.)

Test-driven development (TDD)

The TDD traits, Suite and FunSuite, provide options for people who prefer xUnit style testing.

FunSuite: Recommended for getting started with ScalaTest quickly, in conjunction with BeforeAndAfter and Assertions, because this combination is easiest to learn for people familiar with xUnit frameworks such as JUnit. Because tests are functions, you can share tests or programmatically register them, but at the cost of one generated class file per test. FunSuite brings into scope no implicit conversions except the one it inherits from Assertions, the implicit conversion that puts === on everything, so the likelyhood for implicit conversion conflicts is minimized.

Suite: The base trait of ScalaTest's family of style traits, Suite carries no state internally (it uses reflection to discover and invoke test methods), so it is very lightweight in memory. Because its tests are methods, not functions, you can't share or otherwise programmatically generate tests, but this saves you one class file per test compared to FunSuite. For larger projects where class file generation is a concern, Suite may be a good option.

Behavior-driven development (BDD)

The BDD traits, FunSpec, WordSpec, FlatSpec, and FreeSpec, provide options for people who prefer to think of tests as specifications. Compared to the TDD traits, the BDD traits gives you the added benefits of behavior-driven development: the tests, and their generated artifacts, read like a specification of your software. This allows the tests to not only ensure your software meets requirements, but also to allow stakeholders to communicate about those requirements via the generated artifacts.

FunSpec: Provides describe clauses offering unlimited nesting that can help you organize and minimize duplication in your specification text. Because tests are functions, you can share tests or programmatically register them, but at the cost of one generated class file per test. FunSpec brings into scope no implicit conversions except the one it inherits from Assertions, the implicit conversion that puts === on everything, so the likelyhood for implicit conversion conflicts is minimized. Lastly, the syntax will be familiar to users of RSpec, making it easier for such users to get started. Compiling a FunSpec will generate one extra class file per test and one per nested scope.

WordSpec: Providing implicit conversions that enable you to write specification text by placing pre-selected words after strings, WordSpec removes the tiny bit of boilerplate in required by FunSpec—the describe and it tokens, and the parentheses around the text—at the cost of a slightly higher risk for implicit conversion conflicts. WordSpec also differs from FunSpec in that it enforces, rather than just suggests, a structure for your specification text by chosing the words (should, can, must, when, and that) that can follow the strings. This may help some users (such as users new to BDD) by showing them how to compose the sentences, but for other users it may feel constraining to always need to fit every sentence of specification text into the same structure. WordSpec also only allows three levels of nesting, whereas FunSpec provides unlimited nesting. Some users may find the nesting limit helps them compose their text, but others may find it constraining. (Note: If you like the lack of boilerplate that WordSpec's implicit conversions on strings provides, but want unlimited nesting and unconstrained structure, look at FreeSpec). Lastly, if you are porting legacy specs1 tests to ScalaTest, WordSpec can make porting easier given its DSL is similar in structure to that of the Specification class in specs1. Compiling a WordSpec will generate one extra class file per test and one per nested scope.

FlatSpec: Provides implicit conversions that enable you to write specification text by placing pre-selected words after strings, like WordSpec, but unlike WordSpec, does not allow nesting. The main tradeoffs are: WordSpec's nesting makes it easier to eliminate duplication in specification text, but can make it harder to read the specification text in the source code because you must hop from a test's nearest nested clause to each enclosing nested clause, and stitch the text together in your mind. By contrast, FlatSpec's lack of nesting makes it harder to eliminate duplication in specification text, but easier to read it in the source code: other than the "subject" clause, which can be replaced by it, the text remains together right next to the test code. Like WordSpec, FlatSpec differs from FunSpec in that it provides a prescription for how to structure your specification text by chosing the words (should, can, and must) that can follow the subject strings. This may help some users (such as users new to BDD) by showing them how to compose the sentences, but for other users it may feel constraining to always need to fit every sentence of specification text into the same structure. Fundamentally, FlatSpec differs from its BDD siblings, FunSpec, WordSpec, and FlatSpec, in that it is the only one among these traits that disallows nesting. Compiling a FlatSpec will generate one extra class file per test.

FreeSpec: Like WordSpec and FlatSpec, FreeSpec defines implicit conversions on strings. These implicit conversions remove the tiny bit of boilerplate required by FunSpec (the describe and it clauses, and the parentheses around the text), at the cost of a slightly higher risk of implicit conversion conflicts compared to FunSpec. FreeSpec differs from its siblings primarily in that it allows unlimited nesting and total freedom in how you structure your specification text. Although experienced BDD-ers may feel liberated by FreeSpec's unstructured approach, beginners may feel less sure how to proceed than with a more prescriptive style trait like FunSpec or WordSpec. One other use case for FreeSpec, since its text fragments are not connected by any English words, is to facilitate writing specification text in a (human) language other than English. Compiling a FreeSpec will generate one extra class file per test and one per nested scope.

Acceptance and property-based testing

Although the traits described in this section could also be categorized as BDD traits, the following traits are geared more towards specific applications of BDD: FeatureSpec is designed for acceptance testing; PropSpec for property-based testing.

FeatureSpec: Although also usable for unit testing, FeatureSpec's main intended use is for acceptance testing. It is similar to FunSpec in its structure, but different in that it only allows one level of nesting (one feature clause that can contain a series of scenario clauses), and that the words "feature" and "scenario" show up in the report text whereas FunSpec's "describe" and "it" do not. Because tests are functions, you can share tests or programmatically register them, but at the cost of one generated class file per test. FeatureSpec brings into scope no implicit conversions except the one it inherits from Assertions, the implicit conversion that puts === on everything, so the likelyhood for implicit conversion conflicts is minimized. Compiling a FeatureSpec will generate one extra class file per feature and one per scenario.

PropSpec: A style trait identical in structure to FunSuite (the only difference being the word property is used instead of test), PropSpec is designed to make it more natural to write suites comprised exclusively of property checks, whether table-driven, generator-driven, ScalaCheck-style, or some combination of these. Property checks can be sprinkled into any style of test, but some users prefer to write property-based tests exclusively. Using a PropSpec in such situations makes it more obvious to readers of the code that the suite is comprised solely of property checks.

The style traits in org.scalatest.fixture

The org.scalatest.fixture package provides style traits that support the same styles as their corresponding sister traits in org.scalatest, with one and only one difference: each test takes a fixture parameter. These traits facilitate a functional style of writing tests, helping you factor out fixture code common to different tests without introducing mutable state into your test class. The main purpose of these traits is to make it easier to write thread-safe test classes in situations where you want to write tests that make use of multiple threads.

If you need to write multi-threaded tests, you'll need to ensure the test class that contains them is thread safe. Were you to use the traditional imperative approach to sharing fixtures, you would need to synchronize access to the shared mutable state. An easier, less error-prone, approach to thread safety is to write your test class in a functional style, and that is what the traits in the org.scalatest.fixture package are designed to facilitate. By passing shared data into each test as a parameter, you can avoid the need to synchronize access to shared mutable objects stored in instance variables. Note that if you mix ParallelTestExecution to run tests in parallel, each test will be run in its own instance of the test class, which will usually eliminate the need to make the test class thread safe. Thus the primary reason you'll need to write a thread-safe test class is if it contains one or more tests that themselves make use of multiple threads.

Although designed primarily to facilitate multi-threaded tests, the org.scalatest.fixture traits can also be used by users who prefer to write code in a functional style. The only downsides are 1) they require a tiny bit of extra boilerplate compared to their org.scalatest sisters, the "fixture =>" required at the beginning of each test body, and 2) the withFixture(OneArgTest) methods only stack if the FixtureParam type matches. Thus you usually won't be able to enjoy the benefits of stackable fixture traits when using the org.scalatest.fixture style traits. (You can call withFixture(NoArgTest) from withFixture(OneArgTest), and benefit from stackable traits that way, but if you your goal is to write multi-threaded tests, you'll need to make sure the traits being stacked are thread safe.)

Selecting an approach to shared fixtures

ScalaTest supports multiple ways to factor out duplicate "fixture" code appearing in multiple tests. Each approach is designed to address a different set of problems. This section will explain the differences to help you decide which approach is best for your situation.

BeforeAndAfter: Recommended for getting started with ScalaTest quickly, BeforeAndAfter is a simple, clear way to factor out shared fixture code that should be familiar to anyone who has used an xUnit framework such as JUnit. It's before and after constructs are boilerplate-free, but do not enable stackable fixture traits nor give you easy access to run configuration data. For situations in which you don't need these features, BeforeAndAfter is a good, readable choice.

BeforeAndAfterEach: The closest analog to JUnit's setup and tearDown methods, BeforeAndAfterEach's main use case is to enable stackable fixture traits. In larger projects, fixture setup and cleanup code often needs to be reused in different combinations in multiple test classes. By organizing fixtures into traits that extend BeforeAndAfterEach, with beforeEach and afterEach methods that invoke super implementations, you can mix together just the fixture traits you need in each class. BeforeAndAfterEach also includes overloaded variants of beforeEach and afterEach into which a config map is passed, so you can use run configuration data in your fixture code. The main downside of BeforeAndAfterEach compared to BeforeAndAfter is that it has some boilerplate: you must say override def.

withFixture(NoArgTest): Of all the ways you can factor out shared fixtures, overriding withFixture is the most flexible. The NoArgTest passed to withFixture contains an apply method that executes the body of the test. withFixture is responsible for invoking the test function, and the default implementation of withFixture in trait Suite does only that: it invokes the test function and nothing more. You, however, can override withFixture to do something at the beginning and/or end of each test, or anything else you might need to do with the test function.

One important difference between withFixture and traits BeforeAndAfter and BeforeAndAfterEach is that withFixture's code is part of the test. The test has already started when withFixture is passed the test function, and has not yet completed when the test function returns. By contrast, the code you register with BeforeAndAfter's before method is called before the test starts, the code you register with after is called after the test has completed. You can, therefore, use withFixture to accomplish many things not possible with the before and after approaches. For example, you an change the outcome of a test using withFixture.

Because NoArgTest, in addition to providing the test function, also provides the test's name and a config map, you can use these in your fixture code. And because you can look up a test's tags with its name via the tags method on Suite, you can use tags as well. You could, for example, override withFixture to look up the tags for each test via the name passed in NoArgTest, and for tests tagged "TwoOutOfThreeAintBad," say, run the test function three times and report the test as succeeded if the test function succeeds at least two times. Or you could override withFixture to establish a time limit for all tests: if a test passes, but takes longer than the time limit, your withFixture method can change that success to a failure. Although withFixture will not be familiar to new ScalaTest users, because it does not appear in other frameworks, and requires a bit of boilerplate—the override def and, if you have "after" code to run, a try-finally—it gives you the utmost in power compared to the other alternatives.

withFixture(OneArgTest): A variant of withFixture that passes a fixture parameter into tests, withFixture(OneArgTest) helps you write tests in a functional style. One important use case is if you want to write multi-threaded tests. See the earlier section, the style traits in org.scalatest.fixture, for more information.

OneInstancePerTest: By running each test in its own instance of the test class, OneInstancePerTest helps you isolate tests from each other's side effects. One benefit of this approach is that it eliminates "before" boilerplate: because each test gets its own copy of the instance variables, there's no need for a before clause or beforeEach method. On the other hand, OneInstancePerTest does not provide a way to run code after each test. To do so you'd need to use one of the other approaches such as overriding withFixture, in which case it makes less sense to mix in OneInstancePerTest in the first place.

The approach to test isolation taken by OneInstancePerTest is similar to the way JUnit isolates tests, but with one important difference. If instantiating a test class has a side effect, such as a debug message being printed from the constructor via println, in JUnit you would observe that side effect once per test. Thus if a JUnit test class contains ten tests, you would see the debug statement printed out ten times. But with OneInstancePerTest, you would see it eleven times. That's because the first instance of a suite that mixes in OneInstancePerTest does not execute any tests. Instead, it creates one instance per test—ten instances in this case—and executes just one test in each instance. In situations where an extra instantiation side effect would be problematic, therfore, you should not use OneInstancePerTest.

One use case for the OneInstancePerTest trait is to eliminate boilerplate when you need only to run code before, but not after, tests. Another is to serve as a superclass of traits like ParallelTestExecution, which parallelizes tests by creating one instance for running each test, then executing those suites in parallel.

Just use Scala: Another approach to factoring out shared code is to just use Scala to define create-fixture methods, instantiate fixture traits, or create loan-pattern methods that you use directly in the test bodies. Although these approaches are relatively concise, they do require more boilerplate than the approaches described previously, because you must call the methods or instantiate the traits inside each test. Thus, the best use case for these approaches is when different tests in the same test class require different fixtures. In that case, you'll need to indicate which test requires what fixtures anyway, and calling a create fixture method or instantiating a fixture trait, for example, is a good way to do that. One other benefit of this approach is that there is absolutely no magic going on behind the scenes. It is just Scala, so should be straightforward to understand even by readers completely unfamiliar with ScalaTest.

Selecting an approach to assertions

Assertions: Recommended for getting started with ScalaTest quickly, ScalaTest's Assertions trait provides a very easy to learn, easy to use DSL for assertions. The main tradeoff is that the error messages are not as descriptive as those of matchers.

ShouldMatchers: ScalaTest's matchers allow you to make assertions that give a more descriptive error message when they fail compared to the Assertions DSL. Both matchers and Assertions code is clear and easy to read, but matchers reads more like English statements whereas assertions reads more like code. Because there are more of them, matchers will take longer to learn than Assertions, though not much longer. And whereas Assertions are easy to remember, you may have to look in the documentation (such as the quick reference) from time to time to remember how to write matcher expressions you use rarely.

MustMatchers: ScalaTest allows you to use must instead of should in your matcher expressions by mixing in MustMatchers instead of ShouldMatchers. These behave the same way, so the choice is primarily a matter of personal preference. If you don't have a preference, the recommended style is to use ShouldMatchers.