ScalaTest 2.0 Release Notes

ScalaTest 2.0 is a major upgrade to ScalaTest, the culmination of over two years of effort. Although it includes many enhancements, we kept a close eye on compatibility with existing ScalaTest 1.x code. We also tried hard to preserve what people liked about ScalaTest 1.x: that it is simple, the code is clean and clear, it's fully documented, and because of very thorough testing, it “just works.”

For information on how to include ScalaTest in your project, see the download page.

Highlights

ScalaTest 2.0 is such a big upgrade that you could easily get lost all in the details of what's new. This section gives the main highlights subdivided into general categories, with links to more detail. To get a quick idea of what's new in ScalaTest 2.0, scan through this section.

Equality enhancements

Created sister library, ScalaUtils. See scalautils.org for details.
Added ability to customize equality with === in tests and production code, including an “Explicitly DSL.”
Added ability to enforce compile-time type constraints on equality checks with === in tests and production code.
Enhanced ability to do equality checks with tolerance using === with +- in tests and production code.
Added support for normalizing values before equality checks in tests and production code.

Suite enhancements

Added an Outcome result type to withFixture.
Added a Status result type to Suite's run methods, and used Status to ensure BeforeAndAfterAll's afterAll method is executed after any parallel execution completes.
Refactored Suite's run methods to take an Args, to make overriding nicer.
Added markup to style traits to include markup text for specifications.
Added note and alert to stye traits for sending status notifications from tests.
Added Sequential and Stepwise (to pre-existing Suites) combinators for suite composition.
Added support for they (in addition to it) to FunSpec and FlatSpec.
Added ability to tag all tests in a suite class by annotating the class with a tag annotation. For example, can ignore all tests in a suite class with @Ignore.
Added DynaTags and a dynaTags field to the primary constructor of class Filter to enable the dynamic tagging of tests. Dynamic tagging facilitates features such as test name wildcards and rerunning previously failed tests; It enables selection of tests even in nested suites that can't be instantiated directly.
Added the suiteId lifecycle method to trait Suite. This new lifecycle method is intended to provide a unique ID for each suite that executes in a suite of tests, so that tests can be dynamically tagged. (A dynamic tag identifies tests by suiteId and testName.)
Added a rerunner lifecycle method to Suite, which provides an optional, fully qualified name of a suite (with a public, no-arg constructor) that can be used create to rerun its tests. If class has a public, no-arg constructor, then that class's fully qualified name can be returned from rerunner. If it does not have a public, no-arg constructor, but rather is created as a nested suite of a class that does have a public, no-arg constructor, then the nested class can return from rerunner the fully qualified name of that outer, nesting class. To rerun a test in the nested class, ScalaTest can create the nesting class via its public, no-arg constructor, and run it. The nesting class will create the actual class in which the test resides, and run it as a nested suite, thereby rerunning the test.
Added a boolean excludeNestedSuites field to the primary constructor of class Filter. This field supports rerunning selected (such as previously failed) tests.
Added traits Spec and fixture.Spec to serve as the new style trait in which tests are methods, and deprecated the use of Suite as a style trait.
Added a @DoNotDiscover annotation that prevents discovery of an otherwise discoverable Suite class.

SuiteMixin enhancements

Added trait SuiteMixin to serve as the new base class for stackable traits that can be mixed into Suite, and deprecated trait AbstractSuite.
Added TestData trait, made NoArgTest and OneArgTest implement it, and added trait BeforeAndAfterEachTestData to faciliate using the test name, text, scopes, and tags, as well as the config map, in withFixture, beforeEach, and afterEach methods.
Added BeforeAndAfterAllConfigMap to faciliate using the config map in beforeAll and afterAll methods.
Changed the behavior of BeforeAndAfterAll so that it only calls beforeAll and afterAll in the outer instance of a OneInstancePerTest or ParallelTestExecution.
Enhanced BeforeAndAfterAll and BeforeAndAfterAllConfigMap to take expectedTestCount into consideration when deciding whether to invoke beforeAll and afterAll.
Modified OneInstancePerTest so that it calls super.runTests which emits events for scopes (such as describe clauses in FunSpec).
Similarly, modified ParallelTestExecution so that it calls super.runTests which emits events for scopes (such as describe clauses in FunSpec).
Added a Retries trait (and related Retryable tag) and a Catcher class to help deal with flickers: tests that usually pass but occasionally fail.

Assertions enhancements

Added the ability to cancel a test. Added assume and cancel methods to Assertions, and added a cancelAfter method to Timeouts.
Reimplemented assert and assume as macros that produce descriptive error messages in Assertions.
Added Inspectors (forAll, forEvery, forAtLeast, forAtMost, forBetween, forExactly) to enable assertions about collections, including nested collections.
Added LoneElement to assert against single-element collections.
Added TryValues to facilitate assertions about Try values.
Added AsyncAssertions to org.scalatest.concurrent.
Added ScalaFutures to help write tests involving Scala futures.
Added Checkpoints, which allows you to accumulate and report multiple assertions in a single test.

Matchers enhancements

Moved matchers DSL to org.scalatest.Matchers; deprecated ShouldMatchers and MustMatchers.
Refactored matchers to use enabling typeclasses rather than implicit conversions, to make the syntax more broadly usable and easier to customize.
Enhanced contain matcher syntax so it an be used to make richer assertions with oneOf, noneOf, allOf, atLeastOneOf, atMostOneOf, inOrder, only, inOrderOnly, theSameElementsAs, and theSameElementsInOrderAs.
Added inspector shorthands (all, every, atLeast atMost, between, exactly) to the matchers DSL.
Added an empty matcher.
Added a and an matcher syntax to check an object's class.
Added thrownBy matcher syntax for checking for expected exceptions.
Refactored MatchResult to improve runtime performance and enable modification of error messages during matcher composition.
Added Prettifier and PrettyMethods to org.scalautils.
Enhanced have length and have size error messages to include the actual length or size.
Use Ordering instead of Ordered for be </>/<=/>=.

Runner enhancements

For a table of all the available options, see the main Scaladoc documentation for Runner.
Added support for reminders, a summary of failed and/or canceled tests that can be optionally printed out at the end of a run in string (stdout, stderr, file) reporters. This obviates the need to scroll backwards in search of details about failed and/or canceled tests in console output, and also ensures that such details won't have scrolled off the top and out of existence.
Enhanced -s to take a glob in addition to a complete, fully-qualified suite class name.
Added slowpoke notifications, which emit AlertProvided events periodically listing tests that have been running longer than a specified time limit. (-W)
Added ability to memorize and rerun failed and canceled tests. (-M and -A)
Added to Runner a way to specify tests to run by full or partial ("wildcard") name (-z).
Enabled on-the-fly sorting of events (with a timeout) during parallel runs (-PS).
Added unformatted mode to stdout (-oU), stderr (-eU), and file reporters (-fU) for helping debug problematic parallel test runs.
Added a socket reporter (-k), which serializes events to a socket. This facilitates running tests remotely, such as with sbt's forking feature.
Added a -i command line argument to Runner to provide a way to specify a suite by suite ID. This complements the -s command, which allows you to identify a suite by fully qualified name. The -i command was added primarily to allow IDEs and other tools to rerun failed (or otherwise selected) tests.
Added a -t to Runner, a way to specify a specific test name. A -t clauses may follow -s or -i, or be standalone. All -t's immediately following a -s or -i will be considered tests of the suite specified by that -s or -i. If -t <test name> appears standalone, Runner will execute any tests with matching names on any suite discovered in the runpath.
Added a -z argument to Runner, a means to specify test names via a wildcard: any test whose string name includes the string specified with -z will be selected. -z clauses may follow -s or -i clauses, or be standalone. All -z's immediately following a -s or -i will be considered wildcards selecting tests of the suite by that -s or -i. If -z <test name wildcard> appears standalone, Runner will execute any tests with matching names on any suite discovered in the runpath.
Added the ability to sort events on-the-fly when suites are run in parallel, with a tunable timeout (that defaults to 15 seconds) if a suite takes to long to complete. This balances the desire to have sorted output with the desire to see what's happening while you are watching. On-the-fly sorting is disabled by default, and can be enabled with -PS argument to Runner.
Added the ability to specify a timeout for sorting during parallel runs. The -T parameter takes a number of seconds, which is used for test sorting. The value specified with -T plus one second is used for suite sorting.
Added the ability to sort events locally when tests (i.e., not suites) are run in parallel via ParallelTestExecution, with a timeout if a test takes too long. Added sortingTimeout to ParallelTestExecution, the default of which is to use the timeout specified to Runner via -T (or 15 seconds if no -T given), but you can override the method. Added trait DistributedTestSorter to support this feature, and a field holding a DistributedTestSorter in Args

Enhancements for better tools integration

Added new tags CPU, Disk, Network, and reporting these as sbt's built-in resource tags, Tags.CPU, Tags.Disk, and Tags.Network.
Added a location API for better integration with tools, especially IDEs. The location API consists of a sealed family of classes and objects, consisting of abstract base class Location, three case subclasses LineInFile, TopOfMethod, and TopOfClass, and singleton object SeeStackDepthException.
Added a Finders annotation that specifies one or more Finders that can be used to determine selected tests from an AST in IDEs.
Enhanced the ScalaTest Maven Plugin so that all of the new Runner features are accessible to Maven users.
Enhanced the ScalaTest Ant Task so that all of the new Runner features are accessible to Ant users, including Gradle and Buildr users who often go through the Ant task.
Created the ScalaTest Exclipse Plugin, which is a standard part of the Scala IDE Ecosystem.
Defined the New Framework API, and integrated it both with sbt (i.e., we contributed this to sbt) and ScalaTest. The New Framework API makes all features of Runner accessible to the users of sbt, which was not possible with the original Framework API. Support for the new Framework API was released as part of sbt 0.13.

Event model enhancements

Added DiscoveryStarting and DiscoveryCompleted events, and enhanced the GUI reporter to use these events to show a spinning icon during discovery.
Added ScopeOpened and ScopeClosed events to indicate scopes, such as the scope of a describe clause in a FunSpec. Previously a “scope opened” event was indiciated via an InfoProvided events with IndentedText.
Added a MarkupProvided event to indicate some markup text has been provided.
Added a testText field to all test events (TestStarting, TestIgnored, TestPending, TestCanceled, TestFailed, and TestSucceeded), primarily for integration with IntelliJ IDEA. The “test text” is either the complete test name or a suffix of it. In a FunSpec, for example, the test text will be the string passed to the it method, whereas the full test name will be the test text prefixed by the strings passed to enclosing describe method calls.
Added location, an Option[Location] field, to class Event.
Added a suiteId to relevant events: TestStarting, TestSucceeded, TestFailed, TestFailed, TestPending, TestCanceled, SuiteStarting, SuiteCompleted, SuiteAborted. This enables tests run previously to be dynamically tagged in future runs (because a dynamic tag requires both the testName and suiteId).
Added a recordedEvents field to test completion events: TestPending, TestSucceeded, TestFailed, and TestCanceled. Added sealed trait RecordedEvent, which extends Event and has two subtraits: InfoProvided and MarkupProvided. Prior to 2.0, InfoProvided events were recorded during tests and played back after, so that they could appear after the test text in the standard out reporter, and in a color that matched that of the test text. The color of the test text (green if passed, red if failed, yellow if pending, etc.) could not be known until after the test had completed. By sending recorded events along with the test completion event, rather than playing them back afterwords, it is easier for Reporters to figure out when the last event for a test has been received.

Miscellaneous enhancements

Added the ScalaTest Selenium DSL.
Added an HTML reporter. Here's an example HTML report.
To make thread dumps more clear, gave useful names to ScalaTest threads: The main thread is now ScalaTest-main. The run thread is ScalaTest-run. The dispatcher thread is ScalaTest-dispatcher. Thread-pool threads for parallel execution are ScalaTest-N. And main or thread-pool threads running test suites are renamed to ScalaTest-running-<name of suite>, then changed back again once the suite has completed.
Added trait NoArg to support a relatively rare use case motivated by Akka's TestKit.
Added Slow tag for marking slow tests.
Added trait TimesOnInt, which allows you to repeatedly execute a block of code an integer number of times via syntax such as, 3 times println.
Added Or and Every to ScalaUtils, allowing you to represent errors in production code as “alternate return values” (like Either) and to optionally accumulate errors.

Potential breakages

Although we have worked hard to ensure the vast majority of ScalaTest 1.x code to compile under 2.0, ScalaTest 2.0 does include a few enhancements that can break existing code. To see the list, and learn how to migrate your code if affected, see the migration guide.

New deprecations

Deprecated the before/afterEach(TestData) methods in BeforeAndAfterEach in favor of BeforeAndAfterEachTestData.
Deprecated FailureOf in favor of OutcomeOf.
Deprecated expect and expectResult in Assertions in favor of assertResult.
Deprecated the use of Suite as a style trait in favor of newly added Spec.
Deprecated the assert and assume methods whose === and !== operator return Option[String] in favor of assert and assume macros that return Boolean.
Deprecated the plusOrMinus operator used with Matchers in favor of the +- symbol.
Deprecated ShouldMatchers and MustMatchers, both members of package org.scalatest.matchers, have been deprecated in favor of Matchers, which resides in package org.scalatest. (MustMatchers was actually already deprecated in ScalaTest 1.9.2.) For folks using ShouldMatchers, getting rid of the deprecation warning should be as simple as replacing org.scalatest.matchers.ShouldMatchers with org.scalatest.Matchers. For folks using MustMatchers, however, it will unfortnately be slightly more trouble, because the new Matchers trait only supports should not must. So you will need to search and replace your uses of must with should. MustMatchers and must will continue to work for a good long deprecation period, but eventually it will be removed to make way for must possibly coming back later to serve a different purpose. Apologies for this rather large deprecation. Update: MustMatchers was resurrected in 2.1.0 in the org.scalatest package, so there's no need anymore to change must to should. In fact, the org.scalatest.MustMatchers type alias was accidentally left undeprecated in 2.0, so where you are mixing in trait org.scalatest.matchers.MustMatchers, you can in 2.0 get rid of the deprecation warning by mixing in org.scalatest.MustMatchers instead.
Deprecated the beforeAll and afterAll methods of trait BeforeAndAfterAll that take a config map in favor of newly added BeforeAndAfterAllConfigMap.
Deprecated given/when/then/and methods of trait GivenWhenThen in favor of capitalized forms, Given/When/Then/And, because then has been deprecated as an identifier in Scala 2.10.
Deprecated the two previously existing apply methods on Filter, and added two new ones that take an additional suiteId parameter.
Deprecated trait AbstractSuite in favor of newly added trait SuiteMixin, to serve as the new base class for stackable traits that can be mixed into Suite.
Deprecated the apply method on Distributor in favor of a newly added overloaded apply method that takes a Args. During the deprecation period, the old form will call the new form with default args. The purpose of this change was to make more information available to Distributor implementations.

Expired deprecations

Removed BeforeAndAfterEachFunctions and BeforeAndAfterAllFunctions, which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to use BeforeAndAfter instead of BeforeAndAfterEachFunctions, and BeforeAndAfterAll instead of BeforeAndAfterAllFunctions.
Removed MultipleFixtureFeatureSpec, MultipleFixtureFlatSpec, MultipleFixtureFreeSpec, MultipleFixtureFunSuite, MultipleFixturePropSpec, MultipleFixtureSpec, and MultipleFixtureWordSpec, which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to mix in ConfigMapFixture to a fixture.X trait instead.
Removed org.scalatest.SuperSuite, which had been deprecated since ScalaTest 1.5. Please use Suites instead.
Removed the deprecated implicit conversion in the Stopper companion object that converted a Stopper to function type () => Boolean. This implicit conversion was added when the inheritance relationship between Stopper and Function0[Boolean] was severed to make it possible for Stopper to be implemented in Java. (Severing this relationship was originally a request by the IntelliJ IDEA folks, who wanted to write integration code in Java to smooth over binary incompatibilities between different Scala versions.)
Removed the implicit conversion from Reporter to Event => Unit function type in the Reporter companion objecgt, which had been deprecated since ScalaTest 1.5.
Removed the implicit conversion from Rerunner to Function7, which had been deprecated since ScalaTest 1.5.

Detailed history of changes

For historians, the detailed history page preverves relevant sections of the individual release notes for major 2.0 milestones and release candidates. For a quicker to read and more understandable overview, see the Highlights section above.

Acknowledgments

ScalaTest 2.0 is brought to you by Artima, Inc., where it is tended by Bill Venners, Chua Chee Seng, and George Berger. We would like to thank all our users for their suggestions and input, everyone who has contributed source code or reviewed our work-in-progress, and our financial sponsors (who prefer to remain unamed). Without your guidance and support, ScalaTest 2.0 would not have been possible.