ScalaTest 2.0.M5b Release Notes

ScalaTest 2.0 is a major release focused on easing one of the main pain points for Scala users: tools support. The ScalaTest 2.0 effort has involved not only enhancements to ScalaTest, but also to tools themselves, such as IntelliJ IDEA, the Scala IDE for Eclipse, and sbt. We still have a bit more work to do prior to releasing ScalaTest 2.0 final, but 2.0.M5b is a very stable, fully tested and documented milestone release that already includes the changes in the detailed list below. For a higher level view, here are some highlights of 2.0.M5b:

2.0.M5b Highlights:

• Added an HTML reporter that gives a one-page view of the results of one test run. The HTML report includes a summary with a pie chart at the top left and a table of suites below that. Each row of the table gives the summary of one suite. By default the table is sorted to show failing suites, if any, at the top. You can resort the table by clicking on the column names, and view the detailed report of a suite by clicking on the suite name in the table. The detailed report will appear in the right half of the page. You can click on a link at the bottom of the suite report to open it in its own tab for printing. Here's an example HTML report generated from ScalaTest's run of itself. You can request an HTML report via the -h command of Runner.
• Added a Status return type to the four “run methods” of trait Suite: run, runNestedSuites, runTests, and runTest. The Status can be queried to determine whether all the tests and/or nested suites started by the “run method” have completed and, if completed, whether any failure occurred. Created four subclasses of this trait for convenience: objects FailedStatus and SucceededStatus, useful when the “run” has already completed (such as when it was run on the same thread), class StatefulStatus, useful when a “run” has not yet completed (such as when multiple threads are involved), and CompositeStatus, useful for composing multiple Status objects into one Status. Status was added primarily to enable BeforeAndAfterAll to wait until any parallel events started by invoking run complete before executing the “after all” code. This will break any subclass that overrode one of the four run methods, but those are also already broken because of the change to the method's arguments. To fix it, you'll need to declare Status as the return type of your overloaded method, and return a Status.
• Enhanced BeforeAndAfterAll so that it uses the Status object returned by run to wait until any parallel execution of tests and nested suites has completed before executing the “after all” code.
• Dropped implicit conversions from structural types that enabled "have length/size" to be used on any object with an appropriate structure (such as a getSize method). In its place created Length and Size type class traits. This will be a breaking change for anyone who was relying on those structural implicits. This should affect very little code, but if it hits you, you'll need to create a type class as described in the Scaladoc for Length and Size.
• Deprecated given/when/then/and methods of trait GivenWhenThen in favor of capitalized forms, Given/When/Then/And, because then has been deprecated as an identifier in Scala 2.10.
• Bumped matcher collection types from Traversable to GenTraversable, Seq to GenSeq, etc., to make it easier to test with parallel collections.
• Added traits Spec and fixture.Spec to serve as the new style trait in which tests are methods.
• Deprecated trait Suite as a style trait.
• Added trait SuiteMixin to serve as the new base class for stackable traits that can be mixed into Suite, and deprecated trait AbstractSuite.
• Added the ScalaTest Selenium DSL.
• Added trait TimesOnInt, which allows you to repeatedly execute a block of code an integer number of times via syntax such as, 3 times println.
• Added the ability to dynamically tag tests, which facilitates features such as test name wildcards and rerunning previously failed tests; Dynamic tagging enables selection of tests even in nested suites that can't be instantiated directly.
• Added to Runner a way to specify tests to run by full or partial ("wildcard") name.
• Enabled on-the-fly sorting of events (with a timeout) during parallel runs.
• Added the ability to cancel tests if a needed resource is not available.
• Added ability to tag all tests in a suite class by annotating the class with a tag annotation. For example, can ignore all tests in a suite class with @Ignore.
• Added a @DoNotDiscover annotation that prevents discovery of an otherwise discoverable Suite class.
• Simplified Suite's run method signatures, to make them more pleasant to override.
• Enabled pretty output for test methods defined in backticks; Deprecated old-style test method names in Suite in favor of `test: ...` style.
• DSL-ified Suite's execute method, including treating a given test name as a wildcard.
• Reworked much of ScalaTest's extensive Scaladoc documentation to make code examples simpler, clearer, ensuring only the best, recommended style is shown. Added "Recommended Usage" callout boxes, tables, and text to ensure it is easy to understand the intended purpose of each of the various traits and features.
• Added unformatted mode to stdout, stderr, and file reporters for helping debug problematic parallel test runs.
• Added TestData trait that offers information about a test, some of which was already being passed into withFixture. In addition to the config map and test name, TestData also offers test text, scopes, and tags. In addition to being mixed into NoArgTest and OneArgTest, which are passed to withFixture, new forms of beforeEach and afterEach were added to BeforeAndafterEach that take a TestData (and the old forms that took only a config map were deprecated). This gives beforeEach and afterEach methods to the same information enjoyed by withFixture methods.

Detailed list of changes:

The ScalaTest 2.0.M5b release includes the following enhancements, bug fixes, breaking changes, and deprecations:

Enhancements:

• Added traits Spec and fixture.Spec to serve as the new style trait in which tests are methods.
• Deprecated trait Suite as a style trait.
• Added trait SuiteMixin to serve as the new base class for stackable traits that can be mixed into Suite, and deprecated trait AbstractSuite.
• Added trait TimesOnInt, which allows you to repeatedly execute a block of code an integer number of times via syntax such as, 3 times println.
• Added the org.scalatest.selenium package, which provides the ScalaTest Selenium DSL.
for testing web applications with Selenium.
• Added unformatted mode to stdout, stderr, and file reporters for helping debug problematic parallel test runs.
• Added case class Args, to hold the arguments passed into Suite's run, runNestedSuites, runTests, and runTest methods. The purpose of Args is to simplify the signatures of these four "run" lifecycle methods, making the signatures easier to remember and the methods more pleasant to override.
• Added chosenStyles to Args, to hold the (possibly empty) set of styles chosen for the project. Previously this was passed around via the config map.
• Added an overloaded run method to AbstractSuite that takes an optional test name string and a Args. Deprecated the old (and now overloaded) run method that takes seven args and added a final implementation of it to AbstractSuite that calls the new, two-arg form of run.
• Changed runNestedSuites method so instead of taking six parameters it takes a single Args parameter. This will be a breaking change to any subtrait or subclass that overrode or invoked this method. See the breaking changes section for info on how to fix it.
• Changed runTests method so instead of taking an optional test name string plus six parameters it takes the optional test name string and a single Args parameter. This will be a breaking change to any subtrait or subclass that overrode or invoked this method. See the breaking changes section for info on how to fix it.
• Changed runTest method so instead of taking a test name string and six parameters it takes the test name string and a single Args parameter. This will be a breaking change to any subtrait or subclass that overrode or invoked this method. See the breaking changes section for info on how to fix it.
• Narrowed the result types of newInstance in OneInstancePerTest and ParallelTestExecution to include the trait type. I.e., previously the result type of newInstance was just Suite. Now in OneInstancePerTest the result type is Suite with OneInstancePerTest, and in ParallelTestExecution is Suite with ParallelTestExecution.
• Added several overloaded assume methods to trait Assertions. For symmetry, each assume method corresponds to an overloaded assert method with an identical signature. The assume methods behave similarly to the corresponding assert methods, except instead of throwing TestFailedException the assume methods throw TestCanceledException.
• Added several overloaded cancel methods to trait Assertions. For symmetry, each cancel method corresponds to an overloaded fail method with an identical signature. The cancel methods behave similarly to the corresponding fail methods, except instead of throwing TestFailedException the cancel methods throw TestCanceledException.
• Added a TestCanceled event to indicate a test has been canceled.
• Added a TestCanceledException that indicates a test has been canceled. (This is thrown by the cancel and assume methods added in this release to trait Assertions.)
• Added ScopeOpened and ScopeClosed events to indicate "scopes," such as the scope of a describe clause in a FunSpec. Previously a "scope opened" event was indiciated via an InfoProvided events with IndentedText.
• Added a MarkupProvided event to indicate some markup text has been provided.
• Added a markup() method to style traits in which tests are functions (I.e., the same style traits that already offered an info() method.) to allow markup text to be sent to the reporter from those traits. Added trait Documenter, the result type of markup.
• Enhanced stdout, stderr, and file reporters to display markup text.
• Added a testText field to all test events (TestStarting, TestIgnored, TestPending, TestCanceled, TestFailed, and TestSucceeded), primarily for integration with IntelliJ IDEA. The "test text" is either the complete test name or a suffix of it. In a FunSpec, for example, the test text will be the string passed to the it method, whereas the full test name will be the test text prefixed by the strings passed to enclosing describe method calls.
• Added a location API for better integration with tools, especially IDEs. The location API consists of a sealed family of classes and objects, consisting of abstract base class Location, three case subclasses LineInFile, TopOfMethod, and TopOfClass, and singleton object SeeStackDepthException. (For the time being there's also a ToDoLocation we're using to mark places that aren't yet firing proper locations, but this will be removed before 2.0.final is released.)
• Added location, an Option[Location] field, to class Event.
• Gave Ordinal nicer toString output, for example: "Ordinal(99, 1, 0)".
• Added a boolean excludeNestedSuites field to the primary constructor of class Filter. This field supports rerunning selected (such as previously failed) tests.
• Added class DynaTags to support the dynamic tagging of tests. The dynamic tagging feature (new in 2.0) can be used, for example, to select a subset of tests to run (such as tests that previously failed), by tagging them with "Selected" and only including tests tagged "Selected".
• Added a dynaTags field to the primary constructor of class Filter. This field supports the dynamic tagging of tests.
• Added the suiteId lifecycle method to trait AbstractSuite and its subtraits. This new lifecycle method is intended to provide a unique ID for each suite that executes in a suite of tests, so that tests can be dynamically tagged. (A dynamic tag identifies tests by suiteId and testName.)
• Added a suiteId to relevant events: TestStarting, TestSucceeded, TestFailed, TestFailed, TestPending, TestCanceled, SuiteStarting, SuiteCompleted, SuiteAborted. This enables tests run previously to be dynamically tagged in future runs (because a dynamic tag requires both the testName and suiteId).
• Deprecated the previous two apply methods on Filter, and added two new ones that take an additional suiteId parameter.
• Added the rerunner lifecycle method to AbstractSuite and its subtraits, which provides an optional fully qualified name of a suite (with a public, no-arg constructor) that can be used create to rerun its tests. If class has a public, no-arg constructor, then that class's fully qualified name can be returned from rerunner. If it does not have a public, no-arg constructor, but rather is created as a nested suite of a class that does have a public, no-arg constructor, then the nested class can return from rerunner the fully qualified name of that outer, nesting class. To rerun a test in the nested class, ScalaTest can create the nesting class via its public, no-arg constructor, and run it. The nesting class will create the actual class in which the test resides, and run it as a nested suite, thereby rerunning the test.
• Added a socket reporter. This can be selected by passing a -k argument to Runner.
• Added to the summary output of "string" reporters—i.e., the stdout, stderr, and file reporters—how many tests were canceled during a run.
• Added decodedTestName, an Option[String], to test-related events. If defined, it means that the test was named using Scala's backtick notation, such as a test method in a Suite named `test: an empty stack should complain on pop`. The real test name will be encoded as test$colon$u0020an$u0020empty$u0020stack$u0020should$u0020complain$u0020on$u0020pop, whereas the decoded name will be test: an empty stack should complain on pop. This enables test names to be defined with backtick notation and still be presented to users in the more human-friendly, decoded form. If the decoded name would be identical to the encoded name (which will be the case if the name was not expressed in backticks and therefore never encoded in the first place), decodedTestName will be None. The events to which decodedTestName was added are: TestStarting, TestSucceeded, TestFailed, TestIgnored, TestPending, TestCanceled.
• Added decodedSuiteName, an Option[String], to suite-related events. If defined, it means that the suite was named using Scala's backtick notation, such as `Stack Spec`. The real suite name will, by default, be encoded as Stack$u0020Spec, whereas the decoded name will be Stack Spec. This enables suite names to be defined with backtick notation and still be presented to users in the more human-friendly, decoded form. If the decoded name would be identical to the encoded name (which will be the case if the name was not expressed in backticks and therefore never encoded in the first place), decodedSuiteName will be None. The events to which decodedSuiteName was added are: TestStarting, TestSucceeded, TestFailed, TestIgnored, TestPending, TestCanceled, SuiteStarting, SuiteCompleted, SuiteAborted.
• Added a DoNotDiscover annotation that can be used to tell Runner not to discover a suite class that it would otherwise discover.
• Added a Finders annotation that specifies one or more Finders that can be used to determine selected tests from an AST in IDEs.
• Added a DoNotDiscover annotation to prevent Suite classes that would otherwise be discoverable (because they extend Suite and have a public, no-arg constructor) from being discovered by Runner.
• Added a -i command line argument to Runner to provide a way to specify a suite by suite ID. This complements the -s command, which allows you to identify a suite by fully qualified name. The -i command was added primarily to allow IDEs and other tools to rerun failed (or otherwise selected) tests.
• Added a -t to Runner, a way to specify a specific test name. A -t clauses must follow -s or -i. All -t's immediately following a -s or -i will be considered tests of the suite specified by that -s or -i.
• Added a -z argument to Runner, a means to specify test names via a wildcard: any test whose string name includes the string specified with -z will be selected. -z clauses must follow -s or -i clauses. All -z's immediately following a -s or -i will be considered wildcards selecting tests of the suite by that -s or -i. (Note: This approach to wildcard test names was suggested by Mathias Doenitz.)
• Changed the result type of nestedSuites from List to immutable.IndexedSeq. (Under the covers we currently return a Vector.) When ScalaTest 1.0 was initially released, Vector did not yet exist in the Scala library and List was considered by many to be the default Seq type to use. In practice, it was discovered that it is too easy to use List (and other LinearSeq's) in a non-performant way. In 2.0 we wanted to take the opportunity to make the potentially breaking change of replacing List result types with immutable.IndexedSeq. This potentially breaking change is expected to affect very little, if any, user code.
• Added a new overloaded apply method to Distributor that takes a Args, and deprecated the old form. During the deprecation period, the old form will call the new form with default args. The purpose of this change was to make more information available to Distributor implementations.
• Added a recordedEvents field to test completion events: TestPending, TestSucceeded, TestFailed, and TestCanceled. Added sealed trait RecordedEvent, which extends Event and has two subtraits: InfoProvided and MarkupProvided. Prior to 2.0, InfoProvided events were recorded during tests and played back after, so that they could appear after the test text in the standard out reporter, and in a color that matched that of the test text. The color of the test text (green if passed, red if failed, yellow if pending, etc.) could not be known until after the test had completed. By sending recorded events along with the test completion event, rather than playing them back afterwords, it is easier for Reporters to figure out when the last event for a test has been received. Enhanced filtering so that recorded events would be filtered out if the user asked for that.
• Changed a primary/auxiliary constructor pair to a primary constructor with a default parameter in Tracker.
• Added default values for tagsToInclude and tagsToExclude in Filter.
• Updated the JUnit XML Reporter so that it no longer uses ordinals in determining which events belonged to which suites. It now uses suiteId.
• Added the ability to sort events locally when tests are run in parallel via ParallelTestExecution, with a timeout if something takes too long. This balances the desire to have sorted output while you are watching with the desire to see stuff happening while you are watching. Added sortingTimeout to ParallelTestExecution, the default of which is to use the timeout specified to Runner via -T (or 15 seconds if no -T given), but you can override the method. Added trait DistributedTestSorter to support this featuren, and a field holding one in Args
• Added the ability to specify a timeout for sorting during parallel runs. The -T takes a number of seconds, and this is used for test sorting. The value specified with -T plus one second is used for suite sorting.
• Added the ability to sort events locally when suites are run in parallel. This is disabled by default, and can be enabled with -PS argument to Runner.
• Made the suiteId of DiscoverySuite unique in a run.
• Ensure info() invocations in fixture.Suite are recorded and reported in the completion event.
• Modified OneInstancePerTest so that it calls super.runTests which emits events for scopes (such as describe clauses in FunSpec).
• Similarly, modified ParallelTestExecution so that it calls super.runTests which emits events for scopes (such as describe clauses in FunSpec).
• Added duration to TestPending event.
• Changed the overloaded apply factory methods in Filter to one apply method with default args.
• Deprecated the old form of the public constructor in Filter, in favor of just using the apply method in the Filter companion object.
• Adding type aliases in org.scalatest package object for ShouldMatchers and MustMatchers to save an import when using matchers.
• Added new-style test method names (in backticks, starting with "test: ") in Suite and deprecated the old-style names.
• Changed ConfigMapFixture so that it calls withFixture(test.toNoArgTest(...)) in its withFixture(OneArgTest) implementation. Previously it was just calling the test function directly. This enhancement enables trait stacking.
• Added expectResult to Assertions, and deprecated expect. expect will be used for a different purpose in a future version of ScalaTest.
• Did a major overhaul of the ScalaTest Scaladoc documentation. Most examples were rewritten to be simpler and clearer, ensuring they only show the best, recommended style. Callout boxes, tables, and text were added to ensure it is easy to understand the recommended usage of the various tools. Also aded a table of command-line arguments for Runner.
• Enable tagging of all tests in a suite by annotating the suite class with a tag annotation. One common use case for this is to annotate a suite class with @Ignore to ignore all tests in the suite.
• Hacked up a way to get ScalaTest's annotations (@DoNotDiscover, @Ignore, @Style, @TagAnnotation, and @WrapWith), which must be implemented in Java, to appear in ScalaTest's Scaladoc. This was done by pretending they are traits that extend java.lang.annotation.Annotation, and explaining at the top of the documentation that they are really Java @interfaces.
• Renamed Matchers in package org.scalatest.matchers to ClassicMatchers and deprecated Matchers.

Bug fixes

• Report an aborted suite as a failed test when running with JUnitRunner.
• Print should/can/must from WordSpec and fixture.WordSpec in the standard out reporter for failed, canceled, and ignored tests.
• Include a skipped attribute in the JUnit XML output for ignored, pending, and canceled tests.

Deprecations

• Deprecated expect and add expectResult in Assertions.
• Deprecated old-style test method names in Suite.
• Renamed Matchers to ClassicMatchers and deprecated Matchers. This was an implementation class, so likely few if any users actually used this in their code. But if you did, you'll need to rename that to ClassicMatchers to make way for a new more prominant use of Matchers in a future release of ScalaTest.

Removals

• Removed BeforeAndAfterEachFunctions and BeforeAndAfterAllFunctions, which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to use BeforeAndAfter instead of BeforeAndAfterEachFunctions, and BeforeAndAfterAll instead of BeforeAndAfterAllFunctions.
• Removed MultipleFixtureFeatureSpec, MultipleFixtureFlatSpec, MultipleFixtureFreeSpec, MultipleFixtureFunSuite, MultipleFixturePropSpec, MultipleFixtureSpec, and MultipleFixtureWordSpec, which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to mix in ConfigMapFixture to a fixture.X trait instead.
• Removed org.scalatest.SuperSuite, which had been deprecated in ScalaTest 1.5. Please use Suites or Specs instead.
• Removed the deprecated implicit conversion in the Stopper companion object that converted a Stopper to function type () => Boolean. This implicit conversion was added when the inheritance relationship between Stopper and Function0[Boolean] was severed to make it possible for Stopper to be implemented in Java. (Severing this relationship was originally a request by the IntelliJ IDEA folks, who wanted to write integration code in Java to smooth over binary incompatibilities between different Scala versions.)
• Added a default method in the Stopper companion object that returned a stopper that always returns false when applied.

Breaking changes

ScalaTest 2.0 includes a few changes for which it was not possible to go through a deprecation cycle, so some user code written under previous 1.x versions of ScalaTest will not compile unchanged under 2.0. Because the breaking changes do not involve actual test code, but rather the plumbing underneath, the breakages are expected to affect at most 5% of users, and less than 1% of actual code. The vast majority of users should be able to upgrade from 1.8 to 2.0 with a simple recompile, so long as they clear any deprecation warnings prior to the upgrade.

• By defining a final implementation of AbstractSuite's run method that takes seven arguments, subclasses and subtraits that had previously overridden it will break. To fix this, you'll need to override the new form of run that takes two arguments instead. An easy way to fix this is to replace the last six arguments in your overriden run method signature with "args: Args", then add an "import args._" at the top of your method body.
• Changed the result type of nestedSuites from List to immutable.IndexedSeq. (Under the covers we currently return a Vector.) When ScalaTest 1.0 was initially released, Vector did not yet exist in the Scala library and List was considered by many to be the default Seq type to use. In practice, it was discovered that it is too easy to use List (and other LinearSeq's) in a non-performant way. In 2.0 we wanted to take the opportunity to make the potentially breaking change of replacing List result types with immutable.IndexedSeq. This potentially breaking change is expected to affect very little, if any, user code, essentially only code that called nestedSuites then used :: or ::: operators on the result. To fix such a break, use +: instead of :: and ++ instead of :::.
• By establishing a Status return type to run, runNestedSuites, runTests, and runTest, code that previously overrode these methods will need to be modified to return a Status. Most implementations will be able to return the same Status returned when super.run... is invoked. If you need to return your own Status object, you may be able to use one of the convenience classes and objects: FailedStatus, SucceededStatus, StatefulStatus, and CompositeStatus.
• Made major enhancements to the event model, primarily to support tools integration, but also to support a few new features unrelated to tools integration. These enhancements form a braeaking change becuase it is not possible to deprecate extractors in Scala, so pattern matches on the old-style events will likely break. This is the main breakage in ScalaTest 2.0, but it only effects users who created customer Reporters, which was very rare, probably less than 1% of users.
• Because implicit conversions from structural types that enabled "have length/size" to be used on any object with an appropriate structure (such as a getSize method were removed from ScalaTest's matchers in 2.0.M5, any code relying on this feature will no longer compile. This should affect very little code, but if it hits you, you'll need to create a type class as described in the Scaladoc for Length and Size. The reason for this breaking change is that although the structural implicits were rarely used, the compiler had to look at each one each time it needed to apply any implicit conversion. Scala 2.10 required that we add another 22 implicit conversions to this group, which just made the tradeoff seem unworthy. Dropping this feature eliminated a total of 46 implicit conversions. The structural type implicit conversion approach also turned out to not work if an object matched more than one of the structures, because the compile would fail complaining about ambiguous implicit conversions. The type class approach requires a bit more work up front in the (usually rare) cases that you want to use the length/size syntax with a type not already supported by ScalaTest's matchers, but compiles should be slightly faster as a result of fewer implicits being in scope.