Using ScalaTest with sbt

ScalaTest's Framework supports using ScalaTest from sbt. In sbt version 0.10 and higher, you add ScalaTest to your project in three easy steps.

1. add this to your project file:

libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.18"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.18" % "test"

The dependency on Scalactic, ScalaTest's sister library focused on quality through types, is recommended but not required.

2. We also recommend you also include the SuperSafe Community Edition Scala compiler plugin, which will flag errors in your ScalaTest (and Scalactic) code at compile time, by first adding this line to ~/.sbt/0.13/global.sbt:

resolvers += "Artima Maven Repository" at "http://repo.artima.com/releases"

3. Then adding the following line to project/plugins.sbt:

addSbtPlugin("com.artima.supersafe" % "sbtplugin" % "1.1.12")

Buffered Output

By default, sbt buffers log output for each suite until all tests for that suite complete and causing "spurty" output. We recommend you disable sbt's log buffering so you can enjoy ScalaTest's built-in event buffering algorithm, which shows the events of one suite as they occur until that suite either completes or a timeout occurs, at which point ScalaTest switches a different suite's events. Just add this to your sbt build:

logBuffered in Test := false

Running Tests

Your test source files go into src/test/scala. You can run all of them from the sbt console with:

> test

If you want to run particular suites, you can use test-only and provide their fully qualified names in a space separated list:

> test-only org.acme.RedSuite org.acme.BlueSuite

Or you can specify a glob:

> test-only *RedSuite

To run only tests affected by your latest code changes, either in main or test, you can use test-quick:

> test-quick

As with test-only, you can specify a space-separated list of fully qualified names and globs to further reduce the tests selected.

Running Suites in Parallel

With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is common to run tests in parallel. Sbt by default runs suites in parallel, using its own thread pool. In case you need to run your suites serially, you can add the following to your sbt build file:

parallelExecution in Test := false

ScalaTest uses its own thread pool to run ParallelTestExecution suites, its pool size is determined by the following formula:

Pool Size = Available Processors x 2

Showing the Test Classpath

To see classpath that sbt uses to lookup for test classes, you can use the following command:

> show test:full-classpath

Specifying ScalaTest Arguments

You can pass arguments to ScalaTest by using testOptions and Tests.Argument in your sbt build file:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-oD")

The -oD argument above will be pass to ScalaTest for all test runs, you can also pass arguments for individual runs by using test-only and placing them after --, like this:

> test-only org.acme.RedSuite -- -oD

Using Reporters

You can use ScalaTest's reporters by specifying the passing the following arguments to ScalaTest:

• -f[configs...] <filename> - causes test results to be written to the named file
• -u <directory> - causes test results to be written to junit-style xml files in the named directory
• -h <directory> [-Y ] - causes test results to be written to HTML files in the named directory, optionally included the specified CSS file
• -a <number of files to archive> - causes specified number of old summary and durations files to be archived (in summaries/ and durations/ subdirectories) for dashboard reporter (default is two)
• -o[configs...] - causes test results to be written back to sbt, which usually displays it on the standard output
• -e[configs...] - causes test results to be written to the standard error
• -k <host> <port> - causes test results to be written to socket in the named host and port number, using XML format
• -K <host> <port> - causes test results to be written to socket in the named host and port number, using Java object binary format
• -C[configs...] <reporterclass> - causes test results to be reported to an instance of the specified fully qualified Reporter class name

Each reporter argument on the command line can include configuration characters. Configuration characters are specified immediately following the -o, -e, -f, or -C. The following configuration characters, which cause reports to be dropped, are valid for any reporter:

• N - drop TestStarting events
• C - drop TestSucceeded events
• X - drop TestIgnored events
• E - drop TestPending events
• H - drop SuiteStarting events
• L - drop SuiteCompleted events
• O - drop InfoProvided events
• P - drop ScopeOpened events
• Q - drop ScopeClosed events
• R - drop ScopePending events
• M - drop MarkupProvided events

A dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not present information about the event in its report. For example, if you specify -oN, the standard output reporter will never receive any TestStarting events and will therefore never report them. The purpose of these configuration parameters is to allow users to selectively remove events they find add clutter to the report without providing essential information.

The following reporter configuration parameters may additionally be used on standard output (-o), standard error (-e), and file (-f) reporters:

• W - without color
• D - show all durations
• S - show short stack traces
• F - show full stack traces
• U - unformatted mode
• I - show reminder of failed and canceled tests without stack traces
• T - show reminder of failed and canceled tests with short stack traces
• G - show reminder of failed and canceled tests with full stack traces
• K - exclude TestCanceled events from reminder

If you specify a W, D, S, F, U, R, T, G, or K for any reporter other than standard output, standard error, or file reporters, ScalaTest will complain with an error message and not perform the run.

Configuring a standard output, error, or file reporter with D will cause that reporter to print a duration for each test and suite. When running in the default mode, a duration will only be printed for the entire run.

Configuring a standard output, error, or file reporter with F will cause that reporter to print full stack traces for all exceptions, including TestFailedExceptions. Every TestFailedException contains a stack depth of the line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default, mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown by production code. When a TestFailedException is thrown in default mode, only the source filename and line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack trace.

The U unformatted configuration removes some formatting from the output and adds verbosity. The purpose of unformatted (or, "ugly") mode is to facilitate debugging of parallel runs. If you have tests that fail or hang during parallel runs, but succeed when run sequentially, unformatted mode can help. In unformatted mode, you can see exactly what is happening when it is happening. Rather than attempting to make the output look as pretty and human-readable as possible, unformatted mode will just print out verbose information about each event as it arrives, helping you track down the problem you are trying to debug.

By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset terminal colors. Information printed as a result of run starting, completed, and stopped events is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green. The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode (W) will turn off this behavior. No ansi codes will be inserted.

The R, T, and G options enable "reminders" of failed and, optionally, canceled tests to be printed at the end of the summary. This minimizes or eliminates the need to search and scroll backwards to find out what tests failed or were canceled. For large test suites, the actual failure message could have scrolled off the top of the buffer, making it otherwise impossible to see what failed. You can configure the detail level of the stack trace for regular reports of failed and canceled tests independently from that of reminders. To set the detail level for regular reports, use S for short stack traces, F for full stack traces, or nothing for the default of no stack trace. To set the detail level for reminder reports, use T for reminders with short stack traces, G for reminders with full stack traces in reminders, or R for reminders with no stack traces. If you wish to exclude reminders of canceled tests, i.e., only see reminders of failed tests, specify K along with one of R, T, or G, as in "-oRK".

For example, to run a suite using two reporters, the file reporter configured to present every reported event and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test pending, suite starting, suite completed, and info provided events, you would use the following in your sbt build file:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-f", "result.txt", "-eNDXEHLO")

Note that no white space is allowed between the reporter option and the initial configuration parameters. So "-e NDXEHLO" will not work, "-eNDXEHLO" will work.

Include and Exclude Tests with Tags

You can specify tag names of tests to include or exclude from a run. To specify tags to include, use -n followed by a list of tag names to include. Similarly, to specify tags to exclude, use -l followed by a list of tag names to exclude. If tags to include is not specified, then all tests except those mentioned in the tags to exclude (and in the org.scalatest.Ignore tag), will be executed. (In other words, the absence of a -n option is like a wildcard, indicating all tests be included.) If tags to include is specified, then only those tests whose tags are mentioned in the argument following -n and not mentioned in the tags to exclude, will be executed. For more information on test tags, see the documentation for Suite. Here are some examples:

> test-only org.acme.* -- -n CheckinTests
> test-only org.acme.* -- -n FunctionalTests -l org.scalatest.tags.Slow
> test-only org.acme.* -- -n "CheckinTests FunctionalTests" -l "org.scalatest.tags.Slow org.scalatest.tags.Network"

Specifying Chosen Styles

You can optionally specify chosen styles for a ScalaTest run. ScalaTest supports different styles of testing so that different teams can use the style or styles that best suits their situation and culture. But in any one project, it is recommended you decide on one main style for unit testing, and consistently use only that style for unit testing throughout the project. If you also have integration tests in your project, you may wish to pick a different style for them than you are using for unit testing. You may want to allow certain styles to be used in special testing situations on a project, but in general, it is best to minimize the styles used in any given project to a few, or one.

To facilitate the communication and enforcement of a team's style choices for a project, you can specify the chosen styles in your project build. If chosen styles is defined, ScalaTest style traits that are not among the chosen list will abort with a message complaining that the style trait is not one of the chosen styles. The style name for each ScalaTest style trait is its fully qualified name. For example, to specify that org.scalatest.funspec.AnyFunSpec as your chosen style you'd pass this to ScalaTest in your sbt build file:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-y", "org.scalatest.funspec.AnyFunSpec")

If you wanted org.scalatest.funspec.AnyFunSpec as your main unit testing style, but also wanted to allow AnyPropSpec for test matrixes and AnyFeatureSpec for integration tests, you would write:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-y", "org.scalatest.funspec.AnyFunSpec",
    "-y", "org.scalatest.propspec.AnyPropSpec", "-y", "org.scalatest.featurespec.AnyFeatureSpec")

To select org.scalatest.flatspec.AnyFlatSpec as your main unit testing style, but allow org.scalatest.fixture.AnyFlatSpec for multi-threaded unit tests, you'd write:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-y", "org.scalatest.flatspec.AnyFlatSpec",
    "-y", "org.scalatest.fixture.AnyFlatSpec")

The style name for a suite is obtained by invoking its styleName method. Custom style traits can override this method so that a custom style can participate in the chosen styles list.

Because ScalaTest is so customizable, a determined programmer could circumvent the chosen styles check, but in practice -y should be persuasive enough tool to keep most team members in line.

Specifying a span scale factor

If you specify a integer or floating point span scale factor with -F, trait ScaledTimeSpans trait will return the specified value from its implementation of spanScaleFactor. This allows you to tune the "patience" of a run (how long to wait for asynchronous operations) from the command line. For more information, see the documentation for trait ScaledTimeSpans.

Slowpoke notifications

You can request to receive periodic notifications of slowpokes, tests that have been running longer than a given amount of time, specified in seconds by the first integer after -W, the delay. You specify the period between slowpoke notifications in seconds with the second integer after -W, the period. Thus to receive notifications very minute of tests that have been running longer than two minutes, you'd use:

testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-W", "120", "60")

Slowpoke notifications will be sent via AlertProvided events. The standard out reporter, for example, will report such notifications like:

*** Test still running after 2 minutes, 13 seconds: suite name: ExampleSpec, test name: An egg
    timer should take 10 minutes.