The Philosophy and design of ScalaTest

by Bill Venners, creator of ScalaTest

The philosophy

My main goal for creating ScalaTest was the same goal I had for promoting Scala in the first place: to help programmers become more productive. Increased productivity means releasing new features more quickly, with better accuracy and quality. Better accuracy means building the right features. Better quality means fewer defects.

Although writing tests can slow you down in the short term compared to just writing the production code, the time spent writing tests can pay back in increased quality and reduced time in the debugger. Over the long term tests can provide a safety net that enables you to make changes to existing code with greater speed and confidence, though during major refactors, fixing the many broken tests can add a significant extra cost. You can consider this cost another investment in testing, weighing the up-front cost of fixing the broken tests against the potential for returns in increased velocity and quality over time.

Accuracy, building the right features, requires fluid communication between developers and other stakeholders such as management, testers, and customers. Tests can potentially facilitate such communication because tests represent a highly accurate view of the state of the system.

My goal with ScalaTest is to improve the return on investment of testing. I've tried to minimize the cost of testing by making test code concise, plainly obvious, quickly understandable. A guiding design principle of ScalaTest is that different people on a team should be able look at each others test code and know immediately what's going on. I've also tried to bring that clarity out of the code and into the artifacts generated as tests run. My goal with the generated artifacts is to facilitate communication between all stakeholders—developers certainly, but also management, testers, and customers.

Lastly, ScalaTest is not a one-size-fits-all testing framework, it is a platform that can host different styles of testing. Why? Because different people with different problems need different tools to solve them. Even the same people need different tools in different situations. ScalaTest is designed to make it easy for you to customize your testing tool to meet your current needs, and for the built-in traits at least, make it easy for anyone who comes along later to read and understand your code.

The design

ScalaTest is the “scalable test framework.” Like the Scala language on which it is built, ScalaTest is designed to grow with the demands of its users.

Scala is a scalable language in two ways. First, Scala can be molded through library and DSL design to fit widely different tasks. Second, Scala scales both down to small tasks and up to large ones. It feels as natural to use for small tasks like scripting as it does for large tasks like major software projects built by large teams. ScalaTest is scalable in similar ways. First, it can be easily molded by overriding its lifecycle methods to address special testing needs when they arise. Second, it is designed facilitate both small tasks in the Scala interpreter (see Assertions, should.Matchers, and the ScalaTest Shell) and to scale up to testing very large software projects built by large teams.

The design of ScalaTest can be summed up in one sentence:

ScalaTest allows you to ask a suite of tests to run itself.

The terms suite and test are defined abstractly in ScalaTest to enable a wide variety of implementations. A test in ScalaTest is anything with a name that can start and either succeed or fail. A suite is a collection of zero to many tests. ScalaTest's core trait Suite, which represents the concept of a suite of zero to many tests, declares a run method. To ask a Suite of tests to run itself, you invoke its run method.

The run method is one of many lifecycle methods in trait Suite. These lifecycle methods, shown in the table below, define a default way to run a suite of tests and serve as extension points for customization.

Trait Suite is a “functional object” in that it contains no mutable state. What's more, it contains no immutable state either: trait Suite is pure behavior. Its run method does two things, invokes runNestedSuites and runTests. The runNestedSuites method invokes nestedSuites, then invokes run on each element in the returned list. The runTests method invokes testNames and tags, to get a list of test names and their tags. The testNames method uses reflection to discover methods starting with "test", and tags uses reflection to find any tag annotations on the test methods. Given this data, runTests filters the tests by their tags, and for each test to run passes the test name to runTest. The runTest method wraps the named test method in a function that invokes the method via reflection and passes the function to withFixture, which simply invokes the function.

Scaling to different testing needs

How does this design enable ScalaTest to address widely different testing needs? You need look no further than ScalaTest itself for examples. First, ScalaTest facilitates several different styles of testing through its style traits such as AnyFunSuite, AnyFunSpec, and AnyFeatureSpec. One way to think of these style traits is as examples of ScalaTest using its own extension points to customize itself. All three override the same five lifecycle methods: run, runTests, testNames, tags, and runTest. They inherit the remaining lifecycle methods as is from their supertrait Suite.

ScalaTest also provides many stackable traits that can be mixed into a style trait, such as BeforeAndAfterEach, OneInstancePerTest, and ParallelTestExecution. Trait BeforeAndAfterEach declares methods beforeEach and afterEach, then overrides just one lifecycle method, runTest, so that it invokes beforeEach before, and afterEach after, invoking super.runTest. Both OneInstancePerTest and ParallelTestExecution override runTests. OneInstancePerTest's implementation of runTests ensures each test is run in its own instance of the suite class, and ParallelTestExecution's implementation ensures tests are run in parallel.

There is no magic happening behind the scenes. In Scala you could implement something that looks and behaves like a while loop as a curried function that takes a by-name. Similarly you could implement any of ScalaTest's style or stackable traits yourself outside of ScalaTest. As a result, when you do have a special testing need, you will be able to easily accommodate it by overriding one or more of ScalaTest's lifecycle methods in a trait, and mixing that into your test classes. Need to run each test in their own class loader? Need to treat a directory full of files as one suite of tests per file? Need to define a particular order of test or suite execution? Any of these would be easy to implement by overriding lifecycle methods. And because it is so easy to customize, ScalaTest will grow with you as your project scales up and your needs change.

Enabling team productivity

Another design focus of ScalaTest is to facilitate the productivity of teams. Teams in the real world are made up of many different kinds of people. These people are often very busy, sometimes tired or under stress. Many are not fans of testing, and even those who think testing is a good idea may not have much time to invest in becoming an expert in a test framework.

One way in which ScalaTest is designed for teams is that it is easy to get started with because it offers styles familiar to many users. Users of JUnit or TestNG, for example, can continue using those tools with a few productivity enhancements from ScalaTest sprinkled in. Or if they use AnyFunSuite with Assertions, they'll find an even nicer DSL for testing that still makes sense given their xUnit background. Users of RSpec will find AnyFunSpec familiar and easy to get into. Users of Cucumber will find AnyFeatureSpec familiar. In short, one of ScalaTest's design goals is that anyone who has used a test framework in the past should be able to get going with ScalaTest quickly with minimum effort.

Another way ScalaTest is designed for teams is that it is very thoroughly documented in its user guide and Scaladoc, both of which include many examples. The goal is that you should never need to look through ScalaTest source code to figure out what is going on.

Another major design focus of ScalaTest is to use the flexibility of the Scala language to make testing code clear. Because some people working in a team will inevitably be casual, non-expert users of the test framework, ScalaTest's design is focused on making client code “guessable.” The goal is to make client code so plainly obvious that even someone completely unfamiliar with ScalaTest could understand client code written by others without looking anything up in ScalaTest documentation.

In addition to trying to make ScalaTest's syntax as clear and meaningful as possible, the design is also focused on making ScalaTest's behavior as simple as possible. Keeping the test framework simple helps make your test code easier to reason about. The less you need to worry about what ScalaTest is doing behind the scenes, the more you can focus on what your own code is doing.

Lastly, in addition to trying to make code easy to read and understand, ScalaTest is designed to make the code as easy as possible to write. To write ScalaTest code, you will need to look at the documentation for examples, but the library is designed to be easy to remember so that you don't have to look again each time you want to write something. Besides obviousness of code, one other way in which ScalaTest tries to achieve this is through consistency. Once you learn how BeforeAndAfter works, for example, you can use that knowledge when using different style traits, because it is consistent. It works the same everywhere. This “rememberability” is important for teams, because some people on a team will inevitably have many other duties besides writing ScalaTest code, so they will have a tendency to forget what they learned.

Summary

The upshot is that ScalaTest is designed to facilitate productivity of teams by being:

• easy to get into
• easy to read, even by casual users
• easy to remember how to write
• easy to customize to address special needs

Next, find out about Selecting a style.