Resources in Tasty

This article explores the new feature of the Tasty test framework — resources. It was added in the version 0.5 (and got a small fix in 0.5.1).

Update: this post conveys the general idea about resources, but in later versions of tasty (starting from 0.7), the API is a bit different. To learn more, see this subsequent article.

What problem it solves

Often a group of tests need access to a common resource, such as a connection over which they make requests, or a temporary directory in which they create files. When the tests are done, the resource should be released (the connection should be closed, the directory should be removed).

Previously, allocation of the resource could be performed outside of tasty like this:

import Test.Tasty as Tasty
import Control.Exception

main =
  bracket acquire release $ \resource ->
    Tasty.defaultMain ...

This solution, however, has several problems.

First, the resource is initialized in the beginning and released at the very end. Depending on the kind of resource we’re grabbing, this may be inconvenient or even infeasible.

Second, not all modes of running the test suite involve actually running the tests — like, for instance, --help or --list-tests. But because we’re acquiring the resource outside of tasty, we have no way to know that it’s not necessary.

A similar problem occurs when we do run tests, but not all of them. Remember, we can choose which tests to run with the --pattern option. Maybe for the tests we want to run right now that expensive resource isn’t needed, but again, we can’t know that.

So, to avoid these kinds of problems, special support for resources has been introduced.

How to use it

There’s just one new function you need to be aware of, and its signature is very simple:

withResource
  :: IO a         -- acquire resource
  -> (a -> IO ()) -- release resource
  -> TestTree
  -> TestTree

withResouce annotates a TestTree (typically that will be a testGroup) with the actions that should be run before and after the tests, respectively.

Note the similarity to the bracket function from Control.Exception:

bracket
  :: IO a         -- computation to run first ("acquire resource")
  -> (a -> IO b)  -- computation to run last ("release resource")
  -> (a -> IO c)  -- computation to run in-between
  -> IO c         -- returns the value from the in-between computation

A major difference, however, is that the third argument of withResource isn’t a function and doesn’t have direct access to the resource.

Sometimes it’s not a big deal — if you create a temporary directory with a known name, then you just have to know it’s there.

But often you do need to access the resource from the tests. In that case, use an IORef (created outside of tasty) to store the resource once it’s initialized. Here’s an example from the docs:

import Test.Tasty
import Test.Tasty.HUnit
import Data.IORef

-- assumed defintions
data Foo
acquire :: IO Foo
release :: Foo -> IO ()
testWithFoo :: Foo -> Assertion

main = do
  ref <- newIORef $
    -- If you get this error, then either you forgot to actually write to
    -- the IORef, or it's a bug in tasty
    error "Resource isn't accessible"
  defaultMain $
    withResource (do r <- acquire; writeIORef ref r; return r) release (tests ref)

tests :: IORef Foo -> TestTree
tests ref =
  testGroup "Tests"
    [ testCase "x" $ readIORef ref >>= testWithFoo
    ]

Yeah, perhaps not the most elegant way to pass the resource around, but it’s simple and gets the job done.

In the next section I’ll explain why it’s done in this way.

Alternative designs

test-framework’s way

test-framework allows generating tests in the IO monad using the function

buildTest :: IO Test -> Test

This solves the first problem described in the beginning of this article, but not the second one. Indeed, if we want list all the tests, or perform some other action on them without actually running them, we still have to execute the IO action and acquire the resources unnecessarily.

bracket way

Due to the exact same reason we can’t mimic the bracket function and pass the resource to the tests like this

withResource
  :: IO a
  -> (a -> IO ())
  -> (a -> TestTree)
  -> TestTree

Again, to list the tests we’d have to perform the IO action. Or, when in non-running mode, we could pass error "Don't evaluate the resource in non-running mode" as an a. This last option may actually be not as bad — I’d be interested in what others think.

A hypocritical variation of this would be to replace a -> TestTree with Maybe a -> TestTree, and pass Nothing when in non-running mode. I call it hypocritical because the user still would have to use fromJust or a similar partial function to get the resource, but on our side it looks like everything is total. Pro: reminds the user that the resource may not be there and should not be accessed unless we’re running. Con: boilerplate pattern-matching on the resource.

Type-safe way

a -> TestTree is not the only way to indicate that a test tree depends on the resource.

We could use type-level tricks similar to extensible-effects or regions to record which resources can be accessed by tests. I decided not to do this, because such things make code harder to understand and generally confuse users.

In addition, the problem described in the next section, «Non-type-safe way», applies to any type-safe solution, too.

Non-type-safe way

Even without compile-time guarantee that the resources are acquired and of the right type, we still could automate the IORef business and store resources in a data structure like Map ResourceName Dynamic. We could then provide a monadic interface to accessing the properties, such as

getResource :: Typeable a => ResourceName -> TestM a

But then we’d have to teach every test provider how to run tests in our TestM monad. In some cases (HUnit) this is just not possible. On the other hand, all test providers seem to support running simple IO actions inside tests.

A note on parallelism

There are two possible reasons to share a resource across multiple tests. It could be an optimization (to avoid creating and destroying the resource for every single test), or it could be a semantic requirement. In the latter case, one might want not to enable parallelism to avoid tests running simultaneously or in the wrong order.

While it’s possible to use a TVar to ensure ordering of the tests, that would hurt actual parallelism. Tasty can’t know that the test waits on a TVar instead of running, so it won’t be executing other tests during that time either.