Resources in Tasty
December 10, 2013
Update: this post conveys the general idea about resources, but in later versions of tasty (starting from 0.7), the API is a bit different. To learn more, see this subsequent article.
What problem it solves
Often a group of tests need access to a common resource, such as a connection over which they make requests, or a temporary directory in which they create files. When the tests are done, the resource should be released (the connection should be closed, the directory should be removed).
Previously, allocation of the resource could be performed outside of tasty like this:
import Test.Tasty as Tasty import Control.Exception main = bracket acquire release $ \resource -> Tasty.defaultMain ...
This solution, however, has several problems.
First, the resource is initialized in the beginning and released at the very end. Depending on the kind of resource we’re grabbing, this may be inconvenient or even infeasible.
Second, not all modes of running the test suite involve actually running the tests — like, for instance,
--list-tests. But because we’re acquiring the resource outside of tasty, we have no way to know that it’s not necessary.
A similar problem occurs when we do run tests, but not all of them. Remember, we can choose which tests to run with the
--pattern option. Maybe for the tests we want to run right now that expensive resource isn’t needed, but again, we can’t know that.
So, to avoid these kinds of problems, special support for resources has been introduced.
How to use it
There’s just one new function you need to be aware of, and its signature is very simple:
withResource :: IO a -- acquire resource -> (a -> IO ()) -- release resource -> TestTree -> TestTree
withResouce annotates a
TestTree (typically that will be a
testGroup) with the actions that should be run before and after the tests, respectively.
Note the similarity to the
bracket function from
bracket :: IO a -- computation to run first ("acquire resource") -> (a -> IO b) -- computation to run last ("release resource") -> (a -> IO c) -- computation to run in-between -> IO c -- returns the value from the in-between computation
A major difference, however, is that the third argument of
withResource isn’t a function and doesn’t have direct access to the resource.
Sometimes it’s not a big deal — if you create a temporary directory with a known name, then you just have to know it’s there.
But often you do need to access the resource from the tests. In that case, use an
IORef (created outside of tasty) to store the resource once it’s initialized. Here’s an example from the docs:
import Test.Tasty import Test.Tasty.HUnit import Data.IORef -- assumed defintions data Foo acquire :: IO Foo release :: Foo -> IO () testWithFoo :: Foo -> Assertion main = do ref <- newIORef $ -- If you get this error, then either you forgot to actually write to -- the IORef, or it's a bug in tasty error "Resource isn't accessible" defaultMain $ withResource (do r <- acquire; writeIORef ref r; return r) release (tests ref) tests :: IORef Foo -> TestTree tests ref = testGroup "Tests" [ testCase "x" $ readIORef ref >>= testWithFoo ]
Yeah, perhaps not the most elegant way to pass the resource around, but it’s simple and gets the job done.
In the next section I’ll explain why it’s done in this way.
test-framework allows generating tests in the IO monad using the function
buildTest :: IO Test -> Test
This solves the first problem described in the beginning of this article, but not the second one. Indeed, if we want list all the tests, or perform some other action on them without actually running them, we still have to execute the IO action and acquire the resources unnecessarily.
Due to the exact same reason we can’t mimic the
bracket function and pass the resource to the tests like this
withResource :: IO a -> (a -> IO ()) -> (a -> TestTree) -> TestTree
Again, to list the tests we’d have to perform the IO action. Or, when in non-running mode, we could pass
error "Don't evaluate the resource in non-running mode" as an
a. This last option may actually be not as bad — I’d be interested in what others think.
A hypocritical variation of this would be to replace
a -> TestTree with
Maybe a -> TestTree, and pass
Nothing when in non-running mode. I call it hypocritical because the user still would have to use
fromJust or a similar partial function to get the resource, but on our side it looks like everything is total. Pro: reminds the user that the resource may not be there and should not be accessed unless we’re running. Con: boilerplate pattern-matching on the resource.
a -> TestTree is not the only way to indicate that a test tree depends on the resource.
We could use type-level tricks similar to extensible-effects or regions to record which resources can be accessed by tests. I decided not to do this, because such things make code harder to understand and generally confuse users.
In addition, the problem described in the next section, «Non-type-safe way», applies to any type-safe solution, too.
Even without compile-time guarantee that the resources are acquired and of the right type, we still could automate the
IORef business and store resources in a data structure like
Map ResourceName Dynamic. We could then provide a monadic interface to accessing the properties, such as
getResource :: Typeable a => ResourceName -> TestM a
But then we’d have to teach every test provider how to run tests in our
TestM monad. In some cases (HUnit) this is just not possible. On the other hand, all test providers seem to support running simple IO actions inside tests.
A note on parallelism
There are two possible reasons to share a resource across multiple tests. It could be an optimization (to avoid creating and destroying the resource for every single test), or it could be a semantic requirement. In the latter case, one might want not to enable parallelism to avoid tests running simultaneously or in the wrong order.
While it’s possible to use a
TVar to ensure ordering of the tests, that would hurt actual parallelism. Tasty can’t know that the test waits on a
TVar instead of running, so it won’t be executing other tests during that time either.