Code testing in Haskell
A few weeks ago I wrote that for 3 years I used Matlab as my main programming language. Before that however I was developing in Java. While Java can be criticized for its design - and I noticed that some Haskallers look down on it - it learned me one very important thing: code testing. Java’s JUnit was very important in popularizing Test Driven Development (TDD) and code testing in general. Personally, I never used JUnit and relied on TestNG instead. At the time I was learning TDD TestNG’s features were way ahead of JUnit (and it is possible that they still are). Having learned TestNG, testing became very important for me. For the most time I was creating code that did numerical computations according to some sophisticated algorithm. It was very easy to make a mistake and tests allowed me to make my code reliable. Later I used TDD approach in Matlab thanks to Steve Eddins’ xUnit Framework. After starting with Haskell I knew one thing - I need a way to test my code. Luckily Haskell provides even better ways of creating tests than other languages I’ve seen so far. Still, I had some problems with organizing my tests. I spent about a week reading tutorials, talking with people on #haskell IRC channel, mailing on Haskell-cafe list and in the end even mailing authors of particular libraries. This post summarizes all my efforts. It is not meant to be a comprehensive tutorial on using testing libraries, though I will mention some of the basics. I will mostly focus on logical organization of tests, automating tests and adding some enhancements that improve capabilities of existing testing libraries. For demonstration purposes I set up a very simple stub project on github so you can clone it and see how my approach works in practice.
Why?
Before I begin with presenting my approach to testing in Haskell it is important to say why did I even bother to spend so much time and effort trying to figure out test organization. The answer is very simple: none of the approaches I was able to find in Haskell community suited my needs. I wanted to have three things:
- separate tests from the actual source code so that release version of my software doesn’t depend on testing libraries,
- organize tests in such a way that they are easy to manage (e.g. I wanted to be able to quickly locate tests for a particular module),
- automate my tests.
I read some blog posts discussing how to test code in Haskell but none of demonstrated approaches met all the above criteria. For example I noticed that many people advocate putting tests in the same source file as the tested code, arguing that tests act as a specification and should be bundled with the code. Tests are part of specification, that is true. Nevertheless this is not the reason to make our final executable (or library) depend on testing libraries! That is why I had to find my own way of doing things.
Project overview
I will create a very simple library with tests to demonstrate all the
concepts. The library provides two basic signal processing operations - cyclic
shifts. A left cyclic shift by one moves all elements in a signal (list in our
case) to the left and the formerly first element becomes the last one. For
example a cyclic shift to the left of [1,2,3,4]
produces [2,3,4,1]
. Right
shift works in similar way, only it shifts elements in opposite direction.
Shifts by one can be extended to shift signal by any natural number, e.g. shift
right by 3 of a [1,2,3,4,5]
signal yields [3,4,5,1,2]
. Here’s the complete
implementation of all these functions:
module Signal.Utils where
cyclicOneShiftLeft :: (Num a) => [a] -> [a]
:xs) = xs ++ [x]
cyclicOneShiftLeft (x
cyclicOneShiftRight :: (Num a) => [a] -> [a]
= last xs : init xs
cyclicOneShiftRight xs
cyclicShiftLeft :: (Num a) => Int -> [a] -> [a]
= []
cyclicShiftLeft _ []
cyclicShiftLeft n xs| n > 0 = cyclicShiftLeft (n - 1) . cyclicOneShiftLeft $ xs
| otherwise = xs
cyclicShiftRight :: (Num a) => Int -> [a] -> [a]
= []
cyclicShiftRight _ []
cyclicShiftRight n xs| n > 0 = cyclicShiftRight (n - 1) . cyclicOneShiftRight $ xs
| otherwise = xs
Note that cyclicOneShiftLeft
and cyclicOneShiftRight
are partial
functions. They do not work for empty lists (the former one will cause a warning
about non-exhaustive pattern match). On the other hand cyclicShiftLeft
and
cyclicShiftRight
are total functions. They work for any list and any shift
value. These two functions will thus constitute external API of our library.
The above code is placed into module Signal.Utils
. This module - and generally
all modules in a library - exports all its internal functions, thus breaking the
encapsulation principle. The library contains one main module (Signal
) that
imports all modules of the library and exports only those functions that are
meant to be the part of library’s public API. Thus Signal.hs file looks like
this:
module Signal (
cyclicShiftLeft
, cyclicShiftRightwhere
)
import Signal.Utils
Finally, the .cabal file for the library contains such entries:
library
hs-source-dirs: src
exposed-modules: Signal
other-modules: Signal.Utils
build-depends: base
ghc-options: -Wall
This ensures that users will have access only to functions that we exposed via
Signal
module. Internal functions of our library will remain hidden. Why did
we give up on module encapsulation within library? This will become clear in a
moment, when we talk about automating tests.
Overview of Haskell testing libraries
Haskell offers quite a few testing libraries. Among them there are two that seem to be in wide use and are in fact a standard - HUnit and QuickCheck. HUnit, as the name suggests, is a library providing xUnit capabilities in Haskell. The idea of using HUnit is to feed some data to functions that we are testing and compare the actual result returned by them to the result that we expect. If expected and actual results differ the test fails. Here’s a simple example:
testCyclicOneShiftRightHU :: Test
testCyclicOneShiftRightHU =
"Cyclic one shift right" ~: [4,1,2,3] @=? cyclicOneShiftRight [1,2,3,4]
This code creates an assertion that checks if the result of applying function
cyclicShiftLeft
to list [1,2,3,4] returns [2,3,4,1]. This assertion is given a
name and assigned to a test. The test is run and if the assertion is true the
test succeeds. Otherwise it fails. That’s all there is to it. If you used any
testing framework that uses the xUnit approach then you already know what HUnit
is all about. Note also, that we will NOT create tests in the form given
above. Instead we will create tests that create Assertion
:
testCyclicOneShiftLeftAssertion :: Assertion
testCyclicOneShiftLeftAssertion =
[4,1,2,3] @=? cyclicOneShiftRight [1,2,3,4]
This is required for integration with test-framework library, which I will discuss in a moment.
One thing to note is that HUnit lacks assertions that would allow to compare
floating point numbers effectively. A problem with floating points in any
language, not only Haskell, is that comparing them using equality sign my give
unexpected results due to round-off errors. Every xUnit testing framework I’ve
seen so far provided an “almost equal” assertion that allowed to compare floats
with some given precision. Since there is no such assertion in HUnit I created
it myself and placed in the Test.Utils
module:
class AEq a where
(=~) :: a -> a -> Bool
instance AEq Double where
=~ y = abs ( x - y ) < (1.0e-8 :: Double)
x
(@=~?) :: (Show a, AEq a) => a -> a -> HU.Assertion
@=~?) expected actual = expected =~ actual HU.@? assertionMsg
(where
= "Expected : " ++ show expected ++
assertionMsg "\nActual : " ++ show actual
I created AEq
(Almost Equal) type class defining “almost equal” operator and
created instances for Double
, lists and Maybe
(see source code) and then
created HUnit assertion that works just like other assertions. In our code this
assertion is not really necessary, but I included it since I think it is very
helpful if you want to test functions performing numerical computations.
Another approach to testing is offered by QuickCheck. Instead of creating test data a programmer defines properties that tested functions should always obey and QuickCheck library takes care of automatically generating test data. An example property is that if we take a signal of length n and shift it by n (either left or right) we should get the original signal as a result. Here’s how this property looks in QuickCheck:
propLeftShiftIdentity :: [Double] -> Bool
=
propLeftShiftIdentity xs length xs) xs == xs cyclicShiftLeft (
Another property that we can define is that composition of left shift by one and
right shift by one is an identity function. In case of our cyclicOneShiftLeft
and cyclicOneShiftRight
functions this will not exactly be true, because these
functions don’t work for empty lists. This means that empty lists must be
excluded from the test:
propCyclicOneShiftIdentity1 :: [Double] -> Property
=
propCyclicOneShiftIdentity1 xs not (null xs) ==>
== xs cyclicOneShiftLeft (cyclicOneShiftRight xs)
As you can see QuickCheck properties return either Bool
or Property
((You
can think of Property
as something that can be evaluated to true or
false)). When these tests are run QuickCheck generates 100 random lists to see
if the property holds for them. If for some input data the property fails then
QuickCheck reports a failed test together with data that lead to failure.
We know how to write tests. Now it is time to run all of them in one coherent testing suite. For this we will use test-framework. This framework was designed to allow using HUnit and QuickCheck tests together in a uniform fashion. I think this is not the only such framework, but I think that it does its job very well so I did not feel the need to look for anything different. Here is main testing module responsible for running all tests:
module Main (
mainwhere
)
import Test.Framework
import Test.Framework.Providers.QuickCheck2
import Test.Framework.Providers.HUnit
import Signal.UtilsTest
main :: IO ()
= defaultMain tests
main
tests :: [Test]
=
tests
["Signal shifts"
testGroup
["Migrated from HUnit" $ hUnitTestToTests testCyclicOneShiftRightHU
testGroup "L/R one shift composition" propCyclicOneShiftIdentity1
, testProperty "Left shift identity" propLeftShiftIdentity
, testProperty
] ]
The tests
function is the most important one. It groups tests into groups and
assigns names to both groups and individual tests. These names will be useful
for locating tests that failed. Notice the test group named “Migrated from
HUnit”. As the name suggests these are HUnit tests that were adjusted to work
with test-framework, which means that if you already have HUnit tests you can
easily migrate to test-framework. Nevertheless test-framework expects an
Assertion
by default and that is why we created such test earlier. Notice also
that in the project on github there are more tests than shown above. These are
however very similar to the functions already shown.
Automating tests using cabal
It is time to automate our tests so that they can be easily rerun. For that we
will use cabal
, but before we start we need to discuss how to organize our
tests and place them within project’s directories.
In Java it is a standard practice to put source and tests into two separate
directories located in the project root. These directories have identical
internal structure. This is due to two facts. First, Java names packages
according to directory in which they are located ((Haskell uses the same
approach and if I remember correctly it was adapted from Java. )), so files
src/SomePackage/SomeSubpackage/SomeClass.java
and
tests/SomePackage/SomeSubpackage/SomeClassTest.java
are considered to be
in the same package. The second reason is that classes located in the same
package can have access to their protected fields, which allows tests to access
internals of a class. This approach breaks object encapsulation within a single
package, but this is generally acceptable and not a real problem.
I decided to follow similar approach in Haskell. In the project directory I have
src
and tests
directories that allow me to separate application/library code
from tests. Both directories have the same internal structure. Files containing
tests for a module are named like that module but with “Test” appended before
file extension. In my simple project this is demonstrated by file
src/Signal/Utils.hs
and tests/Signal/UtilsTest.hs
. This way it is easy to
locate tests for a particular module. This mimics approach used in Java, but
there is one important difference. In Java tests organized in such a way have
access to unexposed internals of a class, but this does not happen in
Haskell. If a module does not expose its internal functions there is no way for
tests to reach them. I know two solutions to this problem. First is the one I
used - export everything from the modules. It was suggested to me by Matthew
West on Haskell-Cafe. The second one is using CPP language extension, which will
cause source files to be processed by C preprocessor. To use this method our
Signal.Utils
would have to be modified like this:
{-# LANGUAGE CPP #-}
module Signal.Utils (
cyclicShiftLeft
, cyclicShiftRight
#ifdef TEST
, cyclicOneShiftLeft
, cyclicOneShiftRight
#endif
) where
We also have to add cpp-options: -DTEST
entry in test section of project’s
.cabal file (this will be explained in next paragraph). It might also be
convenient to create .ghci
file in the project directory containing :set -DTEST -isrc -itest
, which will enable TEST
flag within ghci. This solution
was pointed to me by Simon Hengel, also on Haskell-Cafe. I didn’t use it because
it doesn’t look very well and feels more like a hack than a real
solution. Nevertheless this is also a way of doing things and it may better suit
your needs than the one I chose.
With all this knowledge we can finally use cabal
’s support for
testing1. For this we must add another
section to .cabal file of our project:
test-suite signal-tests
type: exitcode-stdio-1.0
hs-source-dirs: tests, src
main-is: MainTestSuite.hs
build-depends: base,
HUnit,
QuickCheck,
test-framework,
test-framework-hunit,
test-framework-quickcheck2
Let’s walk through this configuration and see what it does. The type
field
defines testing interface used by tests. Theoretically there are two accepted
values: exitcode-stdio-1.0
and detailed-1.0
. First one means that test
executable works by displaying test results on the screen and indicates possible
failure by non-zero exit code. Second option, detailed-1.0
, is meant for test
suites that export some special symbols that allow test results to be
intercepted and further processed by Cabal. Sadly, while this second options
seems very interesting, it is not fully implemented yet and there is no way to
make use of it. Thus, for the time being, we are left with
exitcode-stdio-1.0
. Rest of the entries should be self-explanatory. The
hs-source-dirs
option points to source directories. Note that it includes both
the src
and tests
directories. Next entry defines a file containing main :: IO ()
. Finally there are dependencies on external libraries.
To run tests you need to perform:
cabal configure --enable-tests
cabal build
cabal test
This will build both the library and testing binary and run the tests. Here’s how the test output looks like:
[killy@xerxes : ~] cabal test
Running 1 test suites...
Test suite wavelet-hs-test: RUNNING...
Test suite wavelet-hs-test: PASS
Test suite logged to: dist/test/haskell-testing-stub-1.0.0-signal-tests.log
1 of 1 test suites (1 of 1 test cases) passed.
The detailed result is logged to a file. If any of the tests fails then whole output from the suite is displayed on the screen (try it by supplying incorrect expected value in a HUnit test).
Cabal has also support for testing code coverage with HPC. To use it run cabal configure --enable-tests --enable-library-coverage
. This should enable HPC when
running tests, automatically exclude testing code from the coverage summaries
and generate HTML files. Sadly, I’ve been affected by some bug which results in
HTML files not being generated and testing code not being excluded from the
report. I reported this to the author so I hope it will get fixed some day.
Enhancing HUnit tests with data providers
In the beginning of my post I mentioned that TestNG library for Java offered better capabilities than JUnit. To me one of key features of TestNG were DataProviders. They allowed user to define a parametrized test function that contained test logic with assertions. For each such parametrized test user had to supply a data provider, that is a function that returned many sets of testing data that could be passed to this single test. This allowed to neatly separate test logic from test data. TestNG of course treated such tests as many different tests and it was possible for one test set to fail and others to pass. This was a big step forward, because earlier solutions to such problems lead either to duplication of test logic (violation of DRY) or locked multiple test data within one test, which caused whole test to fail on first data set that caused failure.
There are no built-in data providers in HUnit but we can easily add them. In
Test.Utils
module I created a function for this:
import qualified Test.Framework as TF
import qualified Test.Framework.Providers.HUnit as TFH
import qualified Test.HUnit as HU
testWithProvider :: String -> (a -> HU.Assertion) -> [a] -> TF.Test
=
testWithProvider testGroupName testFunction . map createTest . zipWith assignName [1::Int ..]
TF.testGroup testGroupName where
= TFH.testCase name $ testFunction dataSet
createTest (name, dataSet) = ("Data set " ++ show setNumber, dataSet) assignName setNumber dataSet
This function is very similar to other functions defined within test-framework
and thus should be considered more an enhancement to test-framework than
HUnit. The testWithProvider
function takes name for a group of tests, a test
function, a list of test data (that’s the data provider) and returns a
Test. Note that last parameter is omitted due to currying. Tests within created
group are named “Dataset n”, where n is the number. This allows to easily locate
failing test data set. Now we can write HUnit tests like this:
testCyclicShiftLeft :: (Int, [Double], [Double]) -> Assertion
=
testCyclicShiftLeft (n, xs, expected) @=~? cyclicShiftLeft n xs
expected
dataCyclicShiftLeft :: [(Int, [Double], [Double])]
=
dataCyclicShiftLeft
[0, [], [] )
( 2, [1,2,3,4], [3,4,1,2] )
, ( 4, [1,2], [1,2] )
, ( ]
Notice that test data are passed as tuples. Finally, we can add these tests to a suite like this:
tests :: [Test]
=
tests
["Signal shifts"
testGroup
[....
"Cyclic left shift" testCyclicShiftLeft
, testWithProvider
dataCyclicShiftLeft
] ]
One might argue that we really don’t need data providers, since there is QuickCheck that generates test data automatically and there is no need for programmer to do it. That is a good point, but I think that data provider capability comes in handy when we want to be sure that border cases of an algorithm are properly tested.
Summary
When I started with code testing in Haskell I had three goals in my mind: separation of tests from the code, organizing them in a manageable and flexible way and finally automating tests. The approach I demonstrated meats all these goals and is based on my experience in other programming languages. So far it works very well for me, but I dare not argue that this is the only way of doing things, not even to say that it’s the best one. As always I’m open to discussion and suggestions for improvements.
LAST MINUTE NEWS: As I was finishing writing of this post, Roman Cheplyaka announced on Haskell-Cafe release of his test-framework-golden library. This library is meant for “Golden testing” which works by writing test output to a file and comparing it with some expected (“golden”) file. I never used this approach, but this library also integrates with test-framework so it could be used in my sample project without problems. That’s the kind of test extensibility I like!
UPDATE (21/10/2012): Read a follow-up post about code benchmarking in Haskell.
When I say cabal I really mean cabal-install, a command-line tool used for installing Haskell packages, not the Cabal library. The confusion arises because cabal-install executable is named cabal.↩︎