Happy, Alex, and GHC 7.8

Published on March 8, 2014

As we approach the 7.8 release of GHC, more and more people are running into problems with packages that use Alex and/or Happy for parsing.

The errors look like

templates/GenericTemplate.hs:104:22:
    Couldn't match expected type ‛Bool’
                with actual type ‛Happy_GHC_Exts.Int#’
    In the expression:
      (n Happy_GHC_Exts.<# (0# :: Happy_GHC_Exts.Int#))
    In a stmt of a pattern guard for
                   a case alternative:
      (n Happy_GHC_Exts.<# (0# :: Happy_GHC_Exts.Int#))
    In a case alternative:
        n | (n Happy_GHC_Exts.<# (0# :: Happy_GHC_Exts.Int#))
          -> (happyReduceArr Happy_Data_Array.! rule) i tk st
          where
              rule
                = (Happy_GHC_Exts.I#
                     ((Happy_GHC_Exts.negateInt#
                         ((n Happy_GHC_Exts.+# (1# :: Happy_GHC_Exts.Int#))))))

for Happy and

    Pattern bindings containing unlifted types should use an outermost bang pattern:
      ((I# (ord_c))) = ord c
    In the expression:
      let
        (base) = alexIndexInt32OffAddr alex_base s
        ((I# (ord_c))) = ord c
        (offset) = (base +# ord_c)
        ....
      in
        case new_s of {
          -1# -> (new_acc, input)
          _ -> alex_scan_tkn
                 user orig_input (len +# 1#) new_input new_s new_acc }
    In a case alternative:
        Just (c, new_input)
          -> let
               (base) = alexIndexInt32OffAddr alex_base s
               ((I# (ord_c))) = ord c
               ....
             in
               case new_s of {
                 -1# -> (new_acc, input)
                 _ -> alex_scan_tkn
                        user orig_input (len +# 1#) new_input new_s new_acc }
    In the second argument of ‘seq’, namely
      ‘case alexGetChar input of {
         Nothing -> (new_acc, input)
         Just (c, new_input)
           -> let
                (base) = ...
                ....
              in
                case new_s of {
                  -1# -> ...
                  _ -> alex_scan_tkn
                         user orig_input (len +# 1#) new_input new_s new_acc } }’

for Alex. (These are not all the error messages that are produced by GHC, but hopefully enough that this article is googlable.)

First I give instructions on how to fix these problems, and then explain why they arise in the first place.

TL;DR: how do I fix the package?

As a maintainer

Install the latest versions of alex and happy. GHC 7.8 support was added in alex-3.1.0 and happy-1.19.0, but later versions contain additional bugfixes.

Double-check that cabal picks the latest versions of these tools: in your package’s source tree run

cabal configure -v | grep -e alex -e happy

The output should look like

Using alex version 3.1.3 found on system at: /home/feuerbach/bin/alex
Using happy version 1.19.3 found on system at: /home/feuerbach/bin/happy

Bump the package’s version (the fourth component is enough: e.g. 1.2.3 -> 1.2.3.1), build the package and upload:
```
cabal build
cabal sdist
cabal upload dist/$pkg-$version.tar.gz
```

That’s it; no actual source code modification to your package is necessary. If you’re curious as to why this works, read on.

As a user

First of all, check that you have latest alex and happy installed. That by itself can resolve your problem.

If it doesn’t, notify the package maintainer(s) about this problem and send them a link to this article. Only they are in a position to fix the problem properly.

Until the maintainer(s) react, you can fix the problem locally as follows:

Get into the source tree:
```
cabal get $pkg
cd $pkg-$version
```
(If cabal says it doesn’t know about the get command, you have to update it with
```
cabal install cabal-install
```
The command was called unpack before, but since you are using GHC 7.8 now, the older versions of cabal will get you in trouble anyway.)
Now that you’re in the source tree, run
```
cabal clean
```
You may think «but I’ve just downloaded a fresh copy of the package’s source — surely it is clean!» Not really; read on for the details.
Finally,
```
cabal install
```
should run without any alex- or happy-related errors.

What’s going on here?

Code produced by old Happy and Alex no longer builds

Because Alex and Happy strive to produce the most efficient code, they make use of unboxed types and primitives. And those are affected by certain changes in GHC 7.8:

PrimOps for comparing unboxed values now return Int# instead of Bool
Lazy unlifted bindings are an error now. For background on this one, see #2806, #3278 and #8022.

Happy and Alex were then updated to generate code that builds with the new GHC. So, it seems, just updating happy/alex should do the trick. Not so fast!

cabal includes generated code in the source distribution

When cabal creates a source distribution for uploading to hackage (cabal sdist), it includes the files generated by alex and happy in the tarball. So even when you have the new alex and happy installed, cabal install will not see the need to regenerate .hs files from .x and .y sources, and will run into the errors described above.

That’s why maintainers have to re-upload their tarballs generated with new alex and happy; and until they do, users have to run cabal get and cabal clean.

The rationale behind this cabal behavior is not to force users install alex or happy. Alas, it doesn’t work so well in practice:

Dec 09 19:37:21 dcoutts refold: there's a few problems with our shipped pre-processed sources system
Dec 09 19:37:41 dcoutts it doesn't interact well with using  build-tools: happy
Dec 09 19:38:03 dcoutts if there are shipped sources then obviously we do not need happy
Dec 09 19:38:14 dcoutts the shipped sources currently go in dist
Dec 09 19:38:18 dcoutts that then fails if you clean
Dec 09 19:38:33 dcoutts it only allows one instance of shipped sources
Dec 09 19:38:52 dcoutts e.g. for happy & alex, they can produce ghc-specific output or generic output
Dec 09 19:39:20 dcoutts this is less of a problem these days since in practice there are
                        not other compilers
Dec 09 19:40:19 dcoutts and then this new problem, if we do ship sources, we don't know what version
                        of the pre-processor generated them, so we cannot easily hack around version
                        incompatibilities
Dec 09 19:40:21 refold  yes, using dist is hack
Dec 09 19:40:30 dcoutts the plan was to use a different dir
Dec 09 19:40:31 refold  also fails with a different --builddir
Dec 09 19:40:34 dcoutts right

cabal sdist is not “pure”

We usually think of cabal sdist as a pure function taking in the source tree and producing the tarball. It’s not that simple.

Above I wrote that cabal sdist includes alex and happy-generated sources in the tarball. However, as Mikhail Glushenkov explains, it doesn’t actually generate them. It only includes them if they are already present as an artifact of a previous cabal build.

When I uploaded haskell-src-exts-1.16.0, I wasn’t aware of this and apparently ran cabal clean before cabal sdist. As a consequence, the tarball doesn’t have the dist/ subdirectory with generated files as you can easily check.

In order to install that particular version of haskell-src-exts, a user needs to have happy installed (and if her happy is old, she’ll get the exact same error described above). When I learned about it, I made a point release, 1.16.0.1, which does include the happy output.

So this is another thing that maintainers need to be aware of and watch out for.